utils

Note

While this documentation is extracted from a submodule, as shown below, all information is available from the top level, i.e. you can (should) use:

from token_utils import pairwise

since this will always be guaranteed to work, even if refactoring occurs.

API extracted by Sphinx

class token_utils.utils.BracketStack

Helpful in keeping track of open and close brackets in a token sequence.

It is intended to be strict, and only accept Token instances, with open brackets added before any close one can be added.

__init__()

__weakref__

list of weak references to the object (if defined)

add(bracket)

Adds a bracket to a stack.

If it is a closing bracket that matches the last added one, that last opening one is returned.

If it is a close bracket not matching the last added one, or the first one added, a TypeError is raised.

If an open bracket is added, the NULL token is returned. Note that the NULL token evaluates to False as a condition (e.g. if NULL == if False.)

is_empty()

Return True if the stack is empty, False if it contains brackets.

class token_utils.utils.IndentStack

Class designed to help keep track of indentation associated with the use of specific keywords.

__init__()

__weakref__

list of weak references to the object (if defined)

add_same_indent_keyword(name)

Add a new same indent keyword, similar to elif, else, except, etc.

Useful when working with alternatives to standard Python syntax.

add_top_indent_keyword(name)

Add a new top indent keyword, similar to class, def, try, etc.

Useful when working with alternatives to standard Python syntax.

is_token_in_named_block(token, name)

Returns True if the token is indented more than a token with the name given.

Example usage: figuring out if a ‘def’ represents a method of a class, or a pure function definition.

print_stack()

Useful for diagnostic

top_item()

Returns the token at the top of the stack or a NULL token.

update(line)

Receives a line as a list containing tokens and keeps track of the indentation of the code.

A line will deem to signal the beginning of an indented block if it begins with an appropriate keyword (class, def, if, else, ...) and ends with a colon. For this reason, it is highly recommended to obtain lines using get_logical_lines() since lines obtained using get_physical_lines() may have their last token on a different physical line.

If the line signals a change in indentation (indent or dedent), this function returns the last “top token” starting a line at that indentation. By “top token” we mean a token that can start a block such as if (for if/elif/else), try, etc.

It returns the token NULL if there is no such token.

token_utils.utils.pairwise(iterable, prev=0)

Similar to itertools.pairwise (Python 3.10+). However, it adds a fake token at the end if prev==0 (the default) or at the beginning if prev==1. Any other value will result in a ValueError

Given a list of tokens represented by lower case letters, and a fake token by $, the default corresponds to something like:

pairwise('abcde') → ab bc cd de e$

token_utils.utils.sliding_window(iterable, n, prev=0)

Collect data into overlapping fixed-length chunks or blocks.

This is inspired by an itertools recipe. Given an iterator, and a requested ‘window’ of size ‘n’, by default (prev=0), it adds ‘n-1’ fake token at the end and iterates, emitting ‘n’ items at a time until all the items have been served.

Given a list of tokens represented by lower case letters, and $ representing a fake token, we would have something like:

sliding_window(‘abcde’, 3) → abc bcd cde deF e$$

With prev==1, we would have 1 fake token prepended and one appended; thus

sliding_window(‘abcde’, 3, prev=1) → $ab abc bcd cde de$

We must have 0 <= prev <= n-1, otherwise a ValueError is raised

We essentially have:

sliding_window(iterable, 2) == pairwise(iterator)

token_utils.utils.split_at_token(seq, token)

Split a list of tokens in two sublists: those occuring before the token specified, and those after.