Tokenizing related methods

Note

While this documentation is extracted from a submodule, as shown below, all information is available from the top level, i.e. you can (should) use:

from token_utils import tokenize, untokenize

since this will always be guaranteed to work, even if refactoring occurs.

API extracted by Sphinx

tokenizing.py

All the functions dealing with tokenizing/untokenizing.

token_utils.tokenizing.dedent(tokens, nb)

Given a list of tokens representing a single physical line, produces an equivalent list corresponding to a line of code with the first nb characters removed.

If the list doesn’t include any tokens, or if a negative value for nb is given, a ValueError is raised.

If an attempt to remove non-space characters is made, a TypeError is raised.

token_utils.tokenizing.generate_tokens(source)

Tokenize a source (string) yielding tokens one at a time.

token_utils.tokenizing.get_logical_lines(source, remove_comments=True, remove_semi_colons=True)

This retrieves logical lines, each essentially correponding to a single statement.

Warning: untokenize(get_logical_lines) != source in general. However, the untokenized version executes the same as the original.

When two or more physical lines are joined explitly into logical lines using backslash characters (`) or **implicitly** by using parentheses, square brackets or curly braces opening on a given physical line and closing on another, the ``NL` token is used to indicate that a \n was used. By removing these continuation symbol, or signs, we can potentially combine many physical line into a single logical line ~ statement. (Note that we could have multiple statements separated by semi-colons ;.)

Empty lines (or lines with only comments) also end with an NL token.

This function extract logical lines, removing all space tokens (INDENT, etc.) NL as well as ENDMARKER. This makes it possible to focus on tokens that are relevant for modifying the syntax, with the exception of NEWLINE.

token_utils.tokenizing.get_physical_lines(source, remove_comments=True)

Transforms a source (string) into a list of of list of Tokens, with each (inner) list containing all the tokens found on a given physical line of code except comments that correspond to indentation.

Note that lines visually separated by a continuation character (\) are considered to be part of the same physical line

Thus, for a given (inner) list of tokens, the first token (list[0]) will either be:

• A newline character (NL or NEWLINE token)

• A significant (non-space) token.

• An ENDMARKER token

Similarly, the last token (list[-1]) will be a either a new line character (NL or NEWLINE token) or ENDMARKER token.

Set remove_comments to False to keep comments. In that case, the first token oa an inner list could be a comment.

## If one needs lines with Indentation information still present, ## see Issue #14.

token_utils.tokenizing.print_tokens(source)

Prints tokens found in source, line by line, followed by a

string version of that line.

source is either a string to be tokenized, a list of Token objects, or a list of lists of tokens.

This can be useful as a debugging tool.

token_utils.tokenizing.stringify(tokens, remove_comments=False)

Returns a string built from tokens, or list of tokens.

It is somewhat similar to untokenize except that it doesn’t add any missing information from tokens that might have been removed, inserting spaces instead. For example, removing “two”:

|one two three|  -->
|one     three|

If no token has been removed from a tokenized list, and no tab characters are used for indentation or spacing between tokens, it should return the same content as untokenize.

It allows for easy removal of comments with remove_comments=True. As long as one does not care about tab characters, it is slightly more efficient to use:

stringify(tokenize(source), remove_comments=True)

than:

ideas.utils.remove_comments(source)

token_utils.tokenizing.tokenize(source)

Transforms a source (string) into a list of Tokens.

token_utils.tokenizing.untokenize(tokens)

Returns a string built from tokens, or list of tokens.

This is similar to Python’s own tokenize.untokenize(), except that it preserves spacing between tokens, by using the line information recorded by Python’s tokenize.generate_tokens. As a result, if the original soure code had multiple spaces between some tokens or if escaped newlines were used or if tab characters were present in the original source, those will also be present in the source code produced by untokenize.

Thus source == untokenize(tokenize(source)).