Tutorial: recipes, tips, and tricks
When I started working with what would become token-utils, inside both
friendly/friendly-traceback and ideas, the focus was on either
analysing or modifying a few tokens at a time. As more complex cases
were considered, I continued working on lists of tokens and developed
functions which helped identified individual tokens (get_first(),
get_last(), get_significant_tokens(), etc.) and their
position in the list (get_first_index() for the first “significant” token,
etc.). I developed ad-hoc methods to identify entire statements, keeping
track of open/close brackets, etc., all the while working with lists
of tokens obtained more or less in an ad-hoc way. If you have come
across token-utils before, and have followed a jupyter notebook
tutorial that I wrote, you know what I mean …
The current tutorial is based on a completely different approach, which leads to much more powerful and readable code. The key functions/classes I came up with, and which are described in this one-page tutorial are the following:
get_logical_lines()which I have introduced previously. There is also the slightly less usefulget_physical_lines()which is more or less equivalent to the no longer existentget_lines().My own implementation of
pairwise(), similar in spirit toitertools.pairwise().My own implementation of
sliding_window(), similar to a recipe with the same name found initertoolsas well as the third-party packagemore_itertools.BracketStack(), which facilitates keeping track of matching set of brackets (() {} []).IndentStack(), which helps to keep track of statement indentation.split_at_token(), which is useful when changes are needed in two distant parts of a list of tokens.add_operator(), which is useful for exploring potential addition to Python’s syntax and simplify the analysis as previously separate Python tokens can be combined into a single token.
In addition, the Token class has well over 30 different methods which are very useful
to identify types of tokens (and more) in a very readable way.
In what follows, I briefly demonstrate how to use these functions, using examples of modifying Python’s syntax.
Adding a factorial symbol
The factorial function is included in the math module. In usual mathematical notation, we would write something like:
n! == n * (n-1) * .... * 2 * 1
Let’s see what tokens would represents a simple factorial:
>>> print_tokens("n!")
Extracting physical lines from source:
type=1 (NAME) string='n' start=(1, 0) end=(1, 1) line='n!'
type=60 (ERRORTOKEN) string='!' start=(1, 1) end=(1, 2) line='n!'
type=4 (NEWLINE) string='' start=(1, 2) end=(1, 3) line='n!'
--> line: 'n!'
type=0 (ENDMARKER) string='' start=(2, 0) end=(2, 0) line=''
--> line: ''
While there’s nothing wrong with ! being an ERRORTOKEN for us,
let’s nonetheless add it to the list of valid operators recognized by
Python. When exploring potential changes to Python syntax, it can
be really useful to add new operators to the tokenizer.
>>> add_operator("!", "exclamation")
True
>>> print_tokens("n!")
Extracting physical lines from source:
type=1 (NAME) string='n' start=(1, 0) end=(1, 1) line='n!'
type=54 (OP: EXCLAMATION) string='!' start=(1, 1) end=(1, 2) line='n!'
type=4 (NEWLINE) string='' start=(1, 2) end=(1, 3) line='n!'
--> line: 'n!'
type=0 (ENDMARKER) string='' start=(2, 0) end=(2, 0) line=''
--> line: ''
>>> tokens = tokenize("n!")
>>> tokens[1] == "!"
True
>>> tokens[1].is_operator()
True
We can see that, not only ! is a valid operator token, but it is also
recognized as such by the Token method is_operator().
The function add_operator() returned True. If we were to add
a string already known to be an operator, add_operator()
would return False, with no negative consequence.
While Python does not care about spaces between tokens, humans do. For the mathematically inclined, it wouldn’t be appropriate to write something like:
a = 3 ! # <--
expecting a to be now equal to 6. So, to define a factorial, we will
require ! to come immediately after either a variable name (identifier)
or an integer; by ‘immediately after’, we mean no space between the two
tokens.
Here is how we can do it, using token-utils’ immediately_before() instead
of immediately_after() as it looks more natural in this example.
# factorial_1.py
from token_utils import tokenize, untokenize, add_operator, pairwise
add_operator("!", "exclamation")
def make_factorial(source):
tokens = tokenize(source)
new_tokens = []
for token, next_ in pairwise(tokens):
if (
next_ == "!"
and token.is_immediately_before(next_)
and (token.is_integer() or token.is_identifier())
):
token.string = f"factorial({token.string})"
next_.string = ""
new_tokens.append(token)
return untokenize(new_tokens)
Before we show it in action, we should add a word about pairwise.
token-utils has its own implementation of Python’s itertools.pairwise function.
Given an iterable denoted by ABCDEF,
itertools.pairwise will emit the items in pairs (tuples) like so:
AB BC CD DE EF
Notice how the last item is never emitted as the first item of a pair, so
it might require a special treatment if we want to do something
about each individual token. To “solve this problem”,
token-utils adds a fake token ($) to the iterable list so that each
token of the original sequence appears as a the first element of the pair:
AB BC CD DE EF F$
This way, as we loop through the tokens, we can focus on doing “something”, and then add the first token to the stream. There is also an option to have this extra token at the beginning if it is more convenient to focus on the second element of a pair.
As for the rest of the algorithm, we assume that is should be straightforward
to understand given our description of
the problem, and with the corresponding method names
is_immediately_before(), is_identifier(), and is_integer().
The only subtlety is when we modify next_.string: we can do this as
we haven’t added this token to the new_tokens list.
Here’s a quick demonstration of our transformation:
>>> from factorial_1 import make_factorial
>>> make_factorial("3!")
'factorial(3)'
>>> make_factorial("a = 5!")
'a = factorial(5)'
>>> make_factorial("4 !") # not immediately before/after
'4 !'
Better factorial and BracketStack
Before we describe how to improve our factorial, we make a
brief detour to introduce BracketStack, a class that
keeps track of whether or not brackets, (), {}, [],
are closed or unclosed in a stream. When a closing bracket is added
as a token-utils Token, BracketStack will return
a corresponding opening bracket if is is found, and will
return False otherwise.
Here’s a sample session:
>>> stack = BracketStack()
>>> for token in generate_tokens(" ({[ ]}) []"):
... if token.is_bracket():
... possibly_open = stack.add(token)
... if possibly_open:
... print(possibly_open, token, stack.is_empty())
...
[ ] False
{ } False
( ) True
[ ] True
Going back to our factorial: in addition to appearing after
integers or variable names, in mathematics, the factorial symbol, !,
can appear after a closing parenthesis, ),
as in (n + 1)!, for example.
We can modify our make_factorial() function to include
such cases, again insisting that ! had to appear
immediately after a symbol, such as ), in order to
mean that it is meant to indicate that it is a factorial.
In addition to keeping track of open/close brackets, we have one more difficulty to consider. Imagine that we have something like the following:
A B C D E ( a b c )!
If we proceed to add tokens as we find them to a new_tokens list,
when we arrive at )!, we have to go backwards, retrieve
all the previously added tokens (a b c), until we find the
right open parenthesis, do the appropriate modification and add back
the extracted material. This can actually be done easily using
the function split_at_token().
Here’s a modified make_factorial() that uses this.
# factorial_2.py
from token_utils import (
tokenize,
untokenize,
add_operator,
pairwise,
BracketStack,
split_at_token,
)
add_operator("!", "exclamation")
def make_factorial(source):
tokens = tokenize(source)
stack = BracketStack()
new_tokens = []
for token, next_ in pairwise(tokens):
if (
next_ == "!"
and token.is_immediately_before(next_)
and (token.is_integer() or token.is_identifier())
):
token.string = f"factorial({token.string})"
next_.string = ""
elif next_ == "!" and token.is_immediately_before(next_) and token == ")":
matching_bracket = stack.add(token)
next_.string = ""
new_tokens, remainder = split_at_token(new_tokens, matching_bracket)
matching_bracket.string = "factorial("
new_tokens.append(matching_bracket)
new_tokens.extend(remainder)
else:
if token.is_bracket():
stack.add(token)
new_tokens.append(token)
return untokenize(new_tokens)
Inside split_at_token, we make use of the is_identical() method
to ensure that we find the correct (. Note that we simply need to
modify matching_bracket.string from ( to factorial(.
Instead of simply showing the result of such transformations,
let me instead show it after evaluation by Python using
the ideas project I have already mentioned in which
this is done as an example:
> ideas -a factorial
The following initializing code has been executed in the Ideas console:
from math import factorial
Ideas Console version 0.3.12. [Python version: 3.11.9]
ideas> 3!
6
ideas> 3 + (4 + 7 - 3)!
40323
ideas> n = 8
ideas> (n + 1)!
362880
Test your understanding
Modify the code to add the capability of having a
double factorial.
Define a new operator, !!, so that you don’t have to identify it
as two consecutive ! tokens but as a single token.
Note
Unfortunately, using add_operator does not make it possible to choose
any arbitrary string and have the python tokenizer recognize it as an operator.
For example, trying to have the string .= as a new operator will not work:
. will still be identified as a single token. Simlarly, trying to have a word,
like plus will not work; however, something like |plus would work.
Keeping track of indentation with IndentStack
In Python, the following keywords, or soft keywords, can signal the introduction of an indented block:
class, def, if/elif/else, for/else, try/except/else/finally, while/else, with, match, case
For some code analysis or code modification, it might be very useful of keeping track
of such keywords when they introduce a change in indentation. When they do so, and ignoring
end of line comments, they will appear as the first non-space token
on a logical line terminated by a colon :.
If we use token-utils’ get_logical_lines() to analyze some source code, the last token
of each such line will always be a NEWLINE token, preceded by a colon.
This allowed us to define the utility class IndentStack which we will see in action shortly.
Tip
When starting on a new project, I try to use some simple well-formatted code,
with no comments (or semi-colons ; !!) where each logical line corresponds to a physical line,
with possibly some occasional empty lines to separate some statements.
I then use get_physical_lines(), do “something”, and use untokenize() to see
the final result. This allows me to more easily identify some potential problems.
Later, when I want my code to be more robust, I use get_logical_lines(),
which can combine multiple physical lines into a single line, delete empty lines, etc.,
and often use stringify instead of untokenize for more predictable results.
The example I will use is that of adding a decorator to functions, but not to methods of a class. The code to be transformed is the following:
# decorator_exemple.txt
def f1():
pass
if True:
def f2():
pass
else:
def f3():
pass
def f4():
pass
class Something:
def method():
pass
if (
True
):
pass
First, let’s see how IndentStack() works, using the following program.
# view_stack.py
from token_utils import get_physical_lines, IndentStack, untokenize
with open("decorator_example.txt", "r") as f:
source = f.read()
stack = IndentStack()
# It is highly preferable to use get_logical_lines as this example shows.
for line in get_physical_lines(source):
print(
f"{untokenize(line).rstrip():<25}",
f".update() --> {str(stack.update(line)):<7}",
"==> ",
end=" ",
)
stack.print_stack()
The result is the following:
> py view_stack.py
# decorator_exemple.txt .update() --> None ==> []
.update() --> None ==> []
def f1(): .update() --> def ==> [('def', (3, 0))]
pass .update() --> None ==> [('def', (3, 0))]
.update() --> None ==> [('def', (3, 0))]
if True: .update() --> if ==> [('if', (6, 0))]
def f2(): .update() --> def ==> [('if', (6, 0)), ('def', (7, 4))]
pass .update() --> None ==> [('if', (6, 0)), ('def', (7, 4))]
else: .update() --> if ==> [('if', (6, 0))]
def f3(): .update() --> def ==> [('if', (6, 0)), ('def', (10, 4))]
pass .update() --> None ==> [('if', (6, 0)), ('def', (10, 4))]
.update() --> None ==> [('if', (6, 0)), ('def', (10, 4))]
def f4(): .update() --> def ==> [('def', (13, 0))]
pass .update() --> None ==> [('def', (13, 0))]
.update() --> None ==> [('def', (13, 0))]
class Something: .update() --> class ==> [('class', (16, 0))]
def method(): .update() --> def ==> [('class', (16, 0)), ('def', (17, 4))]
pass .update() --> None ==> [('class', (16, 0)), ('def', (17, 4))]
.update() --> None ==> [('class', (16, 0)), ('def', (17, 4))]
if ( .update() --> None ==> []
True .update() --> None ==> []
): .update() --> None ==> []
pass .update() --> None ==> []
.update() --> None ==> []
As we can see, the last if block was not correctly identified as the line on which it
appeared did not end in a colon. We could have situations, such as inside a comprehension,
where the code is formatted so that an if or for appears as the first token on a
physical line, and does not signal a change in indentation – and the physical line does not end
in a colon.
Before trying to add a decorator, let’s use get_logical_lines and use a method
of IndentStack to see when we are inside a def or class block.
# inside_blocks.py
from token_utils import get_logical_lines, IndentStack, stringify
with open("decorator_example.txt", "r") as f:
source = f.read()
stack = IndentStack()
for line in get_logical_lines(source):
stack.update(line)
first_token = line[0]
inside_class = stack.is_token_in_named_block(first_token, "class")
inside_def = stack.is_token_in_named_block(first_token, "def")
print(
f"{stringify(line).rstrip():20}",
f"{inside_def=};",
f"{inside_class=};",
" stack:",
end=" ",
)
stack.print_stack()
The result is the following:
> py inside_blocks.py
def f1(): inside_def=False; inside_class=False; stack: [('def', (3, 0))]
pass inside_def=True; inside_class=False; stack: [('def', (3, 0))]
if True: inside_def=False; inside_class=False; stack: [('if', (6, 0))]
def f2(): inside_def=False; inside_class=False; stack: [('if', (6, 0)), ('def', (7, 4))]
pass inside_def=True; inside_class=False; stack: [('if', (6, 0)), ('def', (7, 4))]
else: inside_def=False; inside_class=False; stack: [('if', (6, 0))]
def f3(): inside_def=False; inside_class=False; stack: [('if', (6, 0)), ('def', (10, 4))]
pass inside_def=True; inside_class=False; stack: [('if', (6, 0)), ('def', (10, 4))]
def f4(): inside_def=False; inside_class=False; stack: [('def', (13, 0))]
pass inside_def=True; inside_class=False; stack: [('def', (13, 0))]
class Something: inside_def=False; inside_class=False; stack: [('class', (16, 0))]
def method(): inside_def=False; inside_class=True; stack: [('class', (16, 0)), ('def', (17, 4))]
pass inside_def=True; inside_class=True; stack: [('class', (16, 0)), ('def', (17, 4))]
if ( True): inside_def=False; inside_class=False; stack: [('if', (20, 0))]
pass inside_def=False; inside_class=False; stack: [('if', (20, 0))]
Note how the last if block is correctly identified, with three physical lines having
been concatenated to form a single logical line.
Adding a decorator
For our decorator example, we will want both f1, f2, f3, f4 to be decorated, but not method.
Here’s some code to do it, followed by a brief explanation and the result.
# add_decorator.py
from token_utils import get_logical_lines, IndentStack, untokenize, stringify
with open("decorator_example.txt", "r") as f:
source = f.read()
stack = IndentStack()
new_lines = []
for line in get_logical_lines(source):
stack.update(line)
first_token = line[0]
if first_token == "def" and not stack.is_token_in_named_block(first_token, "class"):
new_lines.append([" " * first_token.indentation() + "@decorator\n"])
new_lines.append(line)
print(stringify(new_lines))
If a line starts with the word def and is not inside a class indented block,
we add a line containing @decorator\n, a simple string and not a list of tokens.
This extra line is added with the same indentation as the function definition,
which we obtained by doing " " * token.indentation(). Here’s the result of running
this script:
> py add_decorator.py
@decorator
def f1():
pass
if True:
@decorator
def f2():
pass
else:
@decorator
def f3():
pass
@decorator
def f4():
pass
class Something:
def method():
pass
if ( True):
pass
Adding a better keyword: nobreak
Both for and while loops can use an else clause which seems to
cause a bit of confusion when people see it for the first time.
Raymond Hettinger once suggested that a better keyword to use
instead might have been nobreak as it describes better that the
code in this block is intended to be run if break did not
end a loop.
Suppose we want to use nobreak appropriately, transforming it
into else prior to having it being executed by Python.
Such an example is actually available in the ideas project
I mentioned previously.
First, here’s the code we’ll modify:
# nobreak.txt
while True:
pass
nobreak: # <-- else
pass
if True:
pass
elif something:
pass
nobreak: # keep nobreak
error
try:
...
except:
...
nobreak: # keep nobreak
...
for i in range(3):
...
nobreak: # <-- else
...
if something:
while True:
pass
nobreak: # <-- else
done
We don’t want to be able to use nobreak in an if block
nor in a try block, since we can’t have a break statement.
We include them in this example to demonstrate how we can
selectively exclude them.
The code we use is the following:
# nobreak.py
from token_utils import get_logical_lines, untokenize, IndentStack
with open("nobreak.txt", "r") as f:
source = f.read()
new_lines = []
stack = IndentStack()
stack.add_same_indent_keyword("nobreak")
for line in get_logical_lines(source):
top = stack.update(line)
if not top: # this line does not introduce an indented block
new_lines.append(line)
continue
if line[0] == "nobreak" and top.is_in(["for", "while"]):
line[0].string = "else" # modify in place
new_lines.append(line)
print(untokenize(new_lines))
Unlike the previous example, we use untokenize() to get a final
result as this will include the comments in the output.
Here’s the result:
> py nobreak.py
while True:
pass
else: # <-- else
pass
if True:
pass
elif something:
pass
nobreak: # keep nobreak
error
try:
...
except:
...
nobreak: # keep nobreak
...
for i in range(3):
...
else: # <-- else
...
if something:
while True:
pass
else: # <-- else
done