Tutorial: recipes, tips, and tricks ===================================== When I started working with what would become token-utils, inside both **friendly/friendly-traceback** and **ideas**, the focus was on either analysing or modifying a few tokens at a time. As more complex cases were considered, I continued working on lists of tokens and developed functions which helped identified individual tokens (``get_first()``, ``get_last()``, ``get_significant_tokens()``, etc.) and their position in the list (``get_first_index()`` for the first "significant" token, etc.). I developed ad-hoc methods to identify entire statements, keeping track of open/close brackets, etc., all the while working with lists of tokens obtained more or less in an ad-hoc way. If you have come across token-utils before, and have followed a jupyter notebook tutorial that I wrote, you know what I mean ... .. sidebar:: Suggestions welcome If you can think of other useful examples, newer or better helper functions, **please** do not hesitate to file an issue. The current tutorial is based on a completely different approach, which leads to much more powerful and readable code. The key functions/classes I came up with, and which are described in this one-page tutorial are the following: - ``get_logical_lines()`` which I have introduced previously. There is also the slightly less useful ``get_physical_lines()`` which is more or less equivalent to the no longer existent ``get_lines()``. - My own implementation of ``pairwise()``, similar in spirit to ``itertools.pairwise()``. - My own implementation of ``sliding_window()``, similar to a recipe with the same name found in ``itertools`` as well as the third-party package ``more_itertools``. - ``BracketStack()``, which facilitates keeping track of matching set of brackets (``() {} []``). - ``IndentStack()``, which helps to keep track of statement indentation. - ``split_at_token()``, which is useful when changes are needed in two distant parts of a list of tokens. - ``add_operator()``, which is useful for exploring potential addition to Python's syntax and simplify the analysis as previously separate Python tokens can be combined into a single token. In addition, the ``Token`` class has well over 30 different methods which are very useful to identify types of tokens (and more) in a very readable way. In what follows, I briefly demonstrate how to use these functions, using examples of modifying Python's syntax. Adding a factorial symbol -------------------------- The factorial function is included in the math module. In usual mathematical notation, we would write something like:: n! == n * (n-1) * .... * 2 * 1 Let's see what tokens would represents a simple factorial:: >>> print_tokens("n!") Extracting physical lines from source: type=1 (NAME) string='n' start=(1, 0) end=(1, 1) line='n!' type=60 (ERRORTOKEN) string='!' start=(1, 1) end=(1, 2) line='n!' type=4 (NEWLINE) string='' start=(1, 2) end=(1, 3) line='n!' --> line: 'n!' type=0 (ENDMARKER) string='' start=(2, 0) end=(2, 0) line='' --> line: '' While there's nothing wrong with ``!`` being an ``ERRORTOKEN`` for us, let's nonetheless add it to the list of valid operators recognized by Python. When exploring potential changes to Python syntax, it can be really useful to add new operators to the tokenizer. .. code-block:: >>> add_operator("!", "exclamation") True >>> print_tokens("n!") Extracting physical lines from source: type=1 (NAME) string='n' start=(1, 0) end=(1, 1) line='n!' type=54 (OP: EXCLAMATION) string='!' start=(1, 1) end=(1, 2) line='n!' type=4 (NEWLINE) string='' start=(1, 2) end=(1, 3) line='n!' --> line: 'n!' type=0 (ENDMARKER) string='' start=(2, 0) end=(2, 0) line='' --> line: '' >>> tokens = tokenize("n!") >>> tokens[1] == "!" True >>> tokens[1].is_operator() True We can see that, not only ``!`` is a valid operator token, but it is also recognized as such by the ``Token`` method ``is_operator()``. The function ``add_operator()`` returned ``True``. If we were to add a string already known to be an operator, ``add_operator()`` would return ``False``, with no negative consequence. .. sidebar:: '!' in Python 3.12+ For Python 3.12+, if you use Python's ``tokenize`` module directly, you will see that ``!`` is already defined as an operator. While Python does not care about spaces between tokens, humans do. For the mathematically inclined, it wouldn't be appropriate to write something like:: a = 3 ! # <-- expecting ``a`` to be now equal to 6. So, to define a factorial, we will require ``!`` to come **immediately after** either a variable name (identifier) or an integer; by 'immediately after', we mean no space between the two tokens. Here is how we can do it, using token-utils' ``immediately_before()`` instead of ``immediately_after()`` as it looks more natural in this example. .. literalinclude:: ../../docs_examples/factorial_1.py Before we show it in action, we should add a word about ``pairwise``. token-utils has its own implementation of Python's ``itertools.pairwise`` function. Given an iterable denoted by ``ABCDEF``, ``itertools.pairwise`` will emit the items in pairs (tuples) like so:: AB BC CD DE EF Notice how the last item is never emitted as the first item of a pair, so it might require a special treatment if we want to do something about each individual token. To "solve this problem", token-utils adds a fake token (``$``) to the iterable list so that each token of the original sequence appears as a the first element of the pair:: AB BC CD DE EF F$ This way, as we loop through the tokens, we can focus on doing "something", and then add the first token to the stream. There is also an option to have this extra token at the beginning if it is more convenient to focus on the second element of a pair. As for the rest of the algorithm, we assume that is should be straightforward to understand given our description of the problem, and with the corresponding method names ``is_immediately_before()``, ``is_identifier()``, and ``is_integer()``. The only subtlety is when we modify ``next_.string``: we can do this as we haven't added this token to the ``new_tokens`` list. Here's a quick demonstration of our transformation:: >>> from factorial_1 import make_factorial >>> make_factorial("3!") 'factorial(3)' >>> make_factorial("a = 5!") 'a = factorial(5)' >>> make_factorial("4 !") # not immediately before/after '4 !' Better factorial and BracketStack ---------------------------------- Before we describe how to improve our factorial, we make a brief detour to introduce ``BracketStack``, a class that keeps track of whether or not brackets, ``(), {}, []``, are closed or unclosed in a stream. When a closing bracket is added **as a token-utils Token**, ``BracketStack`` will return a corresponding opening bracket if is is found, and will return ``False`` otherwise. Here's a sample session:: >>> stack = BracketStack() >>> for token in generate_tokens(" ({[ ]}) []"): ... if token.is_bracket(): ... possibly_open = stack.add(token) ... if possibly_open: ... print(possibly_open, token, stack.is_empty()) ... [ ] False { } False ( ) True [ ] True Going back to our factorial: in addition to appearing after integers or variable names, in mathematics, the factorial symbol, ``!``, can appear after a closing parenthesis, ``)``, as in ``(n + 1)!``, for example. We can modify our ``make_factorial()`` function to include such cases, again insisting that ``!`` had to appear immediately after a symbol, such as ``)``, in order to mean that it is meant to indicate that it is a factorial. In addition to keeping track of open/close brackets, we have one more difficulty to consider. Imagine that we have something like the following:: A B C D E ( a b c )! If we proceed to add tokens as we find them to a ``new_tokens`` list, when we arrive at ``)!``, we have to go backwards, retrieve all the previously added tokens (``a b c``), until we find the right open parenthesis, do the appropriate modification and add back the extracted material. This can actually be done easily using the function ``split_at_token()``. Here's a modified ``make_factorial()`` that uses this. .. literalinclude:: ../../docs_examples/factorial_2.py Inside ``split_at_token``, we make use of the ``is_identical()`` method to ensure that we find the correct ``(``. Note that we simply need to modify ``matching_bracket.string`` from ``(`` to ``factorial(``. Instead of simply showing the result of such transformations, let me instead show it after evaluation by Python using the **ideas** project I have already mentioned in which this is done as an example:: > ideas -a factorial The following initializing code has been executed in the Ideas console: from math import factorial Ideas Console version 0.3.12. [Python version: 3.11.9] ideas> 3! 6 ideas> 3 + (4 + 7 - 3)! 40323 ideas> n = 8 ideas> (n + 1)! 362880 .. admonition:: Test your understanding Modify the code to add the capability of having a `double factorial. `_ Define a new operator, ``!!``, so that you don't have to identify it as two consecutive ``!`` tokens but as a single token. .. note:: Unfortunately, using ``add_operator`` does not make it possible to choose any arbitrary string and have the python tokenizer recognize it as an operator. For example, trying to have the string ``.=`` as a new operator will not work: ``.`` will still be identified as a single token. Simlarly, trying to have a word, like ``plus`` will not work; however, something like ``|plus`` would work. Keeping track of indentation with IndentStack --------------------------------------------- In Python, the following keywords, or soft keywords, can signal the introduction of an indented block:: class, def, if/elif/else, for/else, try/except/else/finally, while/else, with, match, case For some code analysis or code modification, it might be very useful of keeping track of such keywords when they introduce a change in indentation. When they do so, and ignoring end of line comments, they will appear as the first non-space token on a **logical line** terminated by a colon ``:``. If we use token-utils' ``get_logical_lines()`` to analyze some source code, the last token of each such line will always be a ``NEWLINE`` token, preceded by a colon. This allowed us to define the utility class ``IndentStack`` which we will see in action shortly. .. tip:: When starting on a new project, I try to use some simple well-formatted code, with no comments (or semi-colons ``;`` !!) where each logical line corresponds to a physical line, with possibly some occasional empty lines to separate some statements. I then use ``get_physical_lines()``, do "something", and use ``untokenize()`` to see the final result. This allows me to more easily identify some potential problems. Later, when I want my code to be more robust, I use ``get_logical_lines()``, which can combine multiple physical lines into a single line, delete empty lines, etc., and often use ``stringify`` instead of ``untokenize`` for more predictable results. The example I will use is that of adding a decorator to functions, but not to methods of a class. The code to be transformed is the following: .. literalinclude:: ../../docs_examples/decorator_example.txt First, let's see how ``IndentStack()`` works, using the following program. .. literalinclude:: ../../docs_examples/view_stack.py The result is the following:: > py view_stack.py # decorator_exemple.txt .update() --> None ==> [] .update() --> None ==> [] def f1(): .update() --> def ==> [('def', (3, 0))] pass .update() --> None ==> [('def', (3, 0))] .update() --> None ==> [('def', (3, 0))] if True: .update() --> if ==> [('if', (6, 0))] def f2(): .update() --> def ==> [('if', (6, 0)), ('def', (7, 4))] pass .update() --> None ==> [('if', (6, 0)), ('def', (7, 4))] else: .update() --> if ==> [('if', (6, 0))] def f3(): .update() --> def ==> [('if', (6, 0)), ('def', (10, 4))] pass .update() --> None ==> [('if', (6, 0)), ('def', (10, 4))] .update() --> None ==> [('if', (6, 0)), ('def', (10, 4))] def f4(): .update() --> def ==> [('def', (13, 0))] pass .update() --> None ==> [('def', (13, 0))] .update() --> None ==> [('def', (13, 0))] class Something: .update() --> class ==> [('class', (16, 0))] def method(): .update() --> def ==> [('class', (16, 0)), ('def', (17, 4))] pass .update() --> None ==> [('class', (16, 0)), ('def', (17, 4))] .update() --> None ==> [('class', (16, 0)), ('def', (17, 4))] if ( .update() --> None ==> [] True .update() --> None ==> [] ): .update() --> None ==> [] pass .update() --> None ==> [] .update() --> None ==> [] As we can see, the last ``if`` block was not correctly identified as the line on which it appeared did not end in a colon. We could have situations, such as **inside** a comprehension, where the code is formatted so that an ``if`` or ``for`` appears as the first token on a physical line, and does not signal a change in indentation -- and the physical line does not end in a colon. Before trying to add a decorator, let's use ``get_logical_lines`` and use a method of ``IndentStack`` to see when we are inside a ``def`` or ``class`` block. .. literalinclude:: ../../docs_examples/inside_blocks.py The result is the following:: > py inside_blocks.py def f1(): inside_def=False; inside_class=False; stack: [('def', (3, 0))] pass inside_def=True; inside_class=False; stack: [('def', (3, 0))] if True: inside_def=False; inside_class=False; stack: [('if', (6, 0))] def f2(): inside_def=False; inside_class=False; stack: [('if', (6, 0)), ('def', (7, 4))] pass inside_def=True; inside_class=False; stack: [('if', (6, 0)), ('def', (7, 4))] else: inside_def=False; inside_class=False; stack: [('if', (6, 0))] def f3(): inside_def=False; inside_class=False; stack: [('if', (6, 0)), ('def', (10, 4))] pass inside_def=True; inside_class=False; stack: [('if', (6, 0)), ('def', (10, 4))] def f4(): inside_def=False; inside_class=False; stack: [('def', (13, 0))] pass inside_def=True; inside_class=False; stack: [('def', (13, 0))] class Something: inside_def=False; inside_class=False; stack: [('class', (16, 0))] def method(): inside_def=False; inside_class=True; stack: [('class', (16, 0)), ('def', (17, 4))] pass inside_def=True; inside_class=True; stack: [('class', (16, 0)), ('def', (17, 4))] if ( True): inside_def=False; inside_class=False; stack: [('if', (20, 0))] pass inside_def=False; inside_class=False; stack: [('if', (20, 0))] Note how the last ``if`` block is correctly identified, with three physical lines having been concatenated to form a single logical line. Adding a decorator ~~~~~~~~~~~~~~~~~~~ For our decorator example, we will want both ``f1, f2, f3, f4`` to be decorated, but not ``method``. Here's some code to do it, followed by a brief explanation and the result. .. literalinclude:: ../../docs_examples/add_decorator.py If a line starts with the word ``def`` and is not inside a ``class`` indented block, we add a line containing ``@decorator\n``, a simple string and not a list of tokens. This extra line is added with the same indentation as the function definition, which we obtained by doing ``" " * token.indentation()``. Here's the result of running this script:: > py add_decorator.py @decorator def f1(): pass if True: @decorator def f2(): pass else: @decorator def f3(): pass @decorator def f4(): pass class Something: def method(): pass if ( True): pass Adding a better keyword: nobreak ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Both ``for`` and ``while`` loops can use an ``else`` clause which seems to cause a bit of confusion when people see it for the first time. Raymond Hettinger once suggested that a better keyword to use instead might have been ``nobreak`` as it describes better that the code in this block is intended to be run if ``break`` did not end a loop. Suppose we want to use ``nobreak`` appropriately, transforming it into ``else`` prior to having it being executed by Python. Such an example is actually available in the **ideas** project I mentioned previously. First, here's the code we'll modify: .. literalinclude:: ../../docs_examples/nobreak.txt We **don't** want to be able to use ``nobreak`` in an ``if`` block nor in a ``try`` block, since we can't have a ``break`` statement. We include them in this example to demonstrate how we can selectively exclude them. The code we use is the following: .. literalinclude:: ../../docs_examples/nobreak.py Unlike the previous example, we use ``untokenize()`` to get a final result as this will include the comments in the output. Here's the result:: > py nobreak.py while True: pass else: # <-- else pass if True: pass elif something: pass nobreak: # keep nobreak error try: ... except: ... nobreak: # keep nobreak ... for i in range(3): ... else: # <-- else ... if something: while True: pass else: # <-- else done