About tokens and token-utils
=============================

About tokens
------------

Using token-utils requires knowing what "tokens" produced
by Python's tokenize module are.
An outstanding tutorial about this module is
`Brown Water Python <https://www.asmeurer.com/brown-water-python/>`_
written by Aaron Meurer.

Origin of token-utils
---------------------

In our description of these tools below, we assume that you are
somewhat familiar with the concept of token objects generated by Python's
tokenize module.  If you are not familiar with those, we suggest that
you read through at least once through the documentation about Python's
tokenize module mentioned above.


.. Admonition:: An excellent tutorial

  `Brown Water Python <https://www.asmeurer.com/brown-water-python/>`_, written by Aaron Meurer,
   is an excellent and very comprehensive tutorial about Python's
  tokenize module.




The main points to understand:

- Using the ``tokenize`` function, a source can be broken down in tokens,
  which, as generated by Python, are 5-tuples carrying information about their
  **type**, their **string** content, their position in the source
  (identified by starting and ending **row**, aka line number, and **column**),
  as well as the content of the line where they are found.
- From a list of tokens, the original source can essentially recreated
  by using the ``untokenize`` function.
  However, as stated in the documentation:

    *The result is guaranteed to tokenize back to match the input so that
    the conversion is lossless and round-trips are assured.
    The guarantee applies only to the token type and
    token string as the spacing between tokens (column positions) may change.*

- To ``untokenize`` using the function from the Python
  standard library, one can use either a list of 5-tuple tokens,
  or a list of two-tuple tokens that include only the **type** and **string**
  information.

.. sidebar:: Perfect round-trip

    Unlike Python's version, the process of tokenizing and untokenizing a source
    using ideas' own ``tokenize`` and ``untokenize`` functions
    is guaranteed to yield back an exact copy of the original source, with all
    the spacing information intact.
    Experience has shown that being able to recover the
    original source with spacing included is **extremely** useful when writing
    tests about the expected results for some source transformation.



.. tip::

    While we show below the full API of the `token_utils` module,
    you might want to first to to next page to see a demonstration
    of its usage, done in an actual programming session using a Jupyter notebook.