Ideas: making it easier to extend Python’s syntax

Code on Github

You have an Excellent Idea ® to change the Python syntax and want to find a way to include your Excellent Idea ® in your Python programs. According to Python Developers Guide, this might be doable if you are willing to follow “a few steps” including:

1. Get a copy of the CPython’s code repository and all the required compilers for your platform.

2. Modify the grammar file to add rules for the new syntax.

3. Modify the AST generation code; this requires a knowledge of C

4. Compile the AST into bytecode

5. Recompile the modified Python interpreter

This … can be a rather daunting task. It might get a bit easier if you grab a copy of the currently unpublished book by Anthony Shaw, CPython Internals and read it from cover to cover, but it will still remain a major task. Furthermore, it would not be easy to share your work with others so that they can try it out.

However, there is a simpler way: it is possible to run code with a modified syntax using import hooks [or, in some cases as shown later, using a custom codec.]

What is an import hook

Skipping over details.

This is a simplified description. A more detailed explanation will be given later.

When you write something like:

import my_module

Python’s import machinery has to do the following:

1. Try to use various tools to find the module requested

2. Get the source code of that module

3. Execute that source code, subject to some information reported in step 1.

An import hook is an additional tool that you create to do these three steps. Once written, you add it to sys.meta_path so that Python’s import machinery can make use of it.

Still, writing import hooks can be rather difficult.

[page 420] …it should be emphasized that Python’s module, package and import mechanism is one of the most complicated parts of the entire language – often poorly understood by even the most seasoned Python programmers unless they’ve devoted effort to peeling back the covers.

… long discussion …

[page 428] Assuming that your head hasn’t completely exploded at this point, … Last, but not least, spending some time sleeping with PEP 302 and the documentation for importlib under your pillow may be advisable.

Python Cookbook, 3rd edition, by David Beazley and Brian K. Jones

ideas is designed to facilitate the creation of such import hooks, and be a repository for examples that can be used as starting points for new ideas.

Instead of figuring out how to write an import hook, using ideas you can focus exclusively on what what might be needed to convert your proposed new syntax into something that Python can understand – ideas will take care of the rest, including inserting it in sys.meta_path.

Warning

Doing something like what is described in this documentation is not recommended for production code.

But it can be fun! ;-)

Quick links to topics

• Additional motivation
• Usage
• Create your own import hook
• A deep dive
• Guide to the examples

Source transformations

• nobreak as a keyword
• repeat as a keyword
• French Python
• French repeat
• Auto-self
• Decimal math
• Fractional math (token)
• Switch statement
• Implicit multiplication
• Unnormalized unicode

AST transformations

• Fractional math (AST)

Bytecode transformations

• Confused math (Bytecode)

More complex examples

• True constants
• Examples never included - first draft

Custom encodings

• Create your own codec

Other modules

• About tokens
• import_hook.py
• console.py

To do

Todo

Try to create a different module object.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\constants.rst, line 123.)

Todo

Add these examples

I need to add some explanations and provide links to the following examples:

• importing “best answer” from StackOverflow

• installing module from pypi on request (David Beazley’s talk)

etc.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\excluded.rst, line 26.)