Ideas: making it easier to extend Python’s syntax

Warning

As of August 18 2026, I’ve started updating this project after a 4 year-long hiatus. The version that can be installed via pypi (using pip) has not been updated yet.

Todo

Remove the above note when the project is uploaded to pypi.

Code on Github

You have an Excellent Idea ® to change the Python syntax and want to find a way to include your Excellent Idea ® in your Python programs. According to Python Developers Guide, this might be doable if you are willing to follow “a few steps” including:

1. Get a copy of the CPython’s code repository and all the required compilers for your platform.

2. Modify the grammar file to add rules for the new syntax.

3. Modify the AST generation code; this requires a knowledge of C

4. Compile the AST into bytecode

5. Recompile the modified Python interpreter

This … can be a rather daunting task. It might get a bit easier if you grab a copy of the currently unpublished book by Anthony Shaw, CPython Internals and read it from cover to cover, but it will still remain a major task. Furthermore, it would not be easy to share your work with others so that they can try it out.

However, there is a simpler way: it is possible to run code with a modified syntax using import hooks. [1]

Important

This entire project is meant as enabling easy experiments with changes to Python’s normal syntax. Some (most?) of the examples are definitely not reliable enough to be included in code used by anyone other than yourself.

Quick links to topics

• What is ideas?
  • What is an import hook?
  • Additional motivation for this project
  • Original motivation
  • Additional motivation
  • About the name
• Usage
  • Basic usage
  • Using the ideas-enabled interactive console
  • Using with IPython or Jupyter notebook/lab
  • Starting from a standard CPython interpreter
  • Using with Pypy
  • Advanced usage
  • Multiple import hooks
  • Always running by default
• Create your own import hook
  • How to do this
  • Actual code
  • API for function_keyword
  • Complete argument list for transform_source
• Improving function as a keyword
  • Building a complete example
• A deep dive
  • Using Ideas
  • Options to create a custom hook
  • About Decode

Many examples

• Guide to the many examples included
  • Source transformations
    • nobreak as a keyword
      • nobreak instead of else in if/else
      • What about try/except?
    • repeat as a keyword
      • Four different cases
    • French Python
      • Importing .pyfr files
    • French repeat
    • Auto-self
    • Decimal math
      • decimal_math.py
      • decimal_math_d.py
      • decimal_math_with.py
    • Fractional math (token)
    • Switch statement
      • This implementation
        • switch.py
    • Implicit multiplication
    • Unnormalized unicode
      • More to come…
    • PEP 843 implementation
      • Implementation
      • Star version
      • lazy keyword
      • export as identifier
      • Example
    • export as a soft keyword
  • AST transformations
    • Fractional math (AST)
  • AST creation
    • Polish expressions
  • Bytecode transformations
    • Confused math (Bytecode)
      • API for confused_math_bc
        • confused_math_bc.py
  • More complex examples
    • True constants
      • Consenting adults
      • Demonstration
      • How does it work
      • Suggestion for you
  • Tranforming a module after creation
    • Patching a module
  • Custom encoding
    • Create your own codec

Other modules

• About tokens
  • About Ideas’ tokens
  • token_utils.py API extracted by Sphinx
• Capture of token_utils interactive demo
  • Getting information
  • Changing information
  • Changing indentation
• import_hook.py
• console.py

To do

Todo

Try to create a different module object.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\examples\constants.rst, line 123.)

Todo

Write this up.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\examples\patching.rst, line 4.)

Todo

Write an example where we include different locations to search.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\possible.rst, line 49.)

Todo

Currently, only one AST example exists and only one AST transformation can be done. A second example should be created and find a way to ensure that both could be applied.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\guide.rst, line 57.)

Todo

Currently, only one Bytecode example exists and only one Bytecode transformation can be done. A second example should be created and find a way to ensure that both could be applied.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\guide.rst, line 81.)

Todo

Currently, a patch requires to specify a module name for it to be applied to that module. It might be worthwhile exploring a case where we use "*" to specify that the patch should be applied to all modules.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\guide.rst, line 104.)

Todo

Remove the above note when the project is uploaded to pypi.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\index.rst, line 10.)

Todo

Remove the above note when the project is uploaded to pypi.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\usage.rst, line 9.)

[1]

Or, in some cases as shown later, using a custom codec.