Ideas: making it easier to extend Python’s syntax

Code on Github

You have an Excellent Idea ® to change the Python syntax and want to find a way to include your Excellent Idea ® in your Python programs. According to Python Developers Guide, this might be doable if you are willing to follow “a few steps” including:

1. Get a copy of the CPython’s code repository and all the required compilers for your platform.

2. Modify the grammar file to add rules for the new syntax.

3. Modify the AST generation code; this requires a knowledge of C

4. Compile the AST into bytecode

5. Recompile the modified Python interpreter

This … can be a rather daunting task. It might get a bit easier if you grab a copy of the currently unpublished book by Anthony Shaw, CPython Internals and read it from cover to cover, but it will still remain a major task. Furthermore, it would not be easy to share your work with others so that they can try it out.

However, there is a simpler way: it is possible to run code with a modified syntax using import hooks. [1]

Important

This entire project is meant as enabling easy experiments with changes to Python’s normal syntax. Some (most?) of the examples are definitely not reliable enough to be included in code used by anyone other than yourself.

Quick links to topics

• What is ideas?
  • What is an import hook?
  • Additional motivation for this project
  • Original motivation
  • Additional motivation
  • About the name
• Usage
  • Basic usage
  • Using the ideas-enabled interactive console
  • Using with IPython or Jupyter notebook/lab
  • Using with Pypy
  • Advanced usage
  • Multiple import hooks
  • Always running by default
• Create your own import hook
  • How to do this
  • Actual code
  • API for function_keyword
  • Complete argument list for transform_source
• Improving function as a keyword
  • Building a complete example
• A deep dive
  • Using Ideas
  • Options to create a custom hook
  • About Decode

Many examples

• Guide to the many examples included
  • Source transformations
    • nobreak as a keyword
      • nobreak instead of else in if/else
      • What about try/except?
    • repeat as a keyword
      • Four different cases
    • French Python
      • Importing .pyfr files
    • French repeat
    • Auto-self
    • Decimal math
      • decimal_math.py
      • decimal_math_d.py
      • decimal_math_with.py
    • Fractional math (token)
    • Switch statement
      • This implementation
        • switch.py
    • Implicit multiplication
    • Unnormalized unicode
      • More to come…
    • from-export: PEP 843
      • Example
      • Proposed implementation
      • Actual implementation
      • export as an identifier
    • export-name: PEP 842
      • A restricted dir
      • Implementation
    • export as a (soft) keyword (PEPs 842 + 843)
      • Example
      • Last words
  • AST transformations
    • Fractional math (AST)
  • AST creation
    • Polish expressions
  • Bytecode transformations
    • Confused math (Bytecode)
      • API for confused_math_bc
        • confused_math_bc.py
  • More complex examples
    • True constants
      • Consenting adults
      • Demonstration
      • How does it work
      • Suggestion for you
  • Tranforming a module after creation
    • Patching a module
  • Custom encoding
    • Create your own codec

Other modules

• About tokens
  • About Ideas’ tokens
  • token_utils.py API extracted by Sphinx
• Capture of token_utils interactive demo
  • Getting information
  • Changing information
  • Changing indentation
• import_hook.py
• console.py

To do

Todo

Currently, only one AST example exists and only one AST transformation can be done. A second example should be created and find a way to ensure that both could be applied.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\guide.rst, line 58.)

Todo

Currently, only one Bytecode example exists and only one Bytecode transformation can be done. A second example should be created and find a way to ensure that both could be applied.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\guide.rst, line 82.)

Todo

Currently, a patch requires to specify a module name for it to be applied to that module. It might be worthwhile exploring a case where we use "*" to specify that the patch should be applied to all modules.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\guide.rst, line 105.)

Todo

Try to create a different module object.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\included\constants.rst, line 123.)

Todo

Write this up.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\included\patching.rst, line 4.)

Todo

Write an example where we include different locations to search.

(The original entry is located in C:\Users\Andre\github\ideas\docs\source\possible.rst, line 49.)

[1]

Or, in some cases as shown later, using a custom codec.