Other modules

In addition to invocation.py already mentioned, and excluding modules defining individual dialects, the following are included:

console

Console adapted from Python’s console found in code.py.

An earlier version of this code was subclassing code.InteractiveConsole. However, as more and more changes were introduced, dealing with code transformation and especially customized error handling, it seemed to make sense to “rewrite” every relevant part in this single module.

class avantpy.console.AvantPyInteractiveConsole(locals=None)

A Python console that tries to emulate the normal Python interpreter except that it support experimental code transformations. It is adapted from cPython’s code.InteractiveConsole and its parent.

Like the normal Python interactive console, it attempts to evaluate code entered one line at a time by a user.

push(line)

Pushes a transformed line to the interpreter.

The line should not have a trailing newline; it may have internal newlines. The line is transformed and appended to a buffer. The interpreter’s run_source() method is called with the concatenated contents of the buffer as source. If this indicates that the command was executed or invalid, the buffer is reset; otherwise, the command is incomplete, and the buffer is left as it was after the line was appended. The return value is True if more input is required, False if the line was dealt with in some way (this is the same as run_source()).

runsource(source, filename='<input>', symbol='single')

Compile and run some source in the interpreter.

Arguments are as for compile_command().

One several things can happen:

1) The input is incorrect; compile_command() raised an exception (SyntaxError or OverflowError). A syntax traceback will be printed by calling the showsyntaxerror() method.

2) The input is incomplete, and more input is required; compile_command() returned None. Nothing happens.

3) The input is complete; compile_command() returned a code object. The code is executed by calling self.runcode() (which also handles run-time exceptions, except for SystemExit).

The return value is True in case 2, False in the other cases (unless an exception is raised). The return value can be used to decide whether to use sys.ps1 or sys.ps2 to prompt the next line.

avantpy.console.start_console(local_vars=None)

Starts a console; modified from code.interact

converter

This module contains all the necessary functions, classes and methods to take a file written in a given dialect and convert it into Python. In doing so, it can identify some mistakes and raise some custom exceptions which can be used to inform users in a way that is less intimidating than the usual Python tracebacks.

The following can definitely be skipped unless you are interested in (most of) the gory details.

AvantPy uses Python’s tokenize module to convert a source into a sequence of tokens. A token might be the name of a variable, an operator, a parenthesis, a string, etc.

AvantPy analyses these tokens, replacing some written into a different dialect until all are converted into standard Python tokens. Then, these are recombined into a string which is the source to be executed.

Python’s tokenize module includes a function, called untokenize, which can be used to combine a series of tokens into a valid program. With a normal Python program, doing something similar to:

new_source = untokenize(tokenize(source))

would be such that executing new_source would be the same as executing source. However, the spacing between tokens would not necessarily be the same for both new_source and the original program. For example, the original program may include a line like:

variable = function( argument )

which might be converted into:

variable =function(argument)

This could make it more difficult to compare the original code with the converted one, as it is possible to do using one of the utilises provided with AvantPy, or any “diff” program. As a result, we do not use Python’s untokenize function, and explicitly keep track of spacing between tokens.

To understand how the conversion process works, it is useful to review all possible cases, from some of the most complex, ending with the simplest ones.

1. nobreak

AvantPy has an additional keyword, named nobreak in the English dialect, which can be used in for or while loops instead of the standard else, as in:

while condition:
    # code
nobreak:
    # code

However, nobreak cannot be used in an if/elif/else blocks to replace else.

Furthermore, nobreak cannot be used instead of else in an assignment such as:

a = 1 if True else 2

To identify if a program includes a nobreak keyword mistakenly, every time we see a leading for, while, if or elif keyword (or their equivalent in a given dialect), we note the indentation (column where the first character is written) and the corresponding keyword. A list containing these keywords is called blocks_with_else in this function.

Later, when we encounter a nobreak keyword at a given indentation, we check to see if the last blocks_with_else keyword found at that same indentation was one for which it made sense to use nobreak or not. If it was a loop, we simply replace nobreak by else. If not, we raise a custom exception which is handled elsewhere.

2. repeat

In addition to the standard Python loops constructs, AvantPy support four additional idioms:

repeat forever:           # while True:
    pass
repeat while condition:   # while condition:
    pass
repeat until condition:   # while not condition:
    pass
repeat n:                 # for some_var in range(n):
    pass

For this last case, n could be an expression, possibly spanning multiple lines.

When we encounter the equivalent to the “repeat” keyword in the selected dialect, we must make sure that it is the first keyword occurring on a logical line; if not, we raise a custom exception.

If repeat is the first keyword on a line, we set a flag (repeat_loop) to True, preparing to look at the next token.

a) If the next token is one of forever, until, while, or their equivalent in the target dialect (remember that including normal Python keywords in a program written in a different dialect is allowed) we can proceed with the rest in a straightforward manner.

b) if that is not the case, we set a different flag (repeat_n) to True so that we can deal with the relevant for loop.

For this last case, the variable in the for loop is a dummy variable; we must ensure that its name is chosen such that it does not occur anywhere else in the source code. This is accomplished using a method called get_unique_variable_names.

3. nobreak and repeat

A repeat loop is essentially a for or a while loop. As such, it could have an else clause which has a clearer meaning if the keyword nobreak is used instead. Thus, just like we mentioned before, we also keep track of where a leading repeat is used.

4. Direct translation

If a token does not match one of the cases described above, we need to see if it is a term used in the dialect; if so, we simply translate it into standard Python.

5. Brackets

In Python, brackets must always come in pairs: (…), […], {…}. In the course of processing the file, if we identify brackets which are not paired correctly, an exception is raised.

6. Remaining tokens

Any remaining token is left as is; it is assumed to be valid Python.

class avantpy.converter.Converter(source, dialect=None, filename=None)

This class uses Python’s tokenize module to convert a source into a sequence of tokens. A token might be the name of a variable, an operator, a parenthesis, a string, etc.

It processes one token at a time, identifying potential misuses, replacing some tokens written into a different dialect until all are converted into standard Python tokens. Then, these are recombined into a string which is the source to be executed.

Python’s tokenize module includes a function, called untokenize, which can be used to combine a series of tokens into a valid program. With a normal Python program, doing something similar to:

new_source = untokenize(tokenize(source))

would be such that executing new_source would be the same as executing source. However, the spacing between tokens would not necessarily be the same for both new_source and the original program. For example, the original program may include a line like:

variable = function( argument )

which might be converted into:

variable =function(argument)

This could make it more difficult to compare the original code with the converted one, as it is possible to do using one of the utilises provided with AvantPy, or any “diff” program. As a result, we do not use Python’s untokenize function, and explicitly keep track of spacing between tokens.

convert()

Uses Python’s tokenize module to generate tokens from a source.

It calls a function to process each token in turn, and combine the results to returned a transformed source.

do_close_bracket(token)

Determines if a closing bracket ), ], or }, matches a previously opened one, and raises an error if that is not the case.

do_until_forever_error(token)

This function is called when we see a keyword equivalent to until or forever in a given dialect without them being preceeded by the keyword repeat. This is not allowed.

handle_nobreak_error(token)

Deals with identified misuses of the nobreak keyword.

init_bookkeeping_vars()

Initialises variables that are used for bookkeeping

init_dialect_vars()

Initialises variables that are dependent on a given dialect.

preserve_repeat_spacing(token)

We ensure spacing of original file is preserved, including space between tokens. When self.just_processed_repeat_kwd is True, it means we just saw the repeat keyword which can simply be dropped; we don’t need add the space between “repeat” and the next token.

process_after_repeat(token)

After we see the repeat keyword, there are four possibilities:

1. the keyword until follows

2. the keyword until follows

3. the keyword forever follows

4. it is meant to be followed by some_expression which evaluates to an integer.

process_nobreak(token)

The nobreak keyword is allowed to replace else in a while or for loop only.

process_repeat(token)

This identifies if repeat begins a new line; if so, we simply set up a flag to indicate that we just saw this keyword, and otherwise drop it.

If it does not begin a new line, we raise an exception.

process_token(token)

Determines what to do for each individual token.

raise_missing_repeat_colon(linenumber, last_col)

Raised if a repeat statement does not end with a colon on the same line with no other colon appearing on that line.

avantpy.converter.convert(source, dialect=None, filename=None)

Normally used from other functions to call for a conversion of a Python source from a given dialect into Python.

avantpy.converter.get_unique_variable_names(source, repeat_kwd, all_count_names=[])

Returns a list of possible variables names that are not found in the original source and have not been used anywhere. These variables will be used in expressions such as

for _COUNT_42 in range(n):

which will replace

repeat n:

avantpy.converter.transcode(source, from_dialect, to_dialect)

Transforms a source (string) written in a known dialect into an equivalent version written in a different dialect. Spacing between tokens is preserved. This means that if a source is transformed from dialect xx to dialect yy, and that the result is transformed back to dialect xx, the original source should be recovered.

Returns either the transformed source as a string, or None if something prevented the transformation to be successful.

avantpy.converter.transcode_file(from_path, to_path)

Convenience function to be used from the command line.

exception_handling

The goal of exception_handling.py is to display information about exceptions being raised in a way that is much easier to understand for beginners than normal Python tracebacks. These “friendly” tracebacks are written so that they can easily be translated into any human language.

avantpy.exception_handling.disable()

Disables custom exception handling

avantpy.exception_handling.extract_filename_linenumber(exc)

Extracting filename and linenumber where an error occurred based on information from traceback.

avantpy.exception_handling.get_partial_source(source, begin, end, marks=None, nb_digits=None)

Extracts a few relevant lines from a source file.

If the number of lines would exceed a certain limit (currently 7) the function is called recursively to only show the a few lines at the beginning and at the end.

avantpy.exception_handling.handle_IfNobreakError(exc, source)

Handles situation where nobreak was wrongly use as a replacement for else in an if/elif/else block.

avantpy.exception_handling.handle_IndentationError(exc, source)

Handles IndentationError.

avantpy.exception_handling.handle_MismatchedBracketsError(exc, source)

Handles situation where a bracket of one kind ‘([{‘ is closed with a bracket of a different kind.

avantpy.exception_handling.handle_MissingLeftBracketError(exc, source)

Handles situation where at bracket of one kind ),],} is closed without a corresponding open bracket.

avantpy.exception_handling.handle_MissingRepeatColonError(exc, source)

Handles situation where a statement beginning with repeat does not end with a colon.

avantpy.exception_handling.handle_MissingRepeatError(exc, source)

Handles situation where either until or forever was used without being preceeded by repeat.

avantpy.exception_handling.handle_NameError(exc, source)

Handles situation where a NameError is raise.

avantpy.exception_handling.handle_NobreakFirstError(exc, source)

Handles situation where nobreak was wrongly use as a replacement for else in statements like var = x if condition else y.

avantpy.exception_handling.handle_NobreakSyntaxError(exc, source)

Handles situation where nobreak is used without a matching for or while loop and not already raised.

avantpy.exception_handling.handle_RepeatFirstError(exc, source)

Handles situation where repeat was uses wrongly as it did not appear as the first keyword of a given statement.

avantpy.exception_handling.handle_TabError(exc, source)

Handles TabError.

avantpy.exception_handling.handle_TryNobreakError(exc, source)

Handles situation where nobreak was wrongly use as a replacement for else in an try/except/else/finally block.

avantpy.exception_handling.handle_UnknownDialectError(exc, *args)

Handles error raised when an unknown dialect is requested

avantpy.exception_handling.handle_UnknownLanguageError(exc, *args)

Handles error raised when an unknown language is requested

avantpy.exception_handling.handle_exception(exc, source)

Generic function to handle exceptions and returns a friendlier traceback than Python.

If the exception is not recognized as one for which a friendlier traceback can be provided, None is returned

avantpy.exception_handling.write_err(msg)

Writes a string to sys.stderr.

avantpy.exception_handling.write_exception_info(exc, source)

Writes the information we have after processing the exception.

exceptions

Custom exceptions

exception avantpy.exceptions.AvantPyException(msg, args)

Base class for custom exceptions.

exception avantpy.exceptions.AvantPySyntaxError(msg, args)

Base class for custom exceptions that are SyntaxErrors

exception avantpy.exceptions.IfNobreakError(msg, args)

Raised if nobreak is used instead of else in an if statement

exception avantpy.exceptions.MismatchedBracketsError(msg, args)

Raised if a left parenthesis ‘(‘, or square bracket ‘[‘ or curly bracket ‘{‘, is closed by a right ‘bracket’ of a different type.

In standard Python, this would be indicated as:

SyntaxError: invalid syntax

exception avantpy.exceptions.MissingLeftBracketError(msg, args)

Raised if a right parenthesis ‘)’, or square bracket ‘]’ or curly bracket ‘}’, is found without having a matching left bracket found previously.

In standard Python, this would be indicated as:

SyntaxError: invalid syntax

exception avantpy.exceptions.MissingRepeatColonError(msg, args)

Raised if a line beginning with repeat does not end with a colon.

exception avantpy.exceptions.MissingRepeatError(msg, args)

Raised if until or forever is used without being preceeded by repeat.

exception avantpy.exceptions.NobreakFirstError(msg, args)

Raised if nobreak is used somewhere other than at the beginning of a line.

exception avantpy.exceptions.NobreakSyntaxError(msg, args)

Raised if nobreak is without a matching block.

exception avantpy.exceptions.RepeatFirstError(msg, args)

Raised if repeat is used somewhere other than at the beginning of a line.

exception avantpy.exceptions.TryNobreakError(msg, args)

Raised if nobreak is used instead of else in try/except statement

exception avantpy.exceptions.UnknownDialectError(msg, args)

Raised when attempting to set dialect to unsupported value.

exception avantpy.exceptions.UnknownLanguageError(msg, args)

Raised when attempting to set lang to unsupported value.

gui

This module provides a GUI to the conversion function.

It makes it possible to load up a source file written in a given dialect and convert it into another dialect or into Python, showing both the original source and the converted one in two different editor frames, with Python keyword, as well as their counterpart in various dialects, highlighted.

Todo

This module needs to be documented properly

class avantpy.gui.App(master=None)

add_ui()

Adds UI elements to the main window

convert_source(event=None)

Converts the content of the source window into the requested dialect.

get_conversion_dialect(event)

Gets the conversion dialect selected from the ComboBox.

get_source(event=None)

Opens a file by looking first from the current directory.

class avantpy.gui.EditorWidget(parent, parent_frame, other=None)

A scrollable text editor, that can save files.

colorize(event=None)

Colorizes the Python keywords and a few builtins, as well as the corresponding version in other dialects.

get_text()

Gets the current content and returns it as a string

insert_text(txt)

Inserts the text in the editor, replacing any previously existing content.

save_file(event=None)

Saves the file currently in the Texteditor

class avantpy.gui.TextLineNumbers(*args, **kwargs)

redraw(*args)

redraw line numbers

import_hook

A custom importer making use of the import hook capability

class avantpy.import_hook.AvantPyLoader(filename)

A custom loader which will transform the source prior to its execution

exec_module(module)

import the source code, converts it before executing it.

class avantpy.import_hook.AvantPyMetaFinder

A custom finder to locate modules. The main reason for this code is to ensure that our custom loader, which does the code transformations, is used.

find_spec(fullname, path, target=None)

Finds the appropriate properties (spec) of a module, and sets its loader.

avantpy.import_hook.import_main(name)

Imports the module that is to be interpreted as the main module.

avantpy is often invoked with a script meant to be run as the main module its source is transformed with the -s (or –source) option, as in:

python -m avantpy -s name

Python identifies avantpy as the main script, which is not what we want.