ANTLR 4 Python 3 Runtime

Warnings

• breaking ANTLR 4.10 and newer versions drop support for Python 3.5. Attempts to install or run the runtime on Python 3.5 will result in `SyntaxError` due to the use of f-strings in the codebase.
  Fix: Upgrade your Python environment to Python 3.6 or higher. For Python 3.5, you must pin the `antlr4-python3-runtime` version to `4.9.x` or earlier (e.g., `pip install antlr4-python3-runtime==4.9.3`).
• gotcha The `antlr4-python3-runtime` package only provides the runtime libraries. To generate Python lexer/parser source files (`.py` files) from your `.g4` grammar, you *must* have a Java Runtime Environment (JRE) installed and download the ANTLR 4 tool (a `.jar` file).
  Fix: Install a JRE and download the `antlr-4.x.x-complete.jar` file from the official ANTLR website. Then use `java -jar antlr-4.x.x-complete.jar -Dlanguage=Python3 YourGrammar.g4` to generate the Python code before attempting to import it.
• breaking A known issue existed in version 4.13.1 where the ANTLR runtime's internal version check might conflict with the version of the generated code, leading to runtime errors.
  Fix: Upgrade to `antlr4-python3-runtime==4.13.2` or later to resolve this specific version disagreement. Always ensure your ANTLR tool version matches your runtime version as closely as possible.
• gotcha Care must be taken when choosing input streams. For string input, `InputStream` is generally used. For file input, `FileStream` is suitable. For reading from `sys.stdin`, `StdinStream` is available. Mismatching can lead to unexpected behavior or errors.
  Fix: Select the appropriate input stream class based on your data source: `InputStream(text)` for strings, `FileStream(filepath)` for files, and `StdinStream()` for standard input.
• gotcha ANTLR 4 provides both Listener and Visitor patterns for traversing parse trees. Listeners are event-driven and perform an LR traversal, calling `enter` and `exit` methods. Visitors offer more control by explicitly requiring you to call `visit()` methods, making them more flexible for complex tree manipulations.
  Fix: Understand the differences between Listeners (generated with `-listener`) and Visitors (generated with `-visitor`) and choose the pattern that best suits your parse tree processing needs. When using Visitors, remember to explicitly call `visit()` on child nodes if you want to traverse them.

Install

• pip install antlr4-python3-runtime Install with pip

Imports

• InputStream, CommonTokenStream, Lexer, Parser, ParseTreeWalker, ParserRuleContext

  from antlr4 import InputStream, CommonTokenStream
  from MyGrammarLexer import MyGrammarLexer # Assumes MyGrammar.g4 was compiled
  from MyGrammarParser import MyGrammarParser # Assumes MyGrammar.g4 was compiled
  from MyGrammarListener import MyGrammarListener # For listener pattern

  While `import antlr4` might work for some top-level components, specific classes are typically imported directly from the `antlr4` package or from the *generated* grammar files (e.g., `MyGrammarLexer.py`, `MyGrammarParser.py`).

Quickstart

This quickstart demonstrates the general workflow for using the ANTLR4 Python runtime. It assumes you have already created a `.g4` grammar file (e.g., `MyGrammar.g4`) and used the ANTLR Java tool to generate `MyGrammarLexer.py`, `MyGrammarParser.py`, and `MyGrammarListener.py` (or `MyGrammarVisitor.py`) files in your project directory. The example then shows how to take an input string, tokenize it with the generated lexer, parse it with the generated parser, and print the resulting parse tree. For actual parsing, replace the dummy classes with your real generated classes and ensure the `startRule()` method matches your grammar's entry point.

import os
from antlr4 import InputStream, CommonTokenStream, ParseTreeWalker

# --- BEGIN: Dummy generated classes for demonstration ---
# In a real scenario, these would be generated by the ANTLR tool:
# java -jar antlr-4.x.x-complete.jar -Dlanguage=Python3 MyGrammar.g4

class MyGrammarLexer:
    def __init__(self, input_stream): pass
    def getAllTokens(self): return []

class MyGrammarParser:
    def __init__(self, token_stream): pass
    def startRule(self): return None # Replace 'startRule' with your grammar's entry rule

class MyGrammarListener:
    def enterEveryRule(self, ctx): pass
    def exitEveryRule(self, ctx): pass
    # ... other enter/exit methods for your grammar rules

# --- END: Dummy generated classes ---

# Assuming MyGrammar.g4 contains:
# grammar MyGrammar;
# startRule: 'hello' ID EOF;
# ID: [a-zA-Z]+;
# WS: [ \t\r\n]+ -> skip;

def parse_input(text):
    input_stream = InputStream(text)
    lexer = MyGrammarLexer(input_stream) # Use your generated Lexer class
    stream = CommonTokenStream(lexer)
    parser = MyGrammarParser(stream)    # Use your generated Parser class
    
    # Optionally, set up error handling
    # parser.removeErrorListeners()
    # parser.addErrorListener(MyCustomErrorListener())
    
    tree = parser.startRule() # Call the entry rule of your grammar
    
    # Optionally, walk the parse tree with a listener or visitor
    # walker = ParseTreeWalker()
    # listener = MyGrammarListener() # Use your generated Listener/Visitor class
    # walker.walk(listener, tree)
    
    print(f"Parse tree: {tree.toStringTree(recog=parser)}")

if __name__ == '__main__':
    # Simulate user input or a file content
    # For a real application, you might read from stdin or a file
    test_input = os.environ.get('ANTLR_TEST_INPUT', 'hello world')
    print(f"Parsing: '{test_input}'")
    parse_input(test_input)
    print("Note: In a real scenario, 'MyGrammarLexer' and 'MyGrammarParser' would be generated .py files from your .g4 grammar using the ANTLR Java tool.")