ASTTokens

3.0.1 · active · verified Sat Mar 28

ASTTokens is a Python library that annotates Abstract Syntax Trees (ASTs) with the positions of tokens and text in the source code that generated them. This functionality is essential for tools that perform source code transformations, such as automated refactoring or syntax highlighting. The current version is 3.0.1, released on November 15, 2025. The library is actively maintained with a stable release cadence, ensuring ongoing support and updates.

Warnings

breaking ASTTokens requires Python 3.8 or higher. Ensure your environment meets this requirement to avoid compatibility issues.
Fix: Upgrade to Python 3.8 or a later version.
gotcha ASTTokens may not support certain node types in Python 3.8 due to known issues. Refer to the documentation for a list of unsupported node types.
Fix: Consider upgrading to a later Python version or consult the documentation for workarounds.

Install

pip install asttokens Install ASTTokens

Imports

ASTTokens
```
from asttokens import ASTTokens
```
Direct import of ASTTokens class for annotation tasks
ASTText
```
from asttokens import ASTText
```
Import ASTText for text-based AST annotations
LineNumbers
```
from asttokens import LineNumbers
```
Import LineNumbers for line and column number conversions

Quickstart

This example demonstrates how to parse and annotate source code using ASTTokens, and how to access the annotated AST and token positions.

import asttokens
import ast

# Sample source code
source = "Robot('blue').walk(steps=10*n)"

# Parse and annotate the source code
atok = asttokens.ASTTokens(source, parse=True)

# Access the annotated AST
tree = atok.tree

# Retrieve the first token of the first node
first_token = tree.body[0].first_token

# Print the start position of the first token
print(f"Start position: {first_token.startpos}")

view raw JSON →