{"id":427,"library":"asttokens","title":"ASTTokens","description":"ASTTokens is a Python library that annotates Abstract Syntax Trees (ASTs) with the positions of tokens and text in the source code that generated them. This functionality is essential for tools that perform source code transformations, such as automated refactoring or syntax highlighting. The current version is 3.0.1, released on November 15, 2025. The library is actively maintained with a stable release cadence, ensuring ongoing support and updates.","status":"active","version":"3.0.1","language":"python","source_language":"en","source_url":"https://github.com/gristlabs/asttokens","tags":["AST","Python","Source Code Transformation","Refactoring","Syntax Highlighting"],"install":[{"cmd":"pip install asttokens","lang":"bash","label":"Install ASTTokens"}],"dependencies":[{"reason":"Provides additional AST capabilities for enhanced functionality","package":"astroid","optional":true}],"imports":[{"note":"Direct import of ASTTokens class for annotation tasks","symbol":"ASTTokens","correct":"from asttokens import ASTTokens"},{"note":"Import ASTText for text-based AST annotations","symbol":"ASTText","correct":"from asttokens import ASTText"},{"note":"Import LineNumbers for line and column number conversions","symbol":"LineNumbers","correct":"from asttokens import LineNumbers"}],"quickstart":{"code":"import asttokens\nimport ast\n\n# Sample source code\nsource = \"Robot('blue').walk(steps=10*n)\"\n\n# Parse and annotate the source code\natok = asttokens.ASTTokens(source, parse=True)\n\n# Access the annotated AST\ntree = atok.tree\n\n# Retrieve the first token of the first node\nfirst_token = tree.body[0].first_token\n\n# Print the start position of the first token\nprint(f\"Start position: {first_token.startpos}\")","lang":"python","description":"This example demonstrates how to parse and annotate source code using ASTTokens, and how to access the annotated AST and token positions."},"warnings":[{"fix":"Upgrade to Python 3.8 or a later version.","message":"ASTTokens requires Python 3.8 or higher. Ensure your environment meets this requirement to avoid compatibility issues.","severity":"breaking","affected_versions":"all"},{"fix":"Consider upgrading to a later Python version or consult the documentation for workarounds.","message":"ASTTokens may not support certain node types in Python 3.8 due to known issues. Refer to the documentation for a list of unsupported node types.","severity":"gotcha","affected_versions":"3.8"}],"env_vars":null,"last_verified":"2026-05-12T13:44:05.874Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Install the module using pip: 'pip install asttokens'.","cause":"The 'asttokens' module is not installed in the Python environment.","error":"ModuleNotFoundError: No module named 'asttokens'"},{"fix":"Update 'asttokens' to a version compatible with Python 3.11 or later.","cause":"The 'formatargspec' function was removed from the 'inspect' module in Python 3.11, causing compatibility issues with 'asttokens'.","error":"ImportError: cannot import name 'formatargspec' from 'inspect'"},{"fix":"Import the class correctly: 'from asttokens import ASTTokens'.","cause":"The 'ASTTokens' class is not directly accessible from the 'asttokens' module due to incorrect import.","error":"AttributeError: module 'asttokens' has no attribute 'ASTTokens'"},{"fix":"Upgrade `asttokens` to the latest version (e.g., `pip install --upgrade asttokens`) to ensure compatibility with recent API changes where `ASTText` is directly importable.","cause":"This error typically occurs when using an older version of the `asttokens` library where the `ASTText` class was not directly exposed in the top-level package, while attempting to use code or examples written for a newer API version.","error":"ImportError: cannot import name 'ASTText' from 'asttokens'"},{"fix":"Initialize `ASTTokens` with `parse=True` to automatically parse the source code, or explicitly pass an `ast.Module` object to the `tree` argument if you've parsed it separately. Example: `atok = asttokens.ASTTokens(source, parse=True)`.","cause":"The `ASTTokens` object was initialized without setting the `parse=True` argument or providing an already parsed AST `tree`, causing its internal `tree` attribute to be `None`. Subsequent attempts to access attributes like `body` on the non-existent tree will fail.","error":"AttributeError: 'NoneType' object has no attribute 'body'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":80,"quickstart_tag":"verified","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.7,"disk_size":"17.9M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":1,"disk_size":"18M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.03,"mem_mb":1.2,"disk_size":"19.8M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":1.2,"disk_size":"20M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":0.9,"disk_size":"11.7M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":0.9,"disk_size":"12M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":1.2,"disk_size":"11.3M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.02,"mem_mb":1,"disk_size":"12M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.7,"disk_size":"17.4M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.01,"mem_mb":0.7,"disk_size":"18M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"verified","tag_description":"quickstart runs on critical runtimes, recently tested","results":[{"runtime":"python:3.10-alpine","exit_code":0},{"runtime":"python:3.10-slim","exit_code":0},{"runtime":"python:3.11-alpine","exit_code":0},{"runtime":"python:3.11-slim","exit_code":0},{"runtime":"python:3.12-alpine","exit_code":0},{"runtime":"python:3.12-slim","exit_code":0},{"runtime":"python:3.13-alpine","exit_code":0},{"runtime":"python:3.13-slim","exit_code":0},{"runtime":"python:3.9-alpine","exit_code":0},{"runtime":"python:3.9-slim","exit_code":0}]}}