{"id":570,"library":"pytokens","title":"A Fast, spec compliant Python 3.14+ tokenizer","description":"Pytokens is an open-source Python library providing a fast, spec-compliant tokenizer for Python 3.14+ that is also capable of running on older Python versions (>=3.8). Currently at version 0.4.1, it appears to be actively maintained with recent releases focusing on packaging and development improvements.","status":"active","version":"0.4.1","language":"python","source_language":"en","source_url":"https://github.com/tusharsadhwani/pytokens","tags":["tokenizer","python3.14","parsing","spec-compliant","cli-tool"],"install":[{"cmd":"pip install pytokens","lang":"bash","label":"Install with pip"}],"dependencies":[{"reason":"Used for type checking during development. Not a runtime dependency.","package":"mypy","optional":true}],"imports":[{"symbol":"pytokens","correct":"import pytokens"}],"quickstart":{"code":"echo \"print('Hello, World!')\" > example.py\npython -m pytokens example.py","lang":"bash","description":"Pytokens is primarily designed for command-line usage to tokenize Python files. This example demonstrates tokenizing a simple Python file. The output will be a stream of tokens (e.g., NAME, STRING, OP, etc.) representing the input file's content."},"warnings":[{"fix":"To disable mypyc compilation (e.g., for local development or debugging), set the environment variable PYTOKENS_USE_MYPYC=0 before installation (e.g., PYTOKENS_USE_MYPYC=0 pip install pytokens).","message":"Pytokens is compiled with mypyc by default for performance. This means the installed module might be a compiled extension (.so or .pyd) rather than pure Python code, which can affect debugging or introspection.","severity":"gotcha","affected_versions":">=0.4.0"},{"fix":"Ensure your environment or tests account for the 3.14+ tokenizer specification if running on older Python versions, especially if precise tokenizer behavior is critical for your application.","message":"While pytokens aims for Python 3.14+ spec compliance and runs on older Python versions (>=3.8), be aware that its tokenization behavior adheres to the 3.14+ specification. This might introduce subtle differences compared to the native tokenizer behavior of older Python interpreters.","severity":"gotcha","affected_versions":"All versions when running on Python < 3.14"},{"fix":"Ensure that Python scripts only contain valid Python syntax. If shell commands are necessary, they should be executed using Python's `subprocess` module or placed in a separate shell script and executed appropriately.","message":"The script contains shell commands (e.g., `echo`, `>`) which result in a `SyntaxError` when executed by the Python interpreter. This typically occurs when a shell command is mistakenly placed in a Python file or when a shell script is run with the Python interpreter.","severity":"breaking","affected_versions":"All Python versions (as this is a fundamental language syntax error)"},{"fix":"Ensure that shell commands are executed in a shell environment (e.g., by structuring the build/test script to correctly separate shell commands from Python script execution, or by using Python's `subprocess` module with `shell=True` if executing shell commands from Python code). If the file was intended to be a shell script, ensure it is executed by a shell interpreter.","message":"The script attempted to execute shell commands directly as Python code, leading to a `SyntaxError`. Python interpreters do not understand shell commands like `echo` or shell redirection (`>`).","severity":"breaking","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-05-12T15:25:50.410Z","next_check":"2026-06-26T00:00:00.000Z","problems":[{"fix":"Install the library using pip: `pip install pytokens`","cause":"The `pytokens` library has not been installed in the current Python environment or the environment is not active.","error":"ModuleNotFoundError: No module named 'pytokens'"},{"fix":"Ensure the input passed to `pytokens.tokenize()` is always a string containing the Python source code. Example: `pytokens.tokenize(\"def func():\\n    pass\")`","cause":"The `pytokens.tokenize` function received an input that was not a string, which is required for the Python source code.","error":"TypeError: an integer is required (got type dict)"},{"fix":"Ensure you are using Python 3.8 or newer; if the issue persists, check PyPI for available wheels for your specific environment.","cause":"Pip could not find a compatible pre-built package for the current Python version (must be >= 3.8) or platform.","error":"ERROR: No matching distribution found for pytokens"},{"fix":"Import the function directly from the package: `from pytokens import tokenize` or use `pytokens.tokenize` after `import pytokens`.","cause":"`tokenize` is a function directly available under the `pytokens` package, not a submodule, so it cannot be imported using dot notation as a nested module.","error":"ModuleNotFoundError: No module named 'pytokens.tokenize'"}],"ecosystem":"pypi","meta_description":null,"install_score":100,"install_tag":"verified","quickstart_score":0,"quickstart_tag":"stale","pypi_latest":null,"install_checks":{"last_tested":"2026-05-12","tag":"verified","tag_description":"installs cleanly on critical runtimes, fast import, recently tested","results":[{"runtime":"python:3.10-alpine","python_version":"3.10","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.8,"disk_size":"18.5M"},{"runtime":"python:3.10-slim","python_version":"3.10","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.03,"mem_mb":1.8,"disk_size":"19M"},{"runtime":"python:3.11-alpine","python_version":"3.11","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.1,"mem_mb":2,"disk_size":"20.4M"},{"runtime":"python:3.11-slim","python_version":"3.11","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.09,"mem_mb":2,"disk_size":"21M"},{"runtime":"python:3.12-alpine","python_version":"3.12","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.1,"mem_mb":1.8,"disk_size":"12.3M"},{"runtime":"python:3.12-slim","python_version":"3.12","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.09,"mem_mb":1.8,"disk_size":"13M"},{"runtime":"python:3.13-alpine","python_version":"3.13","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.05,"mem_mb":2.3,"disk_size":"11.9M"},{"runtime":"python:3.13-slim","python_version":"3.13","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.08,"mem_mb":2.1,"disk_size":"12M"},{"runtime":"python:3.9-alpine","python_version":"3.9","os_libc":"alpine (musl)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.7,"disk_size":"18.0M"},{"runtime":"python:3.9-slim","python_version":"3.9","os_libc":"slim (glibc)","variant":"default","exit_code":0,"wheel_type":null,"failure_reason":null,"install_time_s":null,"import_time_s":0.04,"mem_mb":1.7,"disk_size":"19M"}]},"quickstart_checks":{"last_tested":"2026-04-23","tag":"stale","tag_description":"widespread failures or data too old to trust","results":[{"runtime":"python:3.10-alpine","exit_code":1},{"runtime":"python:3.10-slim","exit_code":1},{"runtime":"python:3.11-alpine","exit_code":1},{"runtime":"python:3.11-slim","exit_code":1},{"runtime":"python:3.12-alpine","exit_code":1},{"runtime":"python:3.12-slim","exit_code":1},{"runtime":"python:3.13-alpine","exit_code":1},{"runtime":"python:3.13-slim","exit_code":1},{"runtime":"python:3.9-alpine","exit_code":1},{"runtime":"python:3.9-slim","exit_code":1}]}}