untokenize

Warnings

• gotcha The `untokenize` library's last release was in February 2014, indicating it is no longer actively maintained. While its core functionality is simple and stable, it will not receive updates for new Python syntax features or potential edge cases that arise with future Python versions.
  Fix: Be aware of its maintenance status and consider thoroughly testing its behavior with newer Python versions (beyond Python 2.7 era) if used in critical projects. For modern, actively maintained tokenization/untokenization needs, consider alternatives or the standard library's `tokenize` module with its known limitations.
• gotcha This library provides an `untokenize` function that is distinct from Python's standard `tokenize.untokenize`. The key difference is that this library prioritizes the preservation of original whitespace between tokens, which the standard library function does not reliably do, potentially leading to different formatting in the output.
  Fix: Always explicitly import `untokenize` as `import untokenize` and call `untokenize.untokenize(tokens)` to ensure you are using this library's version. If you need the standard library's behavior, use `from tokenize import untokenize as std_untokenize`.

Install

• pip install untokenize Install latest version

Imports

• untokenize

  import untokenize

  This library provides its own untokenize function, distinct from the standard library's tokenize.untokenize, which has different behavior regarding whitespace. Importing directly as `untokenize` avoids confusion.

Quickstart

This example demonstrates how to use `untokenize` to reconstruct source code from a token stream, ensuring original whitespace is preserved. It uses `tokenize.tokenize` from the standard library to generate tokens, then passes them to `untokenize.untokenize`.

import tokenize
import io
import untokenize

source_code_with_whitespace = "def hello(  name ):\n    print(f'Hello, {name}!') # A comment"

# Tokenize the source code using the standard library's generator
# The readline callable should return bytes for tokenize.tokenize
tokens_generator = tokenize.tokenize(io.BytesIO(source_code_with_whitespace.encode('utf-8')).readline)

# Convert generator to a list of tokens
tokens = list(tokens_generator)

# Use untokenize library to reconstruct source code, preserving whitespace
reconstructed_code = untokenize.untokenize(tokens)

print("Original:")
print(source_code_with_whitespace)
print("\nReconstructed (with untokenize library):")
print(reconstructed_code)

# For comparison, standard library's untokenize (might format differently)
# from tokenize import untokenize as std_untokenize
# std_reconstructed_code = std_untokenize(tokens)
# print("\nReconstructed (with stdlib tokenize.untokenize):")
# print(std_reconstructed_code)