{"id":4837,"library":"untokenize","title":"untokenize","description":"The `untokenize` library (version 0.1.1) transforms a stream of Python tokens back into source code. Its primary distinction from the standard library's `tokenize.untokenize()` is its ability to preserve the original whitespace between tokens, addressing a common limitation. Released in February 2014, this library is stable for its intended purpose, but is no longer under active development.","status":"maintenance","version":"0.1.1","language":"en","source_language":"en","source_url":"https://github.com/myint/untokenize","tags":["code generation","tokenization","whitespace","source code manipulation"],"install":[{"cmd":"pip install untokenize","lang":"bash","label":"Install latest version"}],"dependencies":[],"imports":[{"note":"This library provides its own untokenize function, distinct from the standard library's tokenize.untokenize, which has different behavior regarding whitespace. Importing directly as `untokenize` avoids confusion.","wrong":"from tokenize import untokenize","symbol":"untokenize","correct":"import untokenize"}],"quickstart":{"code":"import tokenize\nimport io\nimport untokenize\n\nsource_code_with_whitespace = \"def hello(  name ):\\n    print(f'Hello, {name}!') # A comment\"\n\n# Tokenize the source code using the standard library's generator\n# The readline callable should return bytes for tokenize.tokenize\ntokens_generator = tokenize.tokenize(io.BytesIO(source_code_with_whitespace.encode('utf-8')).readline)\n\n# Convert generator to a list of tokens\ntokens = list(tokens_generator)\n\n# Use untokenize library to reconstruct source code, preserving whitespace\nreconstructed_code = untokenize.untokenize(tokens)\n\nprint(\"Original:\")\nprint(source_code_with_whitespace)\nprint(\"\\nReconstructed (with untokenize library):\")\nprint(reconstructed_code)\n\n# For comparison, standard library's untokenize (might format differently)\n# from tokenize import untokenize as std_untokenize\n# std_reconstructed_code = std_untokenize(tokens)\n# print(\"\\nReconstructed (with stdlib tokenize.untokenize):\")\n# print(std_reconstructed_code)\n","lang":"python","description":"This example demonstrates how to use `untokenize` to reconstruct source code from a token stream, ensuring original whitespace is preserved. It uses `tokenize.tokenize` from the standard library to generate tokens, then passes them to `untokenize.untokenize`."},"warnings":[{"fix":"Be aware of its maintenance status and consider thoroughly testing its behavior with newer Python versions (beyond Python 2.7 era) if used in critical projects. For modern, actively maintained tokenization/untokenization needs, consider alternatives or the standard library's `tokenize` module with its known limitations.","message":"The `untokenize` library's last release was in February 2014, indicating it is no longer actively maintained. While its core functionality is simple and stable, it will not receive updates for new Python syntax features or potential edge cases that arise with future Python versions.","severity":"gotcha","affected_versions":"0.1.1"},{"fix":"Always explicitly import `untokenize` as `import untokenize` and call `untokenize.untokenize(tokens)` to ensure you are using this library's version. If you need the standard library's behavior, use `from tokenize import untokenize as std_untokenize`.","message":"This library provides an `untokenize` function that is distinct from Python's standard `tokenize.untokenize`. The key difference is that this library prioritizes the preservation of original whitespace between tokens, which the standard library function does not reliably do, potentially leading to different formatting in the output.","severity":"gotcha","affected_versions":"0.1.1"}],"env_vars":null,"last_verified":"2026-04-12T00:00:00.000Z","next_check":"2026-07-11T00:00:00.000Z"}