Bangla Unicode Normalizer

Common errors

• FileNotFoundError: [Errno 2] No such file or directory: '.../bnunicodenormalizer/romanize_map.json'

  cause The `Normalizer` tried to load the default `romanize_map.json` but could not find it, often due to an incomplete installation or running from a non-standard environment.
  fix Ensure `bnunicodenormalizer` is properly installed via `pip install bnunicodenormalizer`. If providing a custom `romanize_mapping_path`, double-check the file existence and permissions for that path.

• ModuleNotFoundError: No module named 'bnunicodenormalizer'

  cause The `bnunicodenormalizer` package is not installed in the active Python environment.
  fix Install the package using `pip install bnunicodenormalizer`. If using virtual environments, ensure your IDE or terminal is activated to the correct environment.

• AttributeError: 'dict' object has no attribute 'normalized_text'

  cause The output of `bn_normalize(text)` is a dictionary. You are attempting to access a key as an attribute.
  fix Access dictionary keys using square bracket notation, e.g., `result['normalized_text']`. The returned dictionary structure is `{'normalized_text': '...', 'detected_lang': '...'}` (if language detection is enabled).

Warnings

• gotcha The `Normalizer` class can optionally take a `romanize_mapping_path` argument. If a custom path is provided and is incorrect or the file is missing, it will result in a `FileNotFoundError`. If not provided, it attempts to load a default file from the package installation directory.
  Fix: Ensure the provided `romanize_mapping_path` is an absolute and correct path to a valid JSON file, or omit the argument to use the default package-provided mapping.
• breaking As a `0.x.x` version library, minor version increments (e.g., from 0.1.x to 0.2.x) can introduce breaking changes without adhering strictly to SemVer, though no explicit breaking changes are documented between recent `0.1.x` versions.
  Fix: Pin your dependency to an exact version (`bnunicodenormalizer==0.1.7`) in `requirements.txt` or `pyproject.toml` to prevent unexpected updates. Review release notes for new versions before upgrading.
• gotcha The library has a direct dependency on `fasttext`. Installing `fasttext` can sometimes be challenging due to its native dependencies (e.g., C++ compiler). If `fasttext` fails to install correctly, the language detection features of `bnunicodenormalizer` will be unavailable or may cause errors, even if normalization functions still work.
  Fix: If `fasttext` installation fails, consult the `fasttext` documentation for system-specific prerequisites (e.g., `build-essential` on Debian/Ubuntu, XCode command line tools on macOS). You may need to install it separately first (`pip install fasttext`).

Install

• pip install bnunicodenormalizer Install stable version

Imports

• Normalizer

  from bnunicodenormalizer import Normalizer

Quickstart

Demonstrates how to initialize the Normalizer and use it to process a simple Bangla text string. The default initialization attempts to load necessary mapping files from the installed package directory.

from bnunicodenormalizer import Normalizer

# Initialize the normalizer. 
# By default, it attempts to load 'romanize_map.json' from its package directory.
bn_normalize = Normalizer()

text_to_normalize = "এই টেস্টিং টেক্সট।  ১০০ টাকা ।"
result = bn_normalize(text_to_normalize)

normalized_text = result["normalized_text"]
print(f"Original: {text_to_normalize}")
print(f"Normalized: {normalized_text}")

# The result dictionary might also contain 'detected_lang' 
# if fasttext is enabled and detects it.
# print(f"Detected language: {result.get('detected_lang', 'N/A')}")