Whisper Normalizer

0.1.12 · active · verified Wed Apr 15

Whisper Normalizer (version 0.1.12) is a Python package that implements the text standardization and normalization approach used in OpenAI's Whisper ASR model. It provides robust text normalization functionalities, crucial for evaluating Automatic Speech Recognition (ASR) systems by reducing unintentional penalties in metrics like WER and CER. The library extends beyond basic English normalization to include specialized normalizers for Indic languages, addressing challenges such as diacritic preservation. The project appears to have an active development and release cadence.

Warnings

gotcha Using `BasicTextNormalizer` or `EnglishTextNormalizer` for Indic languages can lead to unintended loss of crucial diacritics and semantic changes. These normalizers are optimized for English and general text processing, not for the unique characteristics of Indic scripts.
Fix: Always import and use the appropriate language-specific normalizer from `whisper_normalizer.indic_normalizer` (e.g., `MalayalamNormalizer`, `PunjabiNormalizer`) for Indic languages.
gotcha Between versions 0.1.0 and 0.1.4, the library significantly changed its number normalization backend for Indic languages by switching to the `AI4Bharat IndicNumtowords` package and removed network calls from `EnglishTextNormalizer`. This could alter previously expected normalization outputs, particularly for numbers, currencies, and certain English text patterns.
Fix: Users upgrading from versions prior to 0.1.4 should carefully review normalization outputs in their applications, especially for number-heavy text, to ensure compatibility with the new implementations.
gotcha Do not confuse `whisper-normalizer` (this standalone library) with the internal `whisper.normalizers` submodule found within OpenAI's main `openai-whisper` package. This library is designed for standalone text normalization, while `openai-whisper`'s normalizers are integrated directly into its ASR model pipeline and have a different import path (e.g., `from whisper.normalizers import BasicTextNormalizer`).
Fix: For standalone text normalization tasks, ensure you are importing from `whisper_normalizer`. If integrating with the full OpenAI Whisper ASR model, use its internal `whisper.normalizers`.

Install

pip install whisper-normalizer Install stable version

Imports

EnglishTextNormalizer

from whisper_normalizer.english import EnglishTextNormalizer

BasicTextNormalizer

from whisper_normalizer.basic import BasicTextNormalizer

MalayalamNormalizer
wrong:
```
from whisper_normalizer.basic import BasicTextNormalizer
```
correct:
```
from whisper_normalizer.indic_normalizer import MalayalamNormalizer
```
BasicTextNormalizer can remove crucial diacritics and change meaning in Indic languages. Use language-specific normalizers like MalayalamNormalizer for correct results.

Quickstart

Demonstrates the use of EnglishTextNormalizer, BasicTextNormalizer, and a specific Indic normalizer (MalayalamNormalizer) to process and normalize text. This highlights the different normalization strategies available for various linguistic contexts.

from whisper_normalizer.english import EnglishTextNormalizer
from whisper_normalizer.basic import BasicTextNormalizer
from whisper_normalizer.indic_normalizer import MalayalamNormalizer

# English Text Normalization
english_normalizer = EnglishTextNormalizer()
text_en = "I'm a little teapot, short & stout. Tip me over and pour me out! $20 million."
normalized_en = english_normalizer(text_en)
print(f"English (input): {text_en}")
print(f"English (output): {normalized_en}\n")

# Basic Text Normalization (general purpose, but use with caution for Indic languages)
basic_normalizer = BasicTextNormalizer()
text_basic = "Hello [music] world (coughs). café résumé naïve."
normalized_basic = basic_normalizer(text_basic)
print(f"Basic (input): {text_basic}")
print(f"Basic (output): {normalized_basic}\n")

# Malayalam Text Normalization (example for Indic languages)
malayalam_normalizer = MalayalamNormalizer()
text_ml = "എന്റെ കമ്പ്യൂട്ടറിനു് എന്റെ ഭാഷ."
normalized_ml = malayalam_normalizer(text_ml)
print(f"Malayalam (input): {text_ml}")
print(f"Malayalam (output): {normalized_ml}")

view raw JSON →