Jiwer

Warnings

• breaking The functions `jiwer.compute_measures()` and `jiwer.visualize_measures()` were renamed in version 4.0.0. They are now `jiwer.process_words()` and `jiwer.visualize_alignment()` respectively. Additionally, `process_words` returns a `WordOutput` dataclass instead of a dictionary.
  Fix: Update calls from `jiwer.compute_measures()` to `jiwer.process_words()` and `jiwer.visualize_measures()` to `jiwer.visualize_alignment()`. Adjust code to access results from the returned `WordOutput` or `CharacterOutput` dataclass attributes (e.g., `output.wer`) instead of dictionary keys.
• breaking The behavior for handling empty reference sentences changed in version 4.0.0. Previously, an empty reference with an empty hypothesis could lead to undefined behavior (division by zero). As of 4.0.0, this scenario is explicitly defined to yield zero error, supporting evaluation for models hallucinating on silent audio.
  Fix: Review existing code that processes empty or potentially empty reference/hypothesis pairs. The new behavior is generally safer, but ensure it aligns with your specific evaluation logic.
• breaking The internal representation of alignment chunks changed in version 4.0.0. Alignments are now returned as a list of `jiwer.AlignmentChunk` dataclass objects, replacing the previous tuple-based format. This improves clarity and accessibility of alignment details.
  Fix: If you are directly inspecting or parsing the `alignments` output from `process_words()` or `process_characters()`, update your code to access attributes of the `AlignmentChunk` dataclass (e.g., `chunk.type`, `chunk.ref_start_idx`) instead of tuple indices.
• gotcha Word Error Rate (WER) can exceed 100% (or 1.0). This occurs when the total number of errors (substitutions, deletions, and insertions) is greater than the number of words in the reference text, often due to a high number of insertions by the ASR system.
  Fix: Understand that WER > 1.0 is an expected and valid outcome, indicating poor ASR performance with many extraneous words. Do not cap the WER at 1.0 in your reporting or analysis unless specifically required by a particular standard.
• gotcha When `jiwer.wer()` or `jiwer.cer()` are provided with lists of reference and hypothesis sentences, they internally concatenate all sentences to compute a *single, global* error rate for the entire dataset, which is standard for corpus-level evaluation. It does not return individual error rates per sentence.
  Fix: If sentence-by-sentence error rates are needed, you must iterate through your sentence pairs and call `jiwer.wer()` (or `jiwer.cer()`) for each pair individually, or use `jiwer.process_words()`/`process_characters()` and then access the `.wer` or `.cer` attribute of the returned object for each item in the results list.
• gotcha Jiwer calculates metrics on the raw input strings. For robust and fair evaluation, it is crucial to apply consistent text normalization (e.g., lowercasing, punctuation removal, expansion of contractions) to both reference and hypothesis strings *before* calling jiwer functions.
  Fix: Utilize `jiwer.Compose()` with transformation functions like `jiwer.ToLowerCase()`, `jiwer.RemovePunctuation()`, `jiwer.ExpandCommonEnglishContractions()`, etc., to build a preprocessing pipeline. Apply this pipeline to your reference and hypothesis texts before computing error rates.

Install

• pip install jiwer Install latest version

Imports

• wer

  from jiwer import wer

• cer

  from jiwer import cer

• process_words

  import jiwer
  output = jiwer.process_words(reference, hypothesis)

  The 'compute_measures' function was renamed to 'process_words' in version 4.0.0. The return type also changed from a dictionary to a dataclass (WordOutput).
• process_characters

  import jiwer
  output = jiwer.process_characters(reference, hypothesis)

• Compose

  from jiwer import Compose

  Used for creating transformation pipelines for text normalization.

Quickstart

This quickstart demonstrates how to calculate the Word Error Rate (WER) for both single and multiple reference/hypothesis pairs using `jiwer.wer()`. It also shows how to use `jiwer.process_words()` to obtain a more detailed output, including various error measures and the alignment between the reference and hypothesis.

import jiwer

# Calculate Word Error Rate (WER) for single strings
reference_single = "hello world"
hypothesis_single = "hello duck"
error_single = jiwer.wer(reference_single, hypothesis_single)
print(f"WER (single): {error_single}")

# Calculate WER for multiple sentences (lists of strings)
references_multiple = ["hello world", "i like monthy python"]
hypotheses_multiple = ["hello duck", "i like python"]
error_multiple = jiwer.wer(references_multiple, hypotheses_multiple)
print(f"WER (multiple): {error_multiple}")

# Get detailed output including alignments and all measures
output_details = jiwer.process_words(reference_single, hypothesis_single)
print(f"\nDetailed output WER: {output_details.wer}")
print(f"Detailed output MER: {output_details.mer}")
print(f"Alignments: {output_details.alignments}")