langid.py: Language Identification

Warnings

• gotcha The `langid.classify()` function, by default, returns an unnormalized log-probability estimate, not a standard probability (0-1 range). Interpreting this value directly as a confidence score between 0 and 1 is incorrect.
  Fix: To obtain normalized probabilities (0-1), you must explicitly initialize a `LanguageIdentifier` instance with `norm_probs=True`. Example: `from langid.langid import LanguageIdentifier, MODEL_FILE; identifier = LanguageIdentifier.from_pickled_model(MODEL_FILE, norm_probs=True); identifier.classify(text)`.
• breaking The training tools (`LDfeatureselect.py`, `train.py`) accompanying `langid.py` are explicitly stated to be Python 2-only. Attempting to use them with Python 3 will result in compatibility errors.
  Fix: Users needing to train custom models in modern Python environments should be aware that the original `langid` library's training tools are not Python 3 compatible. Consider using the `py3langid` fork, which has adapted training scripts (though noted as untested), or an alternative library for custom training.
• gotcha For optimal performance when processing large numbers of texts, `langid` should be used as a Python library (importing and calling `classify()`) or as a web service. Repeatedly invoking `langid.py` from the command line for each text incurs significant overhead due to repeated model loading.
  Fix: Initialize the `langid` module once within your application (e.g., `import langid`) and then call `langid.classify()` multiple times. For batch processing of files, the library offers a batch mode (`-b` flag) that utilizes multiprocessing.
• gotcha A modernized and significantly faster fork, `py3langid`, exists and is optimized for Python 3.6+. It offers substantial performance improvements (e.g., 25-30x faster model loading, 5-6x faster classification) over the original `langid` package.
  Fix: For new Python 3.6+ projects or performance-critical applications, consider using `py3langid`. It's designed as a drop-in replacement: `pip install py3langid` and then `import py3langid as langid`.
• gotcha Passing non-string types or strings with unexpected formatting (e.g., a tuple represented as a string, or unhandled encoding issues from external sources like databases) to `langid.classify()` can lead to incorrect classifications or `KeyError` exceptions.
  Fix: Ensure that all input to `langid.classify()` is a clean Unicode string. Always validate and potentially preprocess inputs, especially when sourcing text from databases or other external systems, to prevent unexpected formatting or encoding issues.

Install

• pip install langid Install via pip

Imports

• langid

  import langid

Quickstart

The `langid.classify()` function takes a string and returns a tuple containing the predicted ISO 639-1 language code and an unnormalized log-probability estimate. For normalized probabilities (0-1), a `LanguageIdentifier` instance must be explicitly configured.

import langid

text1 = "This is a sample text in English."
text2 = "Ceci est un exemple de texte en français."
text3 = "Dies ist ein Beispieltext auf Deutsch."

print(f"'{text1}' -> {langid.classify(text1)}")
print(f"'{text2}' -> {langid.classify(text2)}")
print(f"'{text3}' -> {langid.classify(text3)}")

# To get normalized probabilities (0-1 range):
from langid.langid import LanguageIdentifier, MODEL_FILE

identifier = LanguageIdentifier.from_pickled_model(MODEL_FILE, norm_probs=True)
print(f"Normalized for '{text1}' -> {identifier.classify(text1)}")