Fast Language Detection

Warnings

• breaking The configuration system was overhauled in v0.3.0, replacing environment variables (e.g., `FTLANG_CACHE`) with a dedicated `LangDetectConfig` class for explicit management. Existing code relying on environment variables for configuration will break.
  Fix: Migrate to using `LangDetectConfig` objects. For example, pass `config=LangDetectConfig(cache_dir='your/path')` to `LangDetector` or use `FTLANG_CACHE` environment variable before importing and initializing any components if not explicitly setting `cache_dir`.
• gotcha Detection accuracy can be reduced for text samples that are significantly shorter or longer than approximately 80 characters. Inputs are truncated to 80 characters by default.
  Fix: For longer texts, disable truncation via `LangDetectConfig(max_input_length=None)` if you understand the potential performance or accuracy implications. For short texts, be aware of inherent accuracy limitations.
• gotcha Different models have different memory footprints and accuracy. The 'lite' model is memory-friendly (~45-60 MB RSS) and works offline, while the 'full' model (~170-210 MB RSS) offers higher accuracy but consumes more memory. The `model='auto'` setting only falls back to the 'lite' model if a `MemoryError` occurs.
  Fix: Explicitly choose `model='lite'` for memory-constrained environments, `model='full'` for highest accuracy, or `model='auto'` with awareness of its specific fallback condition.
• gotcha The `model='auto'` fallback mechanism is specific to `MemoryError` only. Other issues like `FileNotFoundError`, `PermissionError`, or network-related errors during model loading will raise standard Python exceptions and are not silently handled or fallen back.
  Fix: Implement standard Python error handling (try-except blocks) for potential I/O or network issues when using the library.
• gotcha As of v0.4.0, newline characters in input text are always replaced with spaces internally to prevent errors with the underlying FastText model. This transformation is logged at a DEBUG level and happens silently by default.
  Fix: Be aware that newline characters are processed this way. If exact text formatting is critical for other parts of your pipeline, perform necessary pre-processing before passing text to `fast-langdetect`.
• gotcha The pre-trained FastText language identification models bundled or downloaded by `fast-langdetect` are licensed under the Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0) license. This is separate from the MIT license for the `fast-langdetect` code itself.
  Fix: If redistributing or modifying the model files, ensure compliance with the CC BY-SA 3.0 license terms, including attribution and share-alike conditions.

Install

• pip install fast-langdetect Install stable version

Imports

• detect_language

  from fast_langdetect import detect_language

  Primary function for quick, direct language detection.
• LangDetector

  from fast_langdetect import LangDetector

  Used to create custom detector instances for advanced configuration.
• LangDetectConfig

  from fast_langdetect import LangDetectConfig

  Configuration class for setting parameters like cache directory or input length.

Quickstart

This quickstart demonstrates how to use `fast-langdetect` to detect the language of various text inputs. It shows usage of the default 'lite' model, explicitly selecting the 'full' model for higher accuracy, and using the 'auto' model with fallback behavior.

from fast_langdetect import detect_language

text1 = "Hello, how are you?"
text2 = "Bonjour, comment allez-vous?"
text3 = "Este es un texto muy largo en español, con muchas palabras y frases para probar la detección de idioma."

# Detect language with default settings (lite model)
result1 = detect_language(text1)
print(f"'{text1}' detected as: {result1.lang} (confidence: {result1.score:.2f})")

# Detect language using the 'full' model for potentially higher accuracy
result2 = detect_language(text2, model='full')
print(f"'{text2}' detected as: {result2.lang} (confidence: {result2.score:.2f})")

# Detect language with 'auto' model, which falls back to lite on MemoryError
# Also request top 2 languages
result3 = detect_language(text3, model='auto', k=2)
print(f"'{text3}' detected top 2 as: {result3}")