pycld2 Language Detection

Warnings

• breaking Installation commonly fails with 'Failed building wheel for pycld2' errors, particularly on non-standard architectures (e.g., ARM/aarch64) or Windows, due to missing C/C++ compilers or Python development headers.
  Fix: Ensure that your system has the necessary C/C++ build tools (e.g., GCC, Clang, MSVC) and Python development headers installed before attempting `pip install pycld2`. For Debian/Ubuntu, `sudo apt-get install build-essential python3-dev` is often required. For macOS, `xcode-select --install`. For Windows, install Visual Studio Build Tools.
• gotcha The `detect()` function strictly requires UTF-8 encoded `bytes` or a `str` as input. Passing bytes encoded in other formats (e.g., Latin-1) will raise a `pycld2.error`.
  Fix: Always ensure your input text is either a standard Python `str` (which `pycld2` will encode to UTF-8 internally) or `bytes` that have been explicitly encoded using UTF-8 (e.g., `my_text.encode('utf-8')`).
• gotcha Setting the `debugScoreAsQuads` parameter to `True` in `detect()` can significantly impact performance, potentially causing a 2x performance hit.
  Fix: Only enable `debugScoreAsQuads=True` if you explicitly need this debugging feature, as it incurs a substantial CPU cost.
• deprecated The `hintEncoding` parameter in the `detect()` function is currently not working and provides no biasing hint to the detector.
  Fix: Avoid using `hintEncoding` as it has no effect. Focus on ensuring the input `utf8Bytes` is correctly encoded.

Install

• pip install pycld2 Standard Installation
• sudo apt-get install build-essential python3-dev # On Debian/Ubuntu # On macOS: Install Xcode Command Line Tools (xcode-select --install) # On Windows: Install MSVC Build Tools (part of Visual Studio Community) System Dependencies for Compilation

Imports

• detect

  import pycld2 as cld2
  isReliable, textBytesFound, details = cld2.detect(text)

  The `detect` function is the primary entry point for language detection.

Quickstart

This quickstart demonstrates how to use `pycld2.detect()` for basic language identification and for obtaining detailed language vectors from mixed-language text. The `detect` function returns a tuple containing reliability, bytes found, a list of detected languages with confidence scores, and optionally segment-level language vectors.

import pycld2 as cld2

# Example 1: Basic detection
text_russian = "а неправильный формат идентификатора дн назад"
isReliable, textBytesFound, details = cld2.detect(text_russian)

print(f"Text: '{text_russian}'")
print(f"Is reliable: {isReliable}")
print(f"Detected language: {details[0][0]} ({details[0][1]})")
print(f"Details: {details}")

print('\n---\n')

# Example 2: Detecting multiple languages and getting vectors
text_mixed = """France is the largest country in Western Europe. A accès aux chiens et aux frontaux qui lui ont été il peut consulter. The quick brown fox jumped over the lazy dog."""
isReliable, textBytesFound, details, vectors = cld2.detect(
    text_mixed,
    returnVectors=True
)

print(f"Text: '{text_mixed}'")
print(f"Is reliable: {isReliable}")
print(f"Detected language (summary): {details[0][0]} ({details[0][1]})")
print(f"Segment language vectors: {vectors}")