Segments

2.4.0 · active · verified Sun Apr 12

Segments provides functions to tokenize and segment strings of text into individual characters or graphemes, and into segments according to orthography profiles. It is particularly useful for linguistic data processing using CLDF (Cross-Linguistic Data Formats). The library typically sees a few releases per year, with major versions introducing updates to Unicode standards.

Warnings

gotcha Distinguish carefully between `segments.tokenize` and `Segmenter.segment`. `segments.tokenize` performs basic Unicode grapheme cluster segmentation only. `Segmenter.segment` is used to apply custom orthography rules provided via a `profile`.
Fix: For rule-based segmentation, always instantiate `Segmenter` with a `profile` and use its `.segment()` method. Use `segments.tokenize()` exclusively for standard Unicode grapheme clustering.
breaking Version 2.0.0 introduced 'Unicode Standard tokenization', which may result in subtle changes to grapheme cluster output for certain complex character sequences compared to pre-2.0.0 versions. This is a behavioral change based on updated Unicode standards, not an API break.
Fix: Review tokenization outputs for critical linguistic data when upgrading from versions <2.0.0 to ensure consistency. Adjust downstream processing if specific grapheme clustering assumptions have changed due to updated Unicode interpretations.
gotcha The `profile` argument for `segments.Segmenter` expects a dictionary with a 'rules' key, formatted as a list of lists (e.g., `[['rule_text', 'ipa_trans']]`). Directly passing file paths or non-conforming dictionaries will fail.
Fix: Ensure the `profile` dictionary strictly adheres to the `{'rules': [[...]]}` structure. Refer to `pycldf` documentation for advanced orthography profile handling if loading from files or complex scenarios.

Install

pip install segments Install stable version

Imports

tokenize
```
from segments import tokenize
```
Segmenter
```
from segments import Segmenter
```

Quickstart

Demonstrates basic Unicode grapheme tokenization and custom segmentation using an orthography profile.

import segments

# Unicode grapheme segmentation (standard Unicode rules)
text_unicode = 'ŋ͡m'
graphemes = segments.tokenize(text_unicode)
print(f"Unicode graphemes for '{text_unicode}': {graphemes}")

# Segmentation using an orthography profile
# (example profile for 'ph', 'ch', 'th' as single segments)
orthography_profile = {
    "rules": [
        ["ph", "pʰ"],
        ["ch", "cʰ"],
        ["th", "tʰ"]
    ]
}
segmenter = segments.Segmenter(profile=orthography_profile)
text_profile = 'tʰaiph'
segments_profile = segmenter.segment(text_profile)
print(f"Profile segments for '{text_profile}': {segments_profile}")

# Expected output for verification
assert graphemes == ['ŋ', '͡', 'm']
assert segments_profile == ['tʰ', 'ai', 'pʰ']

view raw JSON →