spaCy Chinese Word Segmentation (pkuseg)

1.0.1 · active · verified Wed Apr 15

spacy-pkuseg is a Chinese word segmentation toolkit for spaCy, forked from pkuseg-python. It provides a `PkusegSegmenter` component to integrate robust Chinese segmentation directly into spaCy's NLP pipeline. The current stable version is 1.0.1, with releases primarily focused on Python and core dependency (like NumPy) compatibility updates.

Warnings

breaking Numpy 2.0 compatibility breakage: spacy-pkuseg v1.0.0 and later require NumPy>=2.0. Earlier versions (<1.0.0) are incompatible with NumPy 2.0 due to binary interface changes.
Fix: If using spacy-pkuseg v1.0.0 or later, ensure NumPy is v2.0 or higher. If using an older version of spacy-pkuseg, pin NumPy to `<2.0` (e.g., `pip install 'numpy<2.0'`).
breaking Fork and renaming from `pkuseg-python`: The package `spacy-pkuseg` (from v0.0.26) is a fork. The import path changed from `pkuseg` to `spacy_pkuseg`. The default model also changed, and serialization for custom user dictionaries switched from `pickle` to `msgpack` (fixed for custom dicts in v0.0.30).
Fix: Update import statements to `from spacy_pkuseg import PkusegSegmenter`. Explicitly specify the desired model (e.g., `config={'model': 'web'}`) if you relied on a different default. Custom user dicts saved with older versions (before v0.0.30's fix) might need to be re-created.
gotcha Incorrect pipeline integration: Placing the `spacy_pkuseg` component incorrectly in the spaCy pipeline can lead to unexpected tokenization results or errors, especially if another tokenization component runs first.
Fix: Always add the `spacy_pkuseg` component to the end of the pipeline using `nlp.add_pipe("spacy_pkuseg", last=True)` to ensure it acts as the primary tokenizer.
gotcha Default model and explicit selection: `spacy-pkuseg` defaults to the `spacy_ontonotes` model if not specified. Users expecting a different model (e.g., 'web', 'news') might not get desired results without explicit configuration.
Fix: If the default `spacy_ontonotes` model is not suitable, explicitly specify the desired model when adding the component: `nlp.add_pipe("spacy_pkuseg", config={"model": "web"}, last=True)`.

Install

pip install spacy-pkuseg Install spacy-pkuseg

Imports

PkusegSegmenter
```
from spacy_pkuseg import PkusegSegmenter
```
The library was forked and renamed from `pkuseg-python` to `spacy-pkuseg`.

Quickstart

This quickstart demonstrates how to load a spaCy model, add the `spacy_pkuseg` component to the pipeline, and process Chinese text to get word-segmented tokens. Ensure a spaCy model is installed first.

import spacy

# Make sure to install a spaCy model, e.g., python -m spacy download en_core_web_sm
nlp = spacy.load("en_core_web_sm")

# Add the pkuseg component to the pipeline
# The default model is 'spacy_ontonotes'
nlp.add_pipe("spacy_pkuseg", last=True)

# To specify a different model or user dictionary:
# nlp.add_pipe("spacy_pkuseg", config={
#     "model": "web", 
#     "user_dict": "path/to/your_dict.txt"
# }, last=True)

text = "北京大学地球与空间科学学院"
doc = nlp(text)

print(f"Original text: {text}")
print(f"Tokens: {[token.text for token in doc]}")

view raw JSON →