pysbd (Python Sentence Boundary Disambiguation)

0.3.4 · active · verified Sat Apr 11

pysbd (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection library that works out-of-the-box across many languages. It aims to provide accurate sentence segmentation even with complex text, abbreviations, and varied punctuation, offering an alternative to neural network-based approaches. The current version is 0.3.4, and the project appears to be actively maintained.

Warnings

breaking When integrating with spaCy, examples for spaCy v2.x using `nlp.add_pipe(PySBDFactory(nlp))` are not compatible with spaCy v3.x. The `add_pipe` API changed to expect a string name of the registered component factory.
Fix: For spaCy v3.x, use `@Language.factory('pysbd_segmenter')` decorator on a custom factory function, then `nlp.add_pipe('pysbd_segmenter')`. Refer to spaCy's updated documentation for custom pipeline components.
gotcha Segmentation results might differ slightly when using `pysbd.Segmenter` directly compared to using `pysbd` as a spaCy pipeline component, especially with quoted text or complex punctuation.
Fix: If precise, identical segmentation is critical across different integration methods, thoroughly test both approaches with your specific text data. Direct usage (`pysbd.Segmenter().segment(text)`) generally aligns with the expected rule-based output.
gotcha The `clean=True` parameter in the `Segmenter` constructor performs aggressive pre-filtering of the input text, removing repeated punctuation, line breaks, URLs, and HTML tags. This might alter the original text more than desired for certain NLP tasks.
Fix: If you need to preserve the original text as much as possible, set `clean=False` (which is the default in some examples but not universally). Implement custom cleaning steps if specific pre-processing is required.
gotcha By default, `pysbd.Segmenter` returns a list of strings. If you require character offsets into the original text for non-destructive tokenization, you must initialize `Segmenter` with `char_span=True`.
Fix: Initialize the segmenter as `segmenter = pysbd.Segmenter(language='en', char_span=True)`. The output will then be a list of `TextSpan` objects, each containing the segmented sentence, its start, and end character indices.
gotcha While highly accurate, `pysbd` is a rule-based system implemented in Python. It may be slower compared to some sentence boundary detection alternatives that are implemented in lower-level languages like C++ or optimized using Cython.
Fix: Consider performance implications for very large text corpora or real-time applications. Benchmark `pysbd` against other tools if speed is a primary concern. For high accuracy where speed is not the absolute bottleneck, `pysbd` remains a strong choice.

Install

pip install pysbd PyPI
conda install anaconda::pysbd Conda

Imports

Segmenter
```
import pysbd; segmenter = pysbd.Segmenter(...)
```
Primary class for sentence segmentation
PySBDFactory
```
from pysbd.utils import PySBDFactory
```
PySBDFactory is specifically for spaCy integration and resides in pysbd.utils, not directly under pysbd or pysbd.segmenter.

Quickstart

This example demonstrates basic sentence segmentation using the `Segmenter` class for English text. The `clean=False` parameter is used to prevent aggressive text cleaning.

import pysbd

text = "Dr. Smith went to the U.S. last week. He said, 'Hello!' How are you?"

# Initialize segmenter for English
segmenter = pysbd.Segmenter(language="en", clean=False)

sentences = segmenter.segment(text)

for i, sent in enumerate(sentences):
    print(f"Sentence {i+1}: {sent}")

view raw JSON →