seqeval: Sequence Labeling Evaluation

1.2.2 · active · verified Sun Apr 12

seqeval is a Python framework for sequence labeling evaluation. It provides metrics like F1 score, precision, recall, and a detailed classification report for tasks such as named-entity recognition and part-of-speech tagging. It is currently at version 1.2.2 and maintains an active release cadence, with updates often focusing on performance improvements and additional evaluation schemes.

Warnings

breaking The `classification_report` behavior changed significantly in v1.0.0. It now allows explicit specification of the evaluation scheme (e.g., IOB1, IOB2, BILOU) which can alter how entities are counted. Older code relying on implicit scheme assumptions might produce different results.
Fix: Review calls to `classification_report` and explicitly pass the desired `scheme` and `mode` parameters (e.g., `classification_report(y_true, y_pred, scheme=IOB2, mode='strict')`) to ensure correct evaluation behavior. Refer to the documentation for supported schemes.
gotcha seqeval's metrics (especially F1, precision, recall) are calculated differently from `scikit-learn`'s for sequence labeling tasks. `seqeval` specifically evaluates *entities*, primarily focusing on non-'O' (Outside) tags, and does not count correctly predicted 'O' tags as true positives, only misclassified ones. `scikit-learn`'s metrics, when applied naively to token-level tags, will include all 'O' tags in its calculations, potentially leading to inflated scores if 'O' tags are abundant and correctly predicted.
Fix: Be aware of this fundamental difference when comparing evaluation results. For true entity-level evaluation in NLP sequence labeling, `seqeval` is generally preferred. If comparing with `scikit-learn`, ensure a clear understanding of what each metric is actually measuring.
gotcha When using `mode='strict'` (e.g., in `f1_score` or `classification_report`), only exact matches for entity spans (including both boundaries and type) are considered correct. This is often the desired behavior for robust evaluation but can result in lower scores compared to the default mode which might be more lenient.
Fix: Understand the implications of `mode='strict'`. If evaluating with `mode='strict'`, ensure your `y_true` and `y_pred` are consistent with a strict tagging scheme (like IOB2 or BILOU) and that this level of strictness is appropriate for your task.
gotcha Input `y_true` and `y_pred` for all `seqeval.metrics` functions must be lists of lists of *strings* (e.g., `[['B-PER', 'I-PER', 'O']]`), representing the sequence tags. Providing numerical label IDs instead of string tags will result in errors.
Fix: Ensure that your true and predicted labels are properly converted from numerical IDs to their corresponding string tags (e.g., 'O', 'B-PER', 'I-PER') before passing them to `seqeval` metric functions.
gotcha Performance of evaluation, particularly in `strict` mode, was significantly improved in version 1.2.1. Older versions might exhibit slower computation times for large datasets or extensive evaluations.
Fix: Upgrade to `seqeval` version 1.2.1 or newer to benefit from performance enhancements, especially if you are performing evaluations in `strict` mode or on large datasets.

Install

pip install seqeval Install stable version

Imports

accuracy_score

from seqeval.metrics import accuracy_score

precision_score

from seqeval.metrics import precision_score

recall_score

from seqeval.metrics import recall_score

f1_score
```
from seqeval.metrics import f1_score
```

classification_report

from seqeval.metrics import classification_report

IOB2
```
from seqeval.scheme import IOB2
```
Used to specify a tagging scheme for metrics when needed.

Quickstart

Calculate the F1-score and generate a detailed classification report for sequence labeling predictions.

from seqeval.metrics import f1_score
from seqeval.metrics import classification_report

y_true = [['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'O', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'B-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]

# Compute F1-score (micro average by default)
print(f"F1 Score (micro): {f1_score(y_true, y_pred):.2f}")

# Compute F1-score with different averaging methods
print(f"F1 Score (average=None): {f1_score(y_true, y_pred, average=None)}")

# Generate a full classification report
report = classification_report(y_true, y_pred, digits=2)
print("\nClassification Report:\n", report)

view raw JSON →