Natural Language Toolkit (NLTK)

3.9.4 · active · verified Sat Mar 28

NLTK (Natural Language Toolkit) is a leading open-source Python library for Natural Language Processing (NLP). It provides easy-to-use interfaces to over 50 corpora and lexical resources, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning. Currently at version 3.9.4, NLTK generally follows a release cadence of a few minor versions per year, with more significant updates addressing security and Python compatibility as needed.

Warnings

breaking NLTK 3.9 introduced a breaking change by replacing pickled models (e.g., for `punkt`, chunkers, taggers) with new pickle-free `_tab` packages to fix security vulnerability CVE-2024-39705. Older versions using pickled models may be insecure or incompatible with newer NLTK versions.
Fix: Upgrade NLTK to version 3.9 or higher. Ensure your application is updated to use the new `_tab` packages or re-download corpora with `nltk.download()` after upgrading. Specifically, NLTK 3.9.3 fixed CVE-2025-14009 related to secure ZIP extraction.
gotcha Many NLTK functionalities (e.g., tokenizers, taggers, corpora) require downloading specific datasets. Failing to download them will result in `Resource Not Found` errors. Running `nltk.download('all')` can be resource-intensive and unsuitable for production environments.
Fix: Before using a specific NLTK module that relies on external data, ensure the necessary data is downloaded. For production, explicitly download only the required packages using `nltk.download('package_name')` once during setup, or use `nltk.data.path.append('/path/to/nltk_data')` to point to pre-downloaded data. For example, `nltk.download('punkt')` for the Punkt tokenizer.

Install

pip install nltk Install NLTK

Imports

nltk
```
import nltk
```
Most common and provides access to core functionalities and submodules like nltk.word_tokenize, nltk.pos_tag.
word_tokenize
```
from nltk.tokenize import word_tokenize
```
Direct import for specific tokenization functions.
PorterStemmer
```
from nltk.stem import PorterStemmer
```
Direct import for a specific stemmer.

Quickstart

This quickstart demonstrates basic text tokenization and Part-of-Speech (POS) tagging using NLTK. It includes checks to download the 'punkt' tokenizer and 'averaged_perceptron_tagger' if they are not already present, which are common requirements for many NLTK operations. This ensures the example is runnable out-of-the-box.

import nltk
from nltk.tokenize import word_tokenize
from nltk.tag import pos_tag

# Download necessary NLTK data (run once)
try:
    nltk.data.find('tokenizers/punkt')
except nltk.downloader.DownloadError:
    nltk.download('punkt')
try:
    nltk.data.find('taggers/averaged_perceptron_tagger')
except nltk.downloader.DownloadError:
    nltk.download('averaged_perceptron_tagger')

text = "NLTK is a powerful library for natural language processing."

# Tokenization
tokens = word_tokenize(text)
print(f"Tokens: {tokens}")

# Part-of-Speech Tagging
tagged_tokens = pos_tag(tokens)
print(f"POS Tagged: {tagged_tokens}")

view raw JSON →