TorchText

Warnings

• breaking TorchText development has stopped, and the 0.18 release is announced as the last stable release. No new features are anticipated, and the library is in maintenance mode.
  Fix: Users should plan to transition to alternative NLP libraries for active development or use TorchText 0.18.0 as a stable, but no longer actively developed, base. For ongoing development, consider PyTorch's native `torch.utils.data` components combined with custom text processing.
• breaking The legacy `torchtext.data.Field` and `Iterator` API was replaced with a more modular approach to align with `torch.utils.data.DataLoader`. This change provides clearer, more flexible components for tokenization, vocabulary, and batching.
  Fix: Replace `Field` with explicit steps involving `get_tokenizer`, `build_vocab_from_iterator`, and a custom `collate_fn` for `torch.utils.data.DataLoader`. Legacy components are available under `torchtext.legacy.data` if temporary compatibility is needed.
• gotcha TorchText releases are tightly coupled with specific PyTorch versions. Installing an incompatible `torch` and `torchtext` version can lead to installation failures or runtime errors.
  Fix: Always install `torchtext` after verifying compatibility with your `torch` version. Refer to the official compatibility matrix or release notes; for example, TorchText 0.18.0 is compatible with PyTorch 2.3.0.
• gotcha Some built-in datasets may have outdated or broken download URLs, making them inaccessible (e.g., Multi30k was reported in older release notes).
  Fix: If a dataset fails to load, check TorchText's GitHub issues for updated URLs or alternative download methods. Users may need to manually download and process data, or consider using other dataset libraries.

Install

• pip install torchtext Latest stable release

Imports

• get_tokenizer

  from torchtext.data.utils import get_tokenizer

  Modern approach for tokenization.
• build_vocab_from_iterator

  from torchtext.vocab import build_vocab_from_iterator

  Modern approach for vocabulary creation.
• Vocab

  from torchtext.vocab import Vocab

  For directly instantiating a vocabulary object.
• AG_NEWS

  from torchtext.datasets import AG_NEWS

  Example of importing a built-in dataset.
• transforms

  from torchtext import transforms

  For common text-processing transformations.
• models

  from torchtext import models

  For pre-trained models like T5 or RoBERTa.
• Field

  from torchtext.legacy import data as legacy_data

  The `Field` class (and `Iterator`/`BucketIterator`) were part of the legacy API. They coupled tokenizer, vocabulary, split, batching, and numericalization into a 'black box', which was replaced by a more modular approach. Legacy components are now found under `torchtext.legacy.data`.

Quickstart

This quickstart demonstrates the modern TorchText API for text classification. It covers accessing a raw dataset, building a vocabulary, defining text processing pipelines using `get_tokenizer` and `build_vocab_from_iterator`, and finally using `torch.utils.data.DataLoader` with a custom `collate_fn` for batching, padding, and numericalization.

import torch
from torchtext.datasets import AG_NEWS
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator
from torch.utils.data import DataLoader

def yield_tokens(data_iter, tokenizer):
    for _, text in data_iter:
        yield tokenizer(text)

def collate_batch(batch, vocab, tokenizer):
    label_list, text_list, offsets = [], [], [0]
    for (_label, _text) in batch:
        label_list.append(int(_label) - 1)
        processed_text = torch.tensor(vocab(tokenizer(_text)), dtype=torch.int64)
        text_list.append(processed_text)
        offsets.append(processed_text.size(0))
    label_list = torch.tensor(label_list, dtype=torch.int64)
    offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)
    text_list = torch.cat(text_list)
    return label_list, text_list, offsets

# 1. Access the raw dataset iterators
train_iter = AG_NEWS(split='train')
test_iter = AG_NEWS(split='test')

# 2. Prepare data processing pipelines
tokenizer = get_tokenizer('basic_english')

# Build vocabulary
vocab = build_vocab_from_iterator(
    yield_tokens(train_iter, tokenizer),
    min_freq=1,
    specials=['<unk>']
)
vocab.set_default_index(vocab['<unk>'])

# Re-initialize iterators for vocabulary (if needed, or use a cached list)
train_iter = AG_NEWS(split='train')
test_iter = AG_NEWS(split='test')

# Create a partial function for collate_batch with vocab and tokenizer
current_collate_batch = lambda batch: collate_batch(batch, vocab, tokenizer)

# 3. Generate data batch and iterator with DataLoader
BATCH_SIZE = 64

train_dataloader = DataLoader(
    list(train_iter), # Convert to list for Map-style dataset behavior
    batch_size=BATCH_SIZE,
    shuffle=True,
    collate_fn=current_collate_batch
)

test_dataloader = DataLoader(
    list(test_iter), # Convert to list for Map-style dataset behavior
    batch_size=BATCH_SIZE,
    shuffle=False,
    collate_fn=current_collate_batch
)

# Example usage:
for i, (labels, texts, offsets) in enumerate(train_dataloader):
    if i == 0:
        print(f"Batch {i+1}:")
        print(f"  Labels: {labels}")
        print(f"  Texts (concatenated token IDs): {texts}")
        print(f"  Offsets (start index of each text in 'texts'): {offsets}")
        break