Hugging Face Datasets

Warnings

• breaking datasets 4.0 (July 2025) removed all support for dataset loading scripts (.py files) and the trust_remote_code parameter. Datasets that relied on custom .py loaders now raise: 'RuntimeError: Dataset scripts are no longer supported, but found X.py'. This breaks many community datasets (hotpotqa, common_voice, superb, gaia-benchmark, etc.).
  Fix: Either: (1) pin datasets<4 as a workaround, or (2) find a Parquet-backed version of the dataset on the Hub, or (3) ask the dataset author to migrate to standard Parquet format. Passing trust_remote_code=True no longer silences the error — it raises a separate error.
• breaking trust_remote_code parameter is entirely removed in datasets 4.0. Passing it raises an error rather than being silently ignored. Code with trust_remote_code=True will break on import or call.
  Fix: Remove trust_remote_code from all load_dataset() calls when using datasets>=4.
• breaking pyarrow version constraints are strict. datasets pins to specific pyarrow ranges. In environments with multiple packages requiring pyarrow, version conflicts cause ImportError or silent data corruption. datasets and pyarrow must be upgraded together.
  Fix: Always upgrade together: pip install -U datasets pyarrow. Check compatibility in the datasets changelog for your target version.
• gotcha map() with num_proc>1 (multiprocessing) uses dill for serialization. Lambda functions and closures that reference non-serializable objects (open file handles, locks, etc.) will silently fail or hang. No clear error is raised.
  Fix: Use named functions instead of lambdas for map(). Avoid referencing non-serializable objects inside map functions. Use batched=True for large datasets to avoid per-example overhead.
• gotcha load_dataset() caches datasets to disk by default in HF_HOME. Re-running always returns the cached version. In CI or when dataset content changes on the Hub, stale cached versions are silently returned.
  Fix: Pass download_mode='force_redownload' to bypass cache. Or delete the cached dataset from ~/.cache/huggingface/datasets/.
• gotcha Gated datasets (some Common Voice, medical, legal datasets) require authentication. load_dataset() raises a 401 or confusing FileNotFoundError if HF_TOKEN is not set or the license has not been accepted.
  Fix: Set HF_TOKEN env var and accept the dataset license on huggingface.co before calling load_dataset().
• gotcha Package name on PyPI is 'datasets' — not 'huggingface-datasets'. pip install huggingface-datasets installs an old, unrelated stub package. Import is also 'from datasets import ...' not 'from huggingface_datasets import ...'.
  Fix: pip install datasets. from datasets import load_dataset.

Install

• pip install datasets Core install
• pip install datasets[audio] With audio support (librosa, soundfile)
• pip install datasets[vision] With image support (Pillow)
• pip install datasets[torch] With PyTorch tensor format support
• pip install 'datasets<4' Pin to v3.x if you need trust_remote_code or .py loading scripts

Imports

• load_dataset

  from datasets import load_dataset

  Package name is 'datasets' on PyPI, not 'huggingface-datasets'. pip install datasets. from datasets import load_dataset.
• Dataset / DatasetDict

  from datasets import Dataset, DatasetDict

  load_dataset() returns DatasetDict (multiple splits) or Dataset (single split). Access splits: ds['train'], ds['test'].
• Audio / Image features

  from datasets import Audio, Image

  Cast columns to Audio() or Image() features to enable automatic decoding. Requires datasets[audio] or datasets[vision] extras.

Quickstart

load_dataset() downloads and caches to HF_HOME. Streaming mode avoids full download for large datasets. map() with batched=True is significantly faster for large datasets. set_format() enables framework-specific tensor output without copying data.

from datasets import load_dataset

# Load public dataset from Hub
ds = load_dataset("rajpurkar/squad")  # returns DatasetDict
print(ds)  # DatasetDict with 'train' and 'validation' splits
print(ds['train'][0])  # first example

# Streaming mode (no full download)
streaming_ds = load_dataset("rajpurkar/squad", split="train", streaming=True)
for example in streaming_ds.take(3):
    print(example['question'])

# Load local files
local_ds = load_dataset("json", data_files="./my_data.jsonl", split="train")

# map() for preprocessing
def tokenize(example):
    return {"tokens": example["text"].split()}

processed = ds['train'].map(tokenize, batched=False)

# Convert to PyTorch tensors
ds['train'].set_format(type='torch', columns=['input_ids', 'attention_mask'])

# Create dataset from dict
from datasets import Dataset
custom_ds = Dataset.from_dict({"text": ["hello", "world"], "label": [0, 1]})