Intake

2.0.9 · active · verified Mon Apr 13

Intake is a lightweight Python package for finding, investigating, loading, and distributing data. It provides a common API for loading data from a wide variety of sources (e.g., CSV, NetCDF, SQL, HDF5, Parquet, Zarr) and enables the creation and management of data catalogs. The current version is 2.0.9, and the project is in a stable maintenance phase for its 2.x series, with less frequent but significant updates.

Warnings

breaking Major API changes occurred between Intake 1.x and 2.x, particularly concerning how drivers are accessed and catalog specifications are defined. Directly using `intake.source.<driver>.SourceClass` is deprecated in favor of `intake.open_<format>(...)` functions.
Fix: Update catalog YAML files to the Intake 2.x schema. Use `intake.open_catalog()` for catalogs and `intake.open_csv()`, `intake.open_parquet()`, etc., for direct source access. Consult the official migration guide.
gotcha Intake relies heavily on a plugin system for specific data formats and remote storage. If you try to open a file type (e.g., Parquet, SQL) or access a remote system (e.g., S3) without the corresponding `intake-<plugin_name>` package installed, you will encounter errors.
Fix: Ensure you install the necessary Intake plugins, e.g., `pip install intake-parquet` for Parquet files, `pip install intake-sql` for SQL databases, or `pip install s3fs` for S3 access (often included in `intake[s3]` extras).
gotcha Confusing `intake.open_catalog()` with direct source opening functions like `intake.open_csv()`. `open_catalog` is for loading YAML catalog files (which can contain multiple sources), whereas `open_csv` (and similar) directly open a single data file without a catalog.
Fix: Use `intake.open_catalog()` when you have a YAML file defining multiple data sources or remote catalogs. Use `intake.open_csv()`, `intake.open_parquet()`, etc., for quick, one-off access to individual files.

Install

pip install intake Core library
pip install intake[parquet,s3,sql] With common plugins (example)

Imports

open_catalog

import intake
catalog = intake.open_catalog('my_catalog.yaml')

open_csv

import intake
df = intake.open_csv('data.csv').read()

Quickstart

This quickstart demonstrates how to open a remote Intake catalog and load a dataset (US States data) into a Pandas DataFrame.

import intake

# Open a public example catalog
catalog = intake.open_catalog("https://raw.githubusercontent.com/intake/intake-examples/master/catalogs/us_states.yml")

# Access a data source from the catalog
df = catalog.states.read()

print(df.head())

view raw JSON →