Ibis: The Portable Python Dataframe Library

12.0.0 · active · verified Sat Apr 11

Ibis is a portable Python dataframe library that provides a Pythonic way to build and execute operations on data in various backends, including SQL databases, data warehouses, and data lakes. It offers a familiar dataframe API that compiles into the backend's native language, enabling local iteration and remote deployment by changing a single line of code. It is currently at version 12.0.0 and maintains an active release cadence with frequent updates.

Warnings

breaking Python 3.9 is no longer supported. Users must upgrade to Python 3.10 or newer.
Fix: Upgrade your Python environment to 3.10 or a later version.
breaking PySpark versions older than 3.5 are no longer supported when using the PySpark backend.
Fix: If using the PySpark backend, ensure PySpark is version 3.5 or newer.
breaking The `DataType.name` attribute has been removed. Use `DataType.__class__.__name__` instead to get the string name of a data type.
Fix: Replace `dt.name` with `dt.__class__.__name__` where `dt` is an Ibis DataType object.
breaking Explicit naming of `memtable`s is no longer supported. Use `create_table` or `create_view` for named objects.
Fix: Instead of explicitly naming ephemeral in-memory tables, use `connection.create_table()` or `connection.create_view()` for persistent or named views.
gotcha Ibis uses lazy evaluation. Operations define an expression graph but do not execute immediately. To get results, you must explicitly call methods like `.execute()` or `.to_pandas()`.
Fix: Always append `.execute()` or `.to_pandas()` (or similar materialization methods) to your Ibis expression chain to trigger computation and retrieve results.
gotcha There is a different PyPI package also named `ibis` which is a web templating framework. Ensure you install `ibis-framework` for the dataframe library.
Fix: Always use `pip install ibis-framework` (with relevant extras) or `conda install ibis-framework` to install the correct library.

Install

pip install 'ibis-framework[duckdb,examples]' Recommended for local development with example data
pip install ibis-framework Core library only
pip install 'ibis-framework[all]' All backends (large install)

Imports

ibis
```
import ibis
```
Installing 'ibis' (a web templating framework) instead of 'ibis-framework' (the dataframe library) is a common mistake. Always install 'ibis-framework' from PyPI.
ibis.options.interactive
```
ibis.options.interactive = True
```
Enables eager evaluation for interactive exploration, though it can be slower for production workloads and is not strictly necessary.

Quickstart

This quickstart demonstrates how to connect to an in-memory DuckDB backend, load an example dataset, define a lazy dataframe expression to calculate the average bill length per penguin species, and then execute it to retrieve the results as a pandas DataFrame. It showcases Ibis's core lazy evaluation pattern.

import ibis

# Connect to an in-memory DuckDB database (default backend)
con = ibis.duckdb.connect(':memory:')

# Load example data (e.g., the 'penguins' dataset)
t = ibis.examples.penguins.fetch()

# Create a table in the connected database
con.create_table('penguins', t.to_pyarrow(), overwrite=True)

# Get a table expression from the connection
table = con.table('penguins')

# Perform a lazy computation: group by species and calculate mean bill length
result_expr = table.group_by('species').agg(avg_bill_length=table.bill_length_mm.mean())

# Execute the expression and fetch results into a pandas DataFrame
df = result_expr.to_pandas()

print(df)

view raw JSON →