dlt (data load tool)

1.24.0 · active · verified Thu Apr 09

dlt is an open-source python-first scalable data loading library that does not require any backend to run. It simplifies data ingestion from various sources to analytical destinations, handling schema evolution, retries, and state management. dlt releases new versions frequently, approximately monthly, often including breaking changes.

Warnings

breaking Pydantic v1 support was removed in dlt 1.22.0. The library now exclusively requires Pydantic v2.
Fix: Upgrade Pydantic to version 2 (`pip install 'pydantic>=2'`). Review your code for any Pydantic v1 specific patterns or configurations that may need updating.
breaking The legacy Streamlit-based pipeline dashboard (`dlt pipeline show`) was removed in dlt 1.23.0.
Fix: The `dlt pipeline show` command no longer functions. Users should adapt to using the new Marimo-based interactive widgets (introduced in 1.22.2) or use `dlt inspect` for command-line pipeline details.
breaking Custom resource metrics within the trace object are now stored in a table format, altering their structure and location in dlt 1.24.0.
Fix: If your code reads incremental metrics from dlt's trace object, you will need to update it to access the new table-based structure and location of these metrics.
breaking The `data_type` contract's semantic changed in dlt 1.22.0. It now applies to the full data type (including precision, nullability), not just variant columns.
Fix: If you used `data_type: freeze` and expected to change properties like `nullable`, `precision`, or `scale` on existing columns, these will now be treated as breaking changes. Review your schema definitions and data type contracts.
breaking dlt 1.23.0 introduced a new compact source configuration lookup path (`sources.<name>.<key>`). This changes config resolution logic.
Fix: While `dlt` still supports the full `sources.<section>.<name>.<key>` path, the new compact lookup may cause unexpected configuration resolution or overrides if not carefully managed. Review your configuration files and ensure keys are resolved as intended.

Install

pip install dlt Install dlt

Imports

dlt
```
import dlt
```
pipeline
```
from dlt import pipeline
```
source
```
from dlt import source
```
resource
```
from dlt import resource
```

Quickstart

This quickstart demonstrates how to create a simple dlt pipeline to extract GitHub issues and load them into a DuckDB destination. Ensure you have `duckdb` installed (`pip install dlt[duckdb]`) and set a `GITHUB_TOKEN` environment variable if you hit rate limits.

import dlt
import os
from dlt.sources.helpers import requests

# Define a dlt source using a decorator
@dlt.source
def github_issues_source(token):
    # Define a dlt resource within the source
    @dlt.resource(write_disposition="append")
    def issues(owner, repo):
        headers = {"Authorization": f"token {token}"}
        url = f"https://api.github.com/repos/{owner}/{repo}/issues"
        # Fetching a single page for demonstration
        response = requests.get(url, headers=headers, params={'per_page': 10})
        response.raise_for_status()
        yield response.json()

    return issues

# Instantiate and run the pipeline
pipeline = dlt.pipeline(
    pipeline_name="github_pipeline",
    destination="duckdb", # Or "bigquery", "snowflake", etc.
    dataset_name="github_data"
)

load_info = pipeline.run(
    github_issues_source(token=os.environ.get('GITHUB_TOKEN', ''))("dlt-hub", "dlt")
)

# Print the outcome
print(load_info)

view raw JSON →