{"id":1462,"library":"dlt","title":"dlt (data load tool)","description":"dlt is an open-source python-first scalable data loading library that does not require any backend to run. It simplifies data ingestion from various sources to analytical destinations, handling schema evolution, retries, and state management. dlt releases new versions frequently, approximately monthly, often including breaking changes.","status":"active","version":"1.24.0","language":"en","source_language":"en","source_url":"https://github.com/dlt-hub/dlt","tags":["ETL","ELT","data loading","data pipeline","data engineering","orchestration"],"install":[{"cmd":"pip install dlt","lang":"bash","label":"Install dlt"}],"dependencies":[],"imports":[{"symbol":"dlt","correct":"import dlt"},{"symbol":"pipeline","correct":"from dlt import pipeline"},{"symbol":"source","correct":"from dlt import source"},{"symbol":"resource","correct":"from dlt import resource"}],"quickstart":{"code":"import dlt\nimport os\nfrom dlt.sources.helpers import requests\n\n# Define a dlt source using a decorator\n@dlt.source\ndef github_issues_source(token):\n    # Define a dlt resource within the source\n    @dlt.resource(write_disposition=\"append\")\n    def issues(owner, repo):\n        headers = {\"Authorization\": f\"token {token}\"}\n        url = f\"https://api.github.com/repos/{owner}/{repo}/issues\"\n        # Fetching a single page for demonstration\n        response = requests.get(url, headers=headers, params={'per_page': 10})\n        response.raise_for_status()\n        yield response.json()\n\n    return issues\n\n# Instantiate and run the pipeline\npipeline = dlt.pipeline(\n    pipeline_name=\"github_pipeline\",\n    destination=\"duckdb\", # Or \"bigquery\", \"snowflake\", etc.\n    dataset_name=\"github_data\"\n)\n\nload_info = pipeline.run(\n    github_issues_source(token=os.environ.get('GITHUB_TOKEN', ''))(\"dlt-hub\", \"dlt\")\n)\n\n# Print the outcome\nprint(load_info)\n","lang":"python","description":"This quickstart demonstrates how to create a simple dlt pipeline to extract GitHub issues and load them into a DuckDB destination. Ensure you have `duckdb` installed (`pip install dlt[duckdb]`) and set a `GITHUB_TOKEN` environment variable if you hit rate limits."},"warnings":[{"fix":"Upgrade Pydantic to version 2 (`pip install 'pydantic>=2'`). Review your code for any Pydantic v1 specific patterns or configurations that may need updating.","message":"Pydantic v1 support was removed in dlt 1.22.0. The library now exclusively requires Pydantic v2.","severity":"breaking","affected_versions":">=1.22.0"},{"fix":"The `dlt pipeline show` command no longer functions. Users should adapt to using the new Marimo-based interactive widgets (introduced in 1.22.2) or use `dlt inspect` for command-line pipeline details.","message":"The legacy Streamlit-based pipeline dashboard (`dlt pipeline show`) was removed in dlt 1.23.0.","severity":"breaking","affected_versions":">=1.23.0"},{"fix":"If your code reads incremental metrics from dlt's trace object, you will need to update it to access the new table-based structure and location of these metrics.","message":"Custom resource metrics within the trace object are now stored in a table format, altering their structure and location in dlt 1.24.0.","severity":"breaking","affected_versions":">=1.24.0"},{"fix":"If you used `data_type: freeze` and expected to change properties like `nullable`, `precision`, or `scale` on existing columns, these will now be treated as breaking changes. Review your schema definitions and data type contracts.","message":"The `data_type` contract's semantic changed in dlt 1.22.0. It now applies to the full data type (including precision, nullability), not just variant columns.","severity":"breaking","affected_versions":">=1.22.0"},{"fix":"While `dlt` still supports the full `sources.<section>.<name>.<key>` path, the new compact lookup may cause unexpected configuration resolution or overrides if not carefully managed. Review your configuration files and ensure keys are resolved as intended.","message":"dlt 1.23.0 introduced a new compact source configuration lookup path (`sources.<name>.<key>`). This changes config resolution logic.","severity":"breaking","affected_versions":">=1.23.0"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}