Airbyte Declarative Manifest Source

6.5.2 · active · verified Thu Apr 16

This library serves as the base source implementation for low-code Airbyte connectors, allowing users to define data sources using a YAML-based declarative manifest. It interprets these manifests to run data synchronization processes without requiring extensive Python coding. The current version is 6.5.2, and it follows the Airbyte platform's rapid release cadence, with major platform updates typically monthly, influencing the underlying CDK and manifest capabilities.

Common errors

```
yaml.parser.ParserError: while parsing a block mapping
```
cause The YAML manifest file has incorrect syntax, often due to indentation errors, missing colons, or improper key-value structure.

fix Use a YAML linter (e.g., `yamllint`) to identify syntax errors. Pay close attention to indentation, which is crucial in YAML. Ensure all keys are followed by a colon and a space.
```
airbyte_cdk.sources.declarative.exceptions.ComponentNotFoundException: Component 'MyCustomComponent' not found in registry.
```
cause The declarative manifest references a component (e.g., a retriever, authenticator, or transformation) that is either misspelled or not available in the installed version of `airbyte-source-declarative-manifest` or `airbyte-cdk`.

fix Double-check the component name against Airbyte's official documentation for declarative components. Ensure your `airbyte-cdk` and `airbyte-source-declarative-manifest` packages are up to date and compatible with the components you are trying to use.
```
Failed to authenticate with OAuth 2.0 client credentials.
```
cause The OAuth 2.0 configuration in the declarative manifest is incorrect, possibly due to outdated fields or misconfigured parameters after the introduction of Declarative OAuth 2.0.

fix Review the Airbyte documentation for Declarative OAuth 2.0 for the correct manifest structure and required parameters. Verify your client ID, client secret, and refresh token (if applicable) are correctly configured in the manifest or environment variables.

Warnings

breaking Declarative OAuth 2.0 was introduced in Airbyte 1.5.0. Existing declarative connectors using older OAuth configurations may require manifest updates to align with the new, standardized OAuth 2.0 flow, particularly within the connector builder context.
Fix: Consult Airbyte's official documentation for Declarative OAuth 2.0 to update your connector manifest's authentication section. Ensure `airbyte-cdk` and `airbyte-source-declarative-manifest` are updated to compatible versions.
gotcha The schema for declarative manifests evolves. A manifest created for an older `airbyte-cdk` or `airbyte-source-declarative-manifest` version might become incompatible with newer versions due to changes in component definitions, available features, or required configurations.
Fix: Regularly check Airbyte's release notes and declarative connector documentation when upgrading `airbyte-cdk` or `airbyte-source-declarative-manifest`. Validate your manifest against the latest schema, especially for breaking changes in components or data types.
gotcha Debugging declarative sources can be more challenging than programmatic ones. Errors often manifest as cryptic YAML parsing issues, misconfigurations of declarative components, or unexpected behavior due to implicit logic, rather than explicit Python stack traces.
Fix: Thoroughly validate your YAML manifest using a linter. Utilize Airbyte's connector builder for initial development and testing. Increase logging verbosity and examine Airbyte job logs carefully for clues related to manifest parsing or component execution.

Install

pip install airbyte-source-declarative-manifest Install latest version

Imports

DeclarativeSource
wrong:
```
from airbyte_source_declarative_manifest import DeclarativeSource
```
correct:
```
from airbyte_cdk.sources.declarative.declarative_source import DeclarativeSource
```
The `airbyte-source-declarative-manifest` package provides the underlying engine; users interact with it primarily through the `airbyte-cdk`'s `DeclarativeSource` class to define and run connectors.

Quickstart

This quickstart demonstrates how to programmatically use the `DeclarativeSource` from `airbyte-cdk` to load and interact with a declarative manifest. This is how the `airbyte-source-declarative-manifest` library is implicitly used. It defines a minimal YAML manifest, saves it, and then runs a 'check' operation against it. In a real Airbyte deployment, the platform would handle loading the manifest and invoking the source.

import json
import os
from pathlib import Path

from airbyte_cdk.sources.declarative.declarative_source import DeclarativeSource

# Create a dummy manifest file for demonstration
manifest_content = """
version: "0.3.0"
resolver:
  type: "inline"
schemas:
  streams:
    - name: "example_stream"
      retriever:
        type: "simple"
        requester:
          url_base: "https://jsonplaceholder.typicode.com"
          path: "/todos/1"
          http_method: "GET"
          authenticator:
            type: "no_auth"
        record_selector:
          field_path: []
      stream_mapper:
        type: "simple"
      primary_key: []
      transformations: []
"""

# Save the manifest to a temporary file
manifest_path = Path("temp_manifest.yaml")
manifest_path.write_text(manifest_content)

# Define the config for the source (can be empty if not needed by manifest)
config = {}

# Initialize the declarative source
source = DeclarativeSource(str(manifest_path))

# Run a check operation (or other CDK operations like discover, read)
# This is how Airbyte internally invokes the source.
status = source.check(None, config)

print(f"Source check status: {status.status.value}")

# Clean up the temporary manifest file
manifest_path.unlink()

view raw JSON →