{"id":5791,"library":"openmetadata-ingestion","title":"OpenMetadata Ingestion Framework","description":"OpenMetadata Ingestion is a Python framework designed to build connectors and ingest metadata from various external systems into an OpenMetadata instance via its APIs. It is commonly used within orchestration frameworks like Apache Airflow to automate metadata extraction and is actively maintained with frequent releases, often tied to OpenMetadata server versions. The current version is 1.12.5.1.","status":"active","version":"1.12.5.1","language":"en","source_language":"en","source_url":"https://github.com/open-metadata/OpenMetadata","tags":["metadata","data governance","data catalog","etl","data ingestion","data quality"],"install":[{"cmd":"pip install \"openmetadata-ingestion==1.12.5.1\"","lang":"bash","label":"Basic Installation"},{"cmd":"pip install \"openmetadata-ingestion[all]==1.12.5.1\"","lang":"bash","label":"Installation with All Connectors"},{"cmd":"pip install \"openmetadata-ingestion[postgres,snowflake]==1.12.5.1\"","lang":"bash","label":"Installation with specific database connectors (e.g., PostgreSQL, Snowflake)"}],"dependencies":[{"reason":"Required for running the ingestion framework; officially supports Python 3.9, 3.10, and 3.11.","package":"Python","optional":false}],"imports":[{"symbol":"MetadataWorkflow","correct":"from metadata.workflow.metadata import MetadataWorkflow"},{"symbol":"OpenMetadata","correct":"from metadata.ingestion.ometa.ometa_api import OpenMetadata"}],"quickstart":{"code":"import os\nimport yaml\nfrom metadata.workflow.metadata import MetadataWorkflow\n\n# Define your OpenMetadata server connection and authentication\n# It's recommended to use environment variables for sensitive data like JWT tokens\nopenmetadata_server_config = {\n    \"hostPort\": \"http://localhost:8585/api\",\n    \"authProvider\": \"openmetadata\",\n    \"securityConfig\": {\n        \"jwtToken\": os.environ.get('OPENMETADATA_JWT_TOKEN', 'YOUR_OM_JWT_TOKEN_HERE')\n    }\n}\n\n# Example: A minimal YAML configuration for ingesting metadata from a dummy source\n# In a real scenario, this would be a full connector config (e.g., MySQL, Snowflake)\nworkflow_config_yaml = f\"\"\"\nsource:\n  type: \"sample-data\"\n  serviceName: \"sample_metadata\"\nsink:\n  type: \"metadata-rest\"\n  config: {}\nworkflowConfig:\n  openMetadataServerConfig:\n    hostPort: {openmetadata_server_config['hostPort']}\n    authProvider: {openmetadata_server_config['authProvider']}\n    securityConfig:\n      jwtToken: {openmetadata_server_config['securityConfig']['jwtToken']}\n\"\"\"\n\n# Load the YAML configuration\nworkflow_config = yaml.safe_load(workflow_config_yaml)\n\n# Create and execute the workflow\nprint(\"Starting Metadata Ingestion Workflow...\")\nworkflow = MetadataWorkflow.create(workflow_config)\ntry:\n    workflow.execute()\n    print(\"Metadata Ingestion Workflow completed successfully.\")\nexcept Exception as err:\n    print(f\"Error during metadata ingestion: {err}\")\nfinally:\n    workflow.print_status()\n    workflow.stop()\n","lang":"python","description":"This quickstart demonstrates a basic metadata ingestion workflow using the `MetadataWorkflow` class. It shows how to define a minimal workflow configuration in YAML, including the OpenMetadata server connection details and using an environment variable for the JWT token for authentication. Replace `sample-data` with your actual connector configuration (e.g., `mysql`, `snowflake`). Ensure your OpenMetadata server is running and accessible."},"warnings":[{"fix":"Ensure that `openmetadata-ingestion` is installed with a version compatible with your OpenMetadata server version, typically using a tilde-equals `~=` version specifier (e.g., `pip install \"openmetadata-ingestion~=1.12.5\"`).","message":"A version mismatch between the `openmetadata-ingestion` Python package and the OpenMetadata server can lead to `ClientInitializationError`, `ValidationError`, or other unexpected behavior. The ingestion package version (e.g., `1.12.5.1`) must match the server version (e.g., `~=1.12.5`).","severity":"breaking","affected_versions":"All versions"},{"fix":"Always upgrade the `openmetadata-ingestion` package to match your OpenMetadata server version, especially after server upgrades, to ensure schema compatibility.","message":"Schema changes introduced in OpenMetadata minor releases (e.g., adding `queryStatementSource`, `queryParserConfig`, `statusLookbackDays` fields) require upgrading the ingestion framework. Running an older ingestion framework with a newer server can cause `ValidationError` during pipeline deployment or Airflow DAG build failures.","severity":"breaking","affected_versions":">=1.11.10"},{"fix":"Update your OpenLineage Kinesis connector configurations to use the new `brokerConfig` object structure.","message":"The OpenLineage Kinesis connection schema was refactored in version 1.11.12. Existing configurations with flat Kafka-specific fields (e.g., `brokersUrl`, `topicName`) must be migrated to a nested `brokerConfig` object.","severity":"breaking","affected_versions":">=1.11.12"},{"fix":"Refer to the official documentation for the updated REST connector connection structure and adjust your YAML configurations accordingly.","message":"The REST connector configuration changed in version 1.11.11 to support OpenAPI schemas from local files (JSON/YAML) in addition to HTTP URLs via the `openAPISchemaConnection` union type. Existing configurations might require updates.","severity":"breaking","affected_versions":">=1.11.11"},{"fix":"Ensure your Python environment is running a supported version (3.9, 3.10, or 3.11). It's recommended to use a virtual environment.","message":"The `openmetadata-ingestion` library requires Python versions 3.9, 3.10, or 3.11. Using unsupported Python versions might lead to compatibility issues or errors.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-14T00:00:00.000Z","next_check":"2026-07-13T00:00:00.000Z"}