{"id":1351,"library":"azure-kusto-ingest","title":"Azure Kusto Ingest Client","description":"The `azure-kusto-ingest` library provides a client for ingesting data into Azure Data Explorer (Kusto) clusters. It supports queued ingestion (batching for high throughput) and streaming ingestion (low latency). The current version is 6.0.3, with frequent bug fix releases and occasional major versions aligning with Python and Azure SDK ecosystem changes. Version 6.0.3 introduced allowing transformation functions for CSV and SCSV formats.","status":"active","version":"6.0.3","language":"en","source_language":"en","source_url":"https://github.com/Azure/azure-kusto-python","tags":["azure","kusto","azure data explorer","data ingestion","cloud","analytics"],"install":[{"cmd":"pip install azure-kusto-ingest","lang":"bash","label":"Base installation"},{"cmd":"pip install azure-kusto-ingest[pandas]","lang":"bash","label":"With Pandas support (for DataFrame ingestion)"}],"dependencies":[{"reason":"Required for `ingest_from_dataframe` functionality, installed via `[pandas]` extra.","package":"pandas","optional":true}],"imports":[{"note":"Used to construct connection strings, part of the core Kusto data library.","symbol":"KustoConnectionStringBuilder","correct":"from azure.kusto.data import KustoConnectionStringBuilder"},{"note":"Main client for queued (batch) data ingestion.","symbol":"KustoIngestClient","correct":"from azure.kusto.ingest import KustoIngestClient"},{"note":"Client for low-latency streaming data ingestion.","symbol":"KustoStreamingIngestClient","correct":"from azure.kusto.ingest import KustoStreamingIngestClient"},{"note":"Configures ingestion behavior like target database/table, data format, and mapping.","symbol":"IngestionProperties","correct":"from azure.kusto.ingest import IngestionProperties"},{"note":"Enum for specifying the format of the ingested data (e.g., CSV, TSV, JSON).","symbol":"DataFormat","correct":"from azure.kusto.data import DataFormat"},{"note":"Used for ingesting data directly from Azure Blob Storage.","symbol":"BlobDescriptor","correct":"from azure.kusto.ingest import BlobDescriptor"}],"quickstart":{"code":"import os\nimport io\nfrom azure.kusto.data import KustoConnectionStringBuilder, DataFormat\nfrom azure.kusto.ingest import KustoIngestClient, IngestionProperties\n\n# Configuration from environment variables (replace with your actual values)\nKUSTO_CLUSTER_URL = os.environ.get('KUSTO_CLUSTER_URL', 'https://yourcluster.region.kusto.windows.net')\nKUSTO_DATABASE = os.environ.get('KUSTO_DATABASE', 'yourdatabase')\nKUSTO_TABLE = os.environ.get('KUSTO_TABLE', 'yourtable')\n\n# Ensure environment variables are set or replace with actual connection string details\n# For AAD app authentication:\n# KCSB = KustoConnectionStringBuilder.with_aad_application_key(\n#     KUSTO_CLUSTER_URL, os.environ.get('KUSTO_CLIENT_ID', ''), os.environ.get('KUSTO_CLIENT_SECRET', ''), os.environ.get('KUSTO_TENANT_ID', '')\n# )\n# For Azure CLI authentication:\nKCSB = KustoConnectionStringBuilder.with_az_cli_authentication(KUSTO_CLUSTER_URL)\n\n# Create an Ingest Client\ningest_client = KustoIngestClient(KCSB)\n\n# Define Ingestion Properties\ningestion_properties = IngestionProperties(\n    database=KUSTO_DATABASE,\n    table=KUSTO_TABLE,\n    data_format=DataFormat.CSV,\n    # For real-time monitoring of ingestion status, set flush_immediately=True\n    # but be aware of performance implications for large batches.\n    flush_immediately=False\n)\n\n# Sample data as a CSV string\ndata_rows = [\n    \"id,name,value\",\n    \"1,TestItemA,100\",\n    \"2,TestItemB,200\"\n]\ndata_csv = \"\\n\".join(data_rows)\n\n# Ingest data from an in-memory stream\nprint(f\"Attempting to ingest data into {KUSTO_DATABASE}.{KUSTO_TABLE}...\")\nwith io.StringIO(data_csv) as stream:\n    result = ingest_client.ingest_from_stream(stream, ingestion_properties)\n\n# For queued ingestion, the result indicates submission, not completion.\n# Actual status must be monitored in Azure Data Explorer using .show commands.\nprint(\"Ingestion job submitted. Check Azure Data Explorer for status details.\")\n# print(f\"Ingestion result object: {result}\") # Uncomment to see the result object\n","lang":"python","description":"This quickstart demonstrates how to perform queued ingestion of in-memory CSV data into an Azure Data Explorer table. It uses Azure CLI authentication for simplicity but can be adapted for other authentication methods. The example sets up the Kusto client, defines ingestion properties, and sends a small CSV string as a stream. Remember to set `KUSTO_CLUSTER_URL`, `KUSTO_DATABASE`, and `KUSTO_TABLE` environment variables."},"warnings":[{"fix":"Upgrade your Python environment to 3.9 or higher. If you must use an older Python version, pin `azure-kusto-ingest` to `<6.0.0` (e.g., `azure-kusto-ingest<6.0.0`).","message":"Starting with version 6.0.0, the minimum supported Python version for `azure-kusto-ingest` (and other Azure SDKs) is Python 3.9. Prior versions supported older Python versions (e.g., v5.x supported Python 3.8, v4.x supported Python 3.7).","severity":"breaking","affected_versions":">=6.0.0"},{"fix":"Review your `KustoConnectionStringBuilder` usage. Use the builder methods (e.g., `with_aad_managed_identity`) instead of relying on parsing specific keyword arguments in the connection string directly. If using older keyword-based strings, downgrade to `<5.0.0`.","message":"Version 5.0.0 introduced breaking changes to `KustoConnectionStringBuilder` keywords, aligning them with other Azure SDKs. Keywords like `msi_auth`, `msi_authentication`, `msi_params`, and `msi_type` were removed from direct parsing.","severity":"breaking","affected_versions":">=5.0.0"},{"fix":"Always use the latest version of `azure-kusto-ingest` with the latest compatible Pandas version. If encountering issues, specifically check the changelog for fixes related to `dataframe_from_result_table` and `ingest_from_dataframe` in your library version relative to your Pandas version. Pinning `pandas` to a known working version might be necessary if unable to update `azure-kusto-ingest`.","message":"Ingesting data from Pandas DataFrames (using `ingest_from_dataframe`) has seen multiple bug fixes across versions related to datetime columns, null values, and specific Pandas versions (e.g., Pandas 3.0). Ensure compatibility.","severity":"gotcha","affected_versions":"<6.0.2"},{"fix":"Upgrade to version `6.0.1` or higher for improved throttling event handling in Managed Streaming. If using streaming ingestion, ensure your cluster is adequately scaled for the expected ingestion rate.","message":"Managed Streaming Ingestion can encounter throttling events. While version 6.0.1 included a fix to handle these events more gracefully, users on older versions might experience issues with managed streaming stability under high load.","severity":"gotcha","affected_versions":"<6.0.1"}],"env_vars":null,"last_verified":"2026-04-09T00:00:00.000Z","next_check":"2026-07-08T00:00:00.000Z"}