Kedro-Datasets

Warnings

• breaking The `MatplotlibWriter` dataset was removed in `kedro-datasets` version 9.0.0. Its functionality has been absorbed and replaced by `MatplotlibDataset`.
  Fix: Migrate any usage of `MatplotlibWriter` to `MatplotlibDataset`.
• deprecated The `overwrite` argument for `ibis.TableDataset` was deprecated in `kedro-datasets` version 9.0.0. It is mapped to the new `mode` argument for backward compatibility but will be removed in a future release.
  Fix: Replace `overwrite=True/False` with `mode='overwrite'` or `mode='append'` respectively. Supported modes include 'append', 'overwrite', 'error'/'errorifexists', and 'ignore'.
• gotcha Many `kedro-datasets` rely on optional dependencies (extras). If you install `kedro-datasets` without specifying the necessary extras (e.g., `[pandas]`, `[spark]`, `[s3]`), you will encounter `ModuleNotFoundError` or `ImportError` when trying to use datasets that require them.
  Fix: Install `kedro-datasets` with the specific extras needed for your datasets, e.g., `pip install kedro-datasets[pandas,spark]`, or `pip install kedro-datasets[all]` for comprehensive coverage.
• gotcha `kedro-datasets` version 9.3.0 introduced compatibility with pandas 3.0. Users on older `kedro-datasets` versions combined with pandas 3.0 might experience unexpected behavior or errors.
  Fix: Ensure `kedro-datasets` is updated to version 9.3.0 or newer if using pandas 3.0. For older `kedro-datasets` versions, stick to pandas <3.0.
• gotcha New "experimental" datasets are frequently introduced (e.g., in 9.2.0 and 9.3.0). These datasets are subject to change, including API modifications or even removal, without necessarily being flagged as 'breaking changes' in minor versions.
  Fix: When using experimental datasets, monitor release notes for potential changes. For production systems, prefer stable, non-experimental datasets or ensure thorough testing with specific experimental versions.

Install

• pip install kedro-datasets Base installation
• pip install kedro-datasets[all] All optional dependencies
• pip install kedro-datasets[pandas,spark,s3] Common optional dependencies

Imports

• CSVDataset

  from kedro_datasets.pandas import CSVDataset

  Old import path from pre-0.18 Kedro versions or incorrect casing. Datasets are now in `kedro_datasets`.
• SparkDataset

  from kedro_datasets.spark import SparkDataset

• MatplotlibDataset

  from kedro_datasets.matplotlib import MatplotlibDataset

  MatplotlibWriter was removed in 9.0.0; use MatplotlibDataset instead.

Quickstart

This quickstart demonstrates how to programmatically initialize, load, and save data using a common dataset type (CSVDataset) from `kedro-datasets`. While `kedro-datasets` is often used within Kedro project configuration (e.g., `catalog.yml`), direct programmatic usage is also fully supported.

import pandas as pd
import os
from kedro_datasets.pandas import CSVDataset

# 1. Create a dummy CSV file
data = {"col1": [1, 2, 3], "col2": ["A", "B", "C"]}
df = pd.DataFrame(data)
filepath = "my_dummy_data.csv"
df.to_csv(filepath, index=False)

print(f"Created dummy data at: {filepath}\n")

# 2. Initialize the CSVDataset
csv_dataset = CSVDataset(filepath=filepath, save_args={"index": False})

# 3. Load data
loaded_df = csv_dataset.load()
print("Loaded DataFrame from CSVDataset:\n")
print(loaded_df)

# 4. Save new data using the dataset
new_data = pd.DataFrame({"col1": [4, 5], "col2": ["D", "E"]})
csv_dataset.save(new_data)
print("\nSaved new data to the CSV file.\n")

# 5. Verify by loading again
reloaded_df = csv_dataset.load()
print("Reloaded DataFrame after saving new data:\n")
print(reloaded_df)

# 6. Clean up the dummy file
os.remove(filepath)
print(f"\nCleaned up dummy data file: {filepath}")