Kedro

Warnings

• breaking Kedro dropped support for Python 3.9 in version 1.1.0. Projects using Kedro 1.1.0 or newer must use Python 3.10 or later.
  Fix: Upgrade your Python environment to 3.10 or a newer supported version.
• breaking The `KedroDataCatalog` class was renamed to `DataCatalog` and became the default catalog implementation in Kedro 1.0.0. While most standard workflows were unaffected, programmatic interactions with the catalog, especially direct instantiation or accessing missing datasets (`__getitem__`), might require updates.
  Fix: Use `DataCatalog` instead of `KedroDataCatalog`. Review any custom catalog interactions for compatibility, especially error handling for missing datasets which now raise `DatasetNotFoundError`.
• gotcha Kedro relies heavily on its project structure (created by `kedro new`) and configuration files in the `conf/` directory. Deviations or manually created projects without the correct structure can lead to `KedroContextError` or `ConfigLoaderError`.
  Fix: Always initialize new projects with `kedro new` and adhere to the generated project structure. When modifying configuration, follow the `conf/base` and `conf/local` conventions.
• gotcha Confusion between `params:` (dynamic parameters passed as node inputs) and `parameters:` (static configuration loaded from `conf/catalog.yml`) is a common pitfall. The new parameter validation (Kedro >=1.3.0) specifically targets `params:` inputs.
  Fix: Use `params:` for values that are passed as inputs to node functions, often dynamic. Use `parameters:` for static, global configurations loaded via the `DataCatalog` and accessed from the `context.params` object or directly as dataset entries.
• gotcha Prior to `kedro==1.1.1`, the `project_version` specified in `src/<project_name>/settings.py` had to *exactly match* the installed Kedro package version (including minor and patch versions) to avoid a `ProjectVersionError`.
  Fix: As of `kedro>=1.1.1`, only major version mismatches are strictly enforced. For older versions, ensure your project's `project_version` matches your installed Kedro package version. For new projects, ensure you're on a recent Kedro version to benefit from the more flexible version check.
• deprecated The `--namespace` CLI flag for `kedro run` was deprecated in version 0.19.15 and is discouraged. Kedro now promotes using proper modular pipelines and explicit dataset prefixing for organization.
  Fix: Refactor pipelines to use modular pipeline features (`Pipeline(namespace=...)`) and explicit dataset naming conventions instead of relying on the deprecated `--namespace` flag.
• gotcha Public APIs marked with the `@experimental` decorator (introduced in 1.2.0) are unstable and may change without backward compatibility guarantees. Use them with caution.
  Fix: Be aware that `@experimental` APIs are subject to change. Avoid using them in production code unless you are prepared to adapt to potential breaking changes in future releases.

Install

• pip install kedro Basic Install
• pip install "kedro[pandas,spark]" Install with Data Connectors (e.g., Pandas, Spark)

Imports

• Pipeline

  from kedro.pipeline import Pipeline

• node

  from kedro.pipeline import node

• DataCatalog

  from kedro.io import DataCatalog

  `KedroDataCatalog` was renamed to `DataCatalog` and became the default in Kedro 1.0.0.
• MemoryDataSet

  from kedro.io import MemoryDataSet

• SequentialRunner

  from kedro.runner import SequentialRunner

• KedroSession

  from kedro.framework.session import KedroSession

Quickstart

This example demonstrates how to define Kedro nodes and combine them into a pipeline. It then uses a `SequentialRunner` and an in-memory `DataCatalog` to execute the pipeline. In a typical Kedro project, `kedro new` creates a project structure, and `kedro run` orchestrates execution via `KedroSession`, loading configurations from `conf/` files.

from kedro.io import DataCatalog, MemoryDataSet
from kedro.pipeline import Pipeline, node
from kedro.runner import SequentialRunner

# 1. Define node functions (plain Python functions)
def greet(name: str) -> str:
    """A node that greets a given name."""
    return f"Hello, {name}!"

def capitalize(text: str) -> str:
    """A node that capitalizes a string."""
    return text.upper()

# 2. Assemble nodes into a pipeline
def create_example_pipeline() -> Pipeline:
    return Pipeline([
        node(
            func=greet,
            inputs="input_name", # Input dataset key
            outputs="greeting_message", # Output dataset key
            name="greet_user_node"
        ),
        node(
            func=capitalize,
            inputs="greeting_message",
            outputs="final_output", # Final output dataset key
            name="capitalize_message_node"
        )
    ])

# 3. Create a DataCatalog with input data
# In a real Kedro project, this is usually defined in conf/base/catalog.yml
catalog = DataCatalog({
    "input_name": MemoryDataSet(data="World"),
    "final_output": MemoryDataSet() # Define an output dataset to store results
})

# 4. Instantiate the pipeline and a runner
my_pipeline = create_example_pipeline()
runner = SequentialRunner()

# 5. Run the pipeline
# In a real Kedro project, `kedro run` via `KedroSession` orchestrates this.
print("Running Kedro pipeline...")
result_catalog = runner.run(my_pipeline, catalog)

# 6. Retrieve results
final_message = result_catalog.load("final_output")
print(f"Pipeline finished. Final message: {final_message}")
# Expected output: Pipeline finished. Final message: HELLO, WORLD!