Metaflow Stubs

Common errors

• Error: MetaflowStepDependencyError: Step 'step_name' failed due to missing or incompatible dependencies.

  cause A required Python package for a step was not found or had a version conflict in the execution environment, particularly when running remotely without explicitly declared dependencies.
  fix Add `@pypi` or `@conda` decorators with the necessary packages and their versions to the `FlowSpec` or the specific `@step` method. Ensure consistency across environments.

• AttributeError: 'NoneType' object has no attribute 'some_attribute' or similar when accessing self.variable from a previous step.

  cause The artifact `self.variable` from a prior step was not properly set or was not serialized/deserialized correctly, or an attempt was made to access it from an unsupported context (e.g., before the preceding step completed).
  fix Ensure the variable is assigned to `self.variable_name` in the producing step and that `self.next()` correctly transitions to the consuming step. When inspecting past runs, use the Metaflow Client API, `from metaflow import current`, to reliably access artifacts.

• TypeError: Object of type MyObject is not JSON serializable

  cause An attempt was made to store a non-serializable Python object directly as a Metaflow artifact (instance variable `self.xyz`), which Metaflow attempts to persist between steps.
  fix Convert non-serializable objects into a serializable format (e.g., JSON, string, bytes, or a well-known data structure like a pandas DataFrame) before assigning them to `self.` attributes.

Warnings

• breaking While Metaflow prioritizes backward compatibility, minor breaking changes can occur in patch versions, especially those addressing bug fixes or internal architectural improvements. These can impact how stubs align with the runtime library.
  Fix: Always review the GitHub release notes of the main `metaflow` library (Netflix/metaflow) before upgrading to understand specific breaking changes and migration steps that might affect stub usage.
• gotcha When scaling Metaflow flows to remote compute environments (e.g., AWS Batch, Kubernetes), locally `pip install`'d or `conda install`'d third-party dependencies are not automatically available. This can lead to runtime errors even if your local environment with stubs is correct.
  Fix: Explicitly declare all external Python dependencies using the `@pypi` or `@conda` decorators on your `FlowSpec` class or individual `@step` methods to ensure reproducibility and correct execution in remote environments. For example: `@pypi(packages={'scikit-learn': '1.0.2'})`.
• gotcha Metaflow does not offer native support for Windows operating systems. Users on Windows must utilize the Windows Subsystem for Linux (WSL) to install and run Metaflow and its stubs, as it relies on a *nix-like environment.
  Fix: Install and configure Windows Subsystem for Linux (WSL) and then install Metaflow (and its stubs) within the WSL environment.
• gotcha Using mutable default arguments in `@step` method signatures (e.g., `items=[]`) can lead to unexpected shared state across different task executions, violating Metaflow's reproducibility principles.
  Fix: Avoid mutable default arguments. Initialize mutable objects (like lists or dictionaries) inside the step method, for example, `items = [] if items is None else items`.
• gotcha Failing to call `self.next()` at the end of a step will terminate the flow execution at that step, preventing subsequent steps from running, even if the logic within the current step completes successfully.
  Fix: Always ensure that `self.next(next_step_method)` is called at the end of every step (except for the final `end` step) to define the flow's execution graph.

Install

• pip install ob-metaflow-stubs Install latest version

Imports

• FlowSpec

  from metaflow import FlowSpec

• step

  from metaflow import step

• Parameter

  from metaflow import Parameter

• current

  from metaflow import current

• retry
  wrong:

  from metaflow.decorators import retry

  correct:

  from metaflow import retry

  While `@retry` is a decorator, it's typically imported directly from the top-level `metaflow` package for consistency with other core components.

Quickstart

This 'Hello World' example defines a basic Metaflow flow with three steps: `start`, `hello`, and `end`. It demonstrates defining steps with `@step` and connecting them with `self.next()`. Run this script using `python your_flow_name.py run` in your terminal to execute the flow locally.

from metaflow import FlowSpec, step

class HelloFlow(FlowSpec):
    """A simple Metaflow flow to validate installation."""

    @step
    def start(self):
        print("HelloFlow is starting.")
        self.message = "Metaflow says: Hi!"
        self.next(self.hello)

    @step
    def hello(self):
        print(self.message)
        self.next(self.end)

    @step
    def end(self):
        print("HelloFlow is all done.")

if __name__ == "__main__":
    HelloFlow()