Metaflow

2.19.22 · active · verified Sat Apr 11

Metaflow is a human-friendly Python library that helps scientists and engineers build and manage real-life data science projects. Originally developed at Netflix, it provides a unified API to the infrastructure stack required for data science projects, from prototype to production. It is actively maintained with frequent patch releases.

Warnings

breaking While Metaflow generally prioritizes backward compatibility, minor breaking changes can occur, especially in patch versions addressing bug fixes or internal architectural improvements. Always review the GitHub release notes before upgrading.
Fix: Consult the `Release Notes` section on the Metaflow documentation or GitHub releases page for specific breaking changes and migration steps before upgrading.
gotcha When scaling Metaflow flows to remote compute environments (e.g., AWS Batch, Kubernetes), locally `pip install`'d or `conda install`'d third-party dependencies are not automatically available. You must explicitly declare these dependencies using the `@pypi` or `@conda` decorators on your flow or individual steps to ensure reproducibility and correct execution in remote environments.
Fix: Use `@pypi(packages={'package_name': 'version'})` or `@conda(packages={'package_name': 'version'})` decorators on your `FlowSpec` class or individual `@step` methods for all external Python dependencies.
gotcha Metaflow's most mature and battle-tested integrations are with Amazon Web Services (AWS), including S3 for storage, Batch for compute, and Step Functions for orchestration. While it supports other cloud providers like Azure and GCP, the level of integration and available features may vary, potentially requiring more manual configuration.
Fix: Review Metaflow's documentation for specific cloud provider integrations to understand the scope of support and any necessary configurations for non-AWS environments.
gotcha Metaflow does not offer native support for Windows operating systems. Users on Windows must utilize the Windows Subsystem for Linux (WSL) to install and run Metaflow, as it relies on a *nix-like environment.
Fix: Install and configure Windows Subsystem for Linux (WSL) and then install Metaflow within the WSL environment.
gotcha Data artifacts (instance variables prefixed with `self.`) are automatically persisted and passed between steps. Directly relying on global variables or modifying external state outside of Metaflow's artifact management can lead to non-reproducible runs, especially in distributed or resumed executions, as these changes might not be tracked or correctly restored.
Fix: Always pass data between steps by assigning it to instance variables (e.g., `self.data_artifact = value`). Avoid relying on global state or external files for inter-step communication, as Metaflow handles serialization and deserialization of `self.` attributes automatically.

Install

pip install metaflow Install Metaflow

Imports

FlowSpec
```
from metaflow import FlowSpec
```
The base class for defining a Metaflow workflow.
step
```
from metaflow import step
```
A decorator to mark a method as a step in the workflow DAG.
Parameter
```
from metaflow import Parameter
```
Used to define command-line parameters for a flow.
card
```
from metaflow import card
```
Decorator for creating rich UI cards for steps.
pypi
```
from metaflow import pypi
```
Decorator for managing step-specific PyPI dependencies for local and remote execution.
conda
```
from metaflow import conda
```
Decorator for managing step-specific Conda dependencies for local and remote execution.

Quickstart

This quickstart defines a basic Metaflow workflow. It consists of three sequential steps: `start`, `hello`, and `end`. The `start` step initializes a message, the `hello` step prints it, and the `end` step marks the completion of the flow. To run this, save it as a Python file (e.g., `hello_flow.py`) and execute `python hello_flow.py run` in your terminal.

import os
from metaflow import FlowSpec, step

class HelloFlow(FlowSpec):
    """A simple Metaflow that prints 'Hi'."""

    @step
    def start(self):
        """This is the 'start' step. All flows must have a step named 'start'."""
        print("HelloFlow is starting.")
        self.message = "Metaflow says: Hi!"
        self.next(self.hello)

    @step
    def hello(self):
        """A step for Metaflow to introduce itself."""
        print(self.message)
        self.next(self.end)

    @step
    def end(self):
        """This is the 'end' step. All flows must have an 'end' step."""
        print("HelloFlow is all done.")

if __name__ == "__main__":
    HelloFlow()

view raw JSON →