Fairlearn

Common errors

• ModuleNotFoundError: No module named 'fairlearn'

  cause The Fairlearn library is not installed in your current Python environment.
  fix Run `pip install fairlearn` to install the library.

• TypeError: MetricFrame.__init__() got an unexpected keyword argument 'metric'

  cause You are attempting to use the old `metric` argument for `MetricFrame` after Fairlearn v0.10.0, which expects `metrics` and keyword-only arguments.
  fix Change `metric=` to `metrics=` in your `MetricFrame` constructor call, and ensure all arguments are passed as keyword arguments (e.g., `y_true=y_test`, `sensitive_features=sf_test`).

• fairlearn requires scikit-learn>=X.Y,<A.B but you have scikit-learn C.D

  cause Your installed `scikit-learn` version is outside the range required by your `fairlearn` version, causing dependency conflicts.
  fix Uninstall your current `scikit-learn` (`pip uninstall scikit-learn`) and then install a compatible version. For example, if Fairlearn requires `>=1.0,<1.7`, run `pip install 'scikit-learn>=1.0,<1.7'`.

• ValueError: sensitive_features cannot be None

  cause You did not provide the `sensitive_features` argument to a Fairlearn assessment or mitigation function (e.g., `MetricFrame`, `GridSearch`), where it is a required input.
  fix Ensure `sensitive_features` is passed as a `pandas.Series`, `numpy.ndarray`, or list-like object to the relevant Fairlearn component.

Warnings

• breaking The `MetricFrame` constructor API changed significantly in v0.7.0, with the `metric` argument being renamed to `metrics` and all arguments becoming keyword-only. The old syntax issued a deprecation warning until v0.10.0, after which it became a breaking change.
  Fix: Update `MetricFrame` calls to use `metrics=` for a single metric or a dictionary of metrics, and pass all arguments (e.g., `y_true=`, `y_pred=`, `sensitive_features=`) as keyword arguments.
• breaking Fairlearn v0.8.0 dropped support for Python 3.6 and 3.7. Attempting to install or run on these Python versions will lead to dependency resolution errors or runtime issues.
  Fix: Upgrade your Python environment to 3.9 or newer. The current recommended minimum is Python 3.9 as of Fairlearn v0.13.0.
• gotcha Fairlearn has tight dependencies on `scikit-learn`. Mismatched scikit-learn versions can lead to `AttributeError`, `TypeError`, or unexpected behavior during model training or fairness assessment. For example, v0.12.0 added specific compatibility for scikit-learn 1.6.
  Fix: Always consult Fairlearn's `pyproject.toml` or `setup.py` on its GitHub repository for the exact `scikit-learn` version constraints. Install `scikit-learn` within the specified range (e.g., `pip install 'scikit-learn>=1.0,<1.7'`).
• gotcha Prior to v0.8.0, passing a custom `grid` object to a `GridSearch` reduction could result in a `KeyError` due to an internal bug.
  Fix: Upgrade Fairlearn to v0.8.0 or later. If upgrading is not possible, avoid passing custom `grid` objects to `GridSearch` in older versions.

Install

• pip install fairlearn Install Fairlearn

Imports

• MetricFrame
  wrong:

  from fairlearn.postprocessing import MetricFrame

  correct:

  from fairlearn.metrics import MetricFrame

  MetricFrame is used for fairness assessment and is located in fairlearn.metrics.
• GridSearch
  wrong:

  from fairlearn.mitigation import GridSearch

  correct:

  from fairlearn.reductions import GridSearch

  GridSearch is one of the reduction-based mitigation algorithms, found in fairlearn.reductions.
• ThresholdOptimizer

  from fairlearn.postprocessing import ThresholdOptimizer

  ThresholdOptimizer is a post-processing mitigation technique.
• CorrelationRemover

  from fairlearn.preprocessing import CorrelationRemover

  CorrelationRemover is a preprocessing mitigation technique.
• fetch_adult

  from fairlearn.datasets import fetch_adult

  Fairlearn provides access to common fairness datasets for examples and testing.

Quickstart

This quickstart demonstrates how to use Fairlearn's `MetricFrame` to assess fairness. It involves creating a simple dataset, training a scikit-learn model, and then evaluating performance metrics across different sensitive feature groups using `MetricFrame`.

import pandas as pd
from fairlearn.metrics import MetricFrame, accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# 1. Create dummy data
data = {
    'feature_1': [10, 12, 11, 15, 13, 9, 14, 16, 11, 13],
    'feature_2': [1, 0, 1, 0, 1, 0, 1, 0, 1, 0],
    'sensitive_feature': ['A', 'B', 'A', 'B', 'A', 'B', 'A', 'B', 'A', 'B'],
    'target': [1, 0, 1, 1, 0, 0, 1, 0, 1, 0]
}
df = pd.DataFrame(data)

X = df[['feature_1', 'feature_2']]
y = df['target']
sensitive_features = df['sensitive_feature']

X_train, X_test, y_train, y_test, sf_train, sf_test = train_test_split(
    X, y, sensitive_features, test_size=0.5, random_state=42
)

# 2. Preprocessing pipeline
preprocessor = ColumnTransformer(
    transformers=[
        ('num', StandardScaler(), ['feature_1']),
        ('cat', OneHotEncoder(handle_unknown='ignore'), ['feature_2'])
    ],
    remainder='passthrough'
)

# 3. Train a model
model = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', LogisticRegression(solver='liblinear', random_state=42))
])
model.fit(X_train, y_train)

# 4. Predict and assess fairness using MetricFrame
y_pred = model.predict(X_test)

metric_frame = MetricFrame(
    metrics=accuracy_score,
    y_true=y_test,
    y_pred=y_pred,
    sensitive_features=sf_test
)

print(f"Overall accuracy: {metric_frame.overall}")
print(f"Accuracy by sensitive feature group:\n{metric_frame.by_group}")
print("Fairlearn quickstart executed successfully.")