pyhdfe

Warnings

• gotcha For extremely large datasets or a very high number of levels in fixed effects, memory consumption can still be significant, despite optimizations for sparse data. Monitor memory usage carefully.
  Fix: Consider downsampling, using more memory-efficient data types, or processing data in chunks if memory becomes a bottleneck. The library is optimized for sparse matrix operations, ensuring your input data structures (e.g., pandas Series/DataFrame) are appropriate can help.
• gotcha The Gauss-Seidel algorithm used for absorption can sometimes converge slowly or fail to converge for certain data structures or with highly collinear fixed effects. This is a common challenge for iterative solvers.
  Fix: If convergence issues arise, consider increasing `max_iter` or `tol` parameters if available (not directly exposed in `hdfe_cluster_col` but for other related functions). Review the structure of your fixed effects for potential issues, or simplify the model if necessary.
• gotcha The `fixed_effects` argument expects a list of 1-D arrays or pandas Series, where each element represents a single fixed effect column. Incorrect formatting (e.g., passing a 2-D array directly) will lead to errors.
  Fix: Ensure that `fixed_effects` is structured as `[pd.Series(fe1), pd.Series(fe2), ...]`. Each Series/array should contain the categorical identifiers for that specific fixed effect.

Install

• pip install pyhdfe Install stable release

Imports

• hdfe

  from pyhdfe import hdfe

  The core functionality is exposed through the 'hdfe' submodule.

Quickstart

This quickstart demonstrates how to use `hdfe.hdfe_cluster_col` to absorb multiple high-dimensional fixed effects from a feature matrix `X` and a target vector `y`. It generates synthetic data with several fixed effect columns and then applies the absorption, returning the transformed (residualized) `X` and `y`.

import numpy as np
import pandas as pd
from pyhdfe import hdfe

# Create some dummy data
np.random.seed(42)
n_obs = 1000
n_fixed_effects = 3

X = pd.DataFrame(np.random.rand(n_obs, 5), columns=[f'x{i}' for i in range(5)])
y = pd.Series(np.random.rand(n_obs))

fixed_effects_data = []
for i in range(n_fixed_effects):
    n_levels = np.random.randint(50, 200) # Varying number of levels
    fixed_effects_data.append(pd.Series(np.random.randint(0, n_levels, n_obs)))

# Absorb fixed effects from X and y
# fixed_effects is a list of 1D arrays/Series representing each fixed effect column
# absorb_cols specifies which columns from X to transform
# drop_cols specifies columns to drop before transformation (often the intercept)

X_transformed, y_transformed = hdfe.hdfe_cluster_col(
    X,
    y,
    fixed_effects=fixed_effects_data,
    absorb_cols=X.columns.tolist(), # Absorb all X columns
    drop_cols=[], # No columns to drop in this example
    get_residuals=True
)

print(f"Original X shape: {X.shape}")
print(f"Transformed X shape: {X_transformed.shape}")
print(f"Original y shape: {y.shape}")
print(f"Transformed y shape: {y_transformed.shape}")