Hyperopt

0.2.7 · deprecated · verified Thu Apr 09

Hyperopt is a Python library for distributed asynchronous hyperparameter optimization, enabling optimization over awkward search spaces including real-valued, discrete, and conditional dimensions. It uses Bayesian optimization algorithms like Tree of Parzen Estimators (TPE) and Random Search to efficiently find optimal hyperparameters for machine learning models. The current PyPI version is 0.2.7.

Warnings

breaking The open-source version of Hyperopt is no longer being actively maintained. Databricks, a notable user, explicitly states this and recommends alternatives like Optuna for single-node optimization or Ray Tune for distributed tuning.
Fix: For new projects, consider using actively maintained hyperparameter optimization libraries such as Optuna or Ray Tune. If bound to Hyperopt, be aware that community support and bug fixes may be limited.
gotcha When using `hp.choice()` for categorical parameters, Hyperopt stores and returns the *index* of the chosen option from the list, not the actual value.
Fix: To retrieve the actual parameter value, use `hyperopt.space_eval(space, best_parameters)` where `space` is your defined search space and `best_parameters` are the results from `fmin`.
gotcha A reported loss of NaN (not a number) often indicates that the objective function passed to `fmin()` returned `NaN`. This can lead to unexpected optimization behavior.
Fix: Review your objective function to ensure it handles all possible input combinations from the search space gracefully and avoids returning NaN values. You might need to adjust the hyperparameter space to prevent problematic inputs.
gotcha When using `SparkTrials` for distributed hyperparameter tuning, Hyperopt determines parallelism at the start. It will *not* dynamically adapt to changes in cluster size if the cluster autoscales.
Fix: Avoid using `SparkTrials` on autoscaling clusters. For GPU clusters, be mindful that `SparkTrials` uses one executor thread per node, reducing maximum parallelism. `SparkTrials` is also primarily designed for single-machine ML models, not inherently distributed ones like MLlib or Horovod.
gotcha Due to its use of stochastic search algorithms, Hyperopt's reported loss does not necessarily decrease monotonically with each evaluation. This is expected behavior and does not indicate an issue.
Fix: Understand that the optimization process is not a smooth descent. Focus on the overall trend towards a lower loss and the final 'best' result, rather than expecting a strictly decreasing loss at every step.

Install

pip install hyperopt PyPI

Imports

fmin
```
from hyperopt import fmin
```
The main function to minimize an objective over a search space.
tpe
```
from hyperopt import tpe
```
Module containing the Tree-structured Parzen Estimator (TPE) algorithm, a common choice for optimization.
hp
```
from hyperopt import hp
```
Module for defining hyperparameter search spaces (e.g., hp.uniform, hp.choice).
Trials
```
from hyperopt import Trials
```
Used to store the results of the optimization (hyperparameters, loss, etc.).

Quickstart

This quickstart demonstrates how to use Hyperopt to find the minimum of a simple mathematical function. It involves defining an objective function, specifying a search space using `hp` functions, creating a `Trials` object to log results, and finally calling `fmin` with a chosen algorithm (TPE in this case) and the maximum number of evaluations.

import numpy as np
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

# 1. Define the objective function to minimize
def objective(args):
    x, y = args
    return {'loss': x ** 2 + y ** 2, 'status': STATUS_OK}

# 2. Define the search space
space = [
    hp.uniform('x', -10, 10),
    hp.uniform('y', -10, 10)
]

# 3. Create a Trials object to store results
trials = Trials()

# 4. Run the optimization
best = fmin(objective, space, algo=tpe.suggest, max_evals=100, trials=trials)

print("Best parameters found:", best)
print("Best loss found:", trials.best_trial['result']['loss'])

view raw JSON →