Optuna

Warnings

• breaking Optuna v4.x removed features deprecated in v2.x. This includes the `optuna.multi_objective` submodule and `MOTPESampler`. If migrating from older versions, consult the v4 migration guide for a complete list of removed APIs.
  Fix: Review the Optuna v4 Migration Guide. For multi-objective optimization, the functionality was integrated into the single-objective API in v2.4.0, so adapt your code to use the unified API.
• breaking The paths for `IntersectionSearchSpace` and `intersection_search_space` moved from `optuna.samplers` to `optuna.search_space`. Additionally, `intersection_search_space` now takes `trials` instead of `study` and the `ordered_dict` argument was removed as dictionaries are now ordered by default.
  Fix: Update import paths and argument signatures for `IntersectionSearchSpace` and `intersection_search_space` to reflect their new locations and API changes.
• deprecated Python 3.8 support was dropped in Optuna v4.x.
  Fix: Upgrade your Python environment to Python 3.9 or newer.
• gotcha For some samplers, such as `TPESampler`, certain arguments have been made keyword-only, and the behavior of `consider_prior` argument might have changed or been simplified.
  Fix: Ensure all arguments for samplers are passed as keyword arguments and review sampler-specific documentation for changes in argument behavior or deprecations.
• gotcha Sophisticated schedulers (e.g., `AsyncHyperBandScheduler`) may not work correctly with multi-objective optimization, as they typically expect a scalar score to compare fitness among trials.
  Fix: When performing multi-objective optimization, be mindful of scheduler compatibility. Consider using samplers and pruners specifically designed or enhanced for multi-objective tasks (e.g., `GPSampler` with multi-objective support from v4.4).

Install

• pip install optuna Install Optuna

Imports

• optuna

  import optuna

• Trial

  from optuna import Trial

  While `from optuna.trial import Trial` works, `from optuna import Trial` is the more common and recommended import pattern for the core `Trial` object as of recent versions.

Quickstart

This quickstart defines an objective function that trains either an SVR or RandomForestRegressor, with hyperparameters sampled by Optuna's `Trial` object. It then creates a study to minimize the mean squared error over 100 trials, showcasing how Optuna dynamically builds search spaces and finds optimal hyperparameters.

import optuna
import sklearn
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_squared_error

def objective(trial: optuna.Trial) -> float:
    # Invoke suggest methods of a Trial object to generate hyperparameters.
    regressor_name = trial.suggest_categorical('regressor', ['SVR', 'RandomForest'])

    if regressor_name == 'SVR':
        svr_c = trial.suggest_float('svr_c', 1e-10, 1e10, log=True)
        regressor_obj = sklearn.svm.SVR(C=svr_c)
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
        regressor_obj = RandomForestRegressor(max_depth=rf_max_depth, random_state=0)

    X, y = fetch_california_housing(return_X_y=True)
    X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=0)

    regressor_obj.fit(X_train, y_train)
    y_pred = regressor_obj.predict(X_val)
    error = mean_squared_error(y_val, y_pred)
    return error

if __name__ == '__main__':
    study = optuna.create_study(direction='minimize')  # Create a new study
    study.optimize(objective, n_trials=100)  # Invoke optimization

    print(f"Best trial value: {study.best_value:.4f}")
    print(f"Best params: {study.best_params}")