CloudML HyperTune

Common errors

• Hyperparameter tuning failed

  cause The hyperparameter tuning service could not retrieve the objective metric from the training job. This often happens due to a mismatch between the `hyperparameter_metric_tag` in your code and the metric configuration in Vertex AI.
  fix Verify that the `hyperparameter_metric_tag` used in `hpt.report_hyperparameter_tuning_metric()` exactly matches the `metric_id` specified in your Vertex AI hyperparameter tuning job configuration. Also ensure the `report_hyperparameter_tuning_metric` call is actually reached and executed during training.

• Trials show status 'Failed' but logs don't show Python errors

  cause The training code might be completing without explicitly reporting a metric, or the metric is being reported incorrectly. Older TensorFlow Estimator-based training might not be correctly outputting to event files in a format recognized by the tuning service if `cloudml-hypertune` isn't used.
  fix Ensure `hpt.report_hyperparameter_tuning_metric` is called with a valid `metric_value` and `hyperparameter_metric_tag` at the end of each evaluation step or epoch. For non-TensorFlow models or custom evaluation loops, `cloudml-hypertune` is the definitive way to report.

• Google Cloud ML Engine does not return objective values when hyperparameter tuning

  cause The `cloudml-hypertune` library's `report_hyperparameter_tuning_metric` function was not invoked or did not successfully transmit the metric. This could be due to an uncaught exception in the training code preventing the call, or an incorrect metric tag.
  fix Debug your training script to ensure `hpt.report_hyperparameter_tuning_metric` is called. Add logging around this call to confirm its execution and the values being passed. Double-check the `hyperparameter_metric_tag` against your Vertex AI job configuration.

Warnings

• gotcha The `cloudml-hypertune` library is specifically designed to work within Google Cloud's hyperparameter tuning services (Vertex AI/AI Platform). It is not a standalone hyperparameter tuning framework and will not function as such outside of this cloud environment for driving optimization.
  Fix: Ensure your training job is submitted to Vertex AI (or deprecated Cloud ML Engine) with a hyperparameter tuning configuration.
• breaking Hyperparameters you wish to tune *must* be exposed as command-line arguments in your training script. The Vertex AI tuning service passes trial-specific hyperparameter values via these arguments, not directly to the `hypertune` library.
  Fix: Modify your training script to parse hyperparameters as `argparse` arguments, ensuring they are named consistently with your tuning job configuration.
• gotcha The `hyperparameter_metric_tag` passed to `hpt.report_hyperparameter_tuning_metric` must exactly match the `metric_id` (or `hyperparameterMetricTag` in older configs) specified in your Vertex AI hyperparameter tuning job configuration. A mismatch will result in trials failing to report metrics or the tuning job not recognizing the objective.
  Fix: Carefully verify that the metric tag string in your code is identical to the one in your Vertex AI hyperparameter tuning job configuration.
• gotcha Version `0.1.0.dev6` was released in December 2019. While still functional and referenced in Google Cloud documentation for Vertex AI, its lack of recent updates may lead to assumptions of abandonment or compatibility issues with very new Python features, though it's a very stable, minimal utility.
  Fix: No direct fix needed, but be aware of its static nature. If encountering unexpected behavior with bleeding-edge Python or ML frameworks, consider if the issue is external to `cloudml-hypertune`.
• gotcha Training runs that result in `NaN` values in loss functions or other unhandled exceptions will cause hyperparameter tuning trials to fail. This not only wastes resources but also prevents the tuning algorithm from learning from that trial.
  Fix: Implement robust error handling and numeric stability checks (e.g., clipping gradients, checking for `NaN`s) in your training code. Ensure `report_hyperparameter_tuning_metric` is called even if training terminates early due to an error, or within a `finally` block if a fallback metric can be determined.

Install

• pip install cloudml-hypertune Install latest version

Imports

• HyperTune
  wrong:

  from cloudml_hypertune import HyperTune

  correct:

  import hypertune
  hpt = hypertune.HyperTune()

  The primary class 'HyperTune' is typically accessed via the top-level 'hypertune' module, not directly imported from 'cloudml_hypertune'.

Quickstart

This quickstart demonstrates how to use `cloudml-hypertune` within a training script to report a metric. The script accepts hyperparameters as command-line arguments, which is a critical requirement for Google Cloud's hyperparameter tuning service to inject trial-specific values. The `HyperTune` instance then reports the objective metric and the current training step. You would run this script within a container on Vertex AI/Cloud ML Engine.

import hypertune
import argparse
import os

def train_model(learning_rate, num_epochs, metric_tag):
    # Simulate model training with hyperparameters
    # In a real scenario, this would be your ML training loop
    print(f"Training with learning_rate={learning_rate}, num_epochs={num_epochs}")
    
    # Simulate a metric, e.g., validation accuracy
    # In a real scenario, you'd get this from your model's evaluation
    metric_value = 0.5 + (learning_rate * 0.1) + (num_epochs * 0.01)
    
    # Report the metric to Cloud AI Platform / Vertex AI
    hpt = hypertune.HyperTune()
    hpt.report_hyperparameter_tuning_metric(
        hyperparameter_metric_tag=metric_tag, # Must match config.yaml objective metricTag
        metric_value=metric_value,
        global_step=num_epochs # Or current training step
    )
    print(f"Reported metric '{metric_tag}': {metric_value} at step {num_epochs}")

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    # Define hyperparameters as command-line arguments
    parser.add_argument(
        '--learning_rate',
        type=float,
        default=0.01,
        help='Learning rate for training.'
    )
    parser.add_argument(
        '--num_epochs',
        type=int,
        default=10,
        help='Number of epochs for training.'
    )
    parser.add_argument(
        '--metric_tag',
        type=str,
        default='accuracy',
        help='Tag for the metric reported to HyperTune.'
    )
    args = parser.parse_args()
    
    train_model(args.learning_rate, args.num_epochs, args.metric_tag)