TorchEval

Common errors

• TypeError: 'BinaryAccuracy' object is not callable

  cause Attempting to call the metric instance directly instead of using its `update()` or `compute()` methods.
  fix Use `metric.update(predictions, targets)` to feed data and `metric.compute()` to get the result. The metric instance itself is not a function.

• RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

  cause Input tensors (predictions and targets) passed to `metric.update()` are on different devices (e.g., one on CPU, one on GPU).
  fix Ensure both `predictions` and `targets` tensors are on the same device as the metric, or on the same device as each other if the metric is device-agnostic (though it often infers from the first input). Use `.to(device)` to move tensors: `predictions.to('cuda'), targets.to('cuda')`.

• ValueError: The 'input' tensor must be 1D for BinaryAccuracy

  cause The input prediction tensor provided to `BinaryAccuracy.update()` has an incorrect shape (e.g., `(N, C)` for a multiclass output, or `(N, 1, H, W)`).
  fix Reshape the `input` tensor to be 1-dimensional, typically `(N,)` or `(N,1)` for binary classification. Use methods like `.squeeze()`, `.view()`, or appropriate indexing to get the correct shape.

• Incorrect or unexpected metric values in distributed training.

  cause Forgetting to synchronize metric states across distributed processes when computing the final result.
  fix In a distributed environment, always use `metric.sync_and_compute()` instead of `metric.compute()` to ensure all local states are gathered and aggregated correctly before calculation.

Warnings

• gotcha In distributed training environments (e.g., using `torch.distributed`), metrics accumulate local states independently on each process. To get the correct global metric value, you must call `metric.sync_and_compute()` instead of `metric.compute()`.
  Fix: Use `metric.sync_and_compute()` when operating in a distributed setting. Ensure `torch.distributed` is initialized before calling this method.
• gotcha Metrics accumulate their internal state across multiple calls to `update()`. If you need to calculate metrics for distinct evaluation periods (e.g., per epoch or per validation run), you must call `metric.reset()` before processing new data, or create a new metric instance.
  Fix: Call `metric.reset()` at the beginning of each new evaluation period to clear previously accumulated data.
• gotcha TorchEval is currently in a pre-1.0 state (0.0.x versions). While efforts are made to maintain stability, minor API changes might occur between releases. Always refer to the latest documentation for precise API details.
  Fix: Periodically check the official documentation and release notes when upgrading for potential API adjustments. Pin minor versions if strict API stability is required.
• gotcha Input tensors for `update()` must adhere to specific shapes and dtypes expected by each metric. For instance, binary metrics typically expect 1D tensors (N,) or (N,1) for predictions and targets.
  Fix: Carefully read the documentation for each specific metric regarding expected input shapes, dtypes, and value ranges (e.g., logits vs. probabilities for classification). Reshape or cast tensors as needed, e.g., `predictions.squeeze(-1)`.

Install

• pip install torcheval Install stable version

Imports

• Metric

  from torcheval.metrics import Metric

• BinaryAccuracy
  wrong:

  from torcheval.metrics.classification import BinaryAccuracy

  correct:

  from torcheval.metrics import BinaryAccuracy

  Metrics are typically exposed directly under `torcheval.metrics`.
• MetricCollection

  from torcheval.metrics import MetricCollection

Quickstart

This example demonstrates how to initialize a `BinaryAccuracy` metric, update it with predictions and targets, compute the current accuracy, and reset its internal state. Metrics accumulate data, so remember to call `reset()` for new evaluation runs (e.g., per epoch).

import torch
from torcheval.metrics import BinaryAccuracy

# Initialize the metric
metric = BinaryAccuracy()

# Simulate model predictions and ground truth labels
# Ensure inputs are tensors and on the correct device
predictions = torch.tensor([0.9, 0.1, 0.8, 0.2, 0.95])
targets = torch.tensor([1, 0, 1, 0, 1])

# Update the metric with a batch of data
metric.update(predictions, targets)

# Get the computed result
accuracy = metric.compute()
print(f"Binary Accuracy: {accuracy.item():.4f}")

# Example with another batch
predictions2 = torch.tensor([0.4, 0.6, 0.7])
targets2 = torch.tensor([0, 1, 0])
metric.update(predictions2, targets2)

# Compute cumulative accuracy
cumulative_accuracy = metric.compute()
print(f"Cumulative Binary Accuracy: {cumulative_accuracy.item():.4f}")

# Reset the metric's internal state
metric.reset()
print(f"Accuracy after reset and recompute: {metric.compute().item():.4f}")