Scoring Rules

0.9.0 · active · verified Fri Apr 17

Scoringrules is a Python library for evaluating probabilistic forecasts using various scoring rules. It provides implementations of Continuous Ranked Probability Score (CRPS), Log Score, Brier Score, Dawid-Sebastiani Score, and more for various distributions and ensembles. The current version is 0.9.0, and the library maintains an active development pace with frequent minor releases and occasional breaking changes.

Common errors

```
TypeError: scoringrules.crps.gaussian() missing 1 required positional argument: 'observations'
```
cause Attempting to use pre-v0.5.0 argument order (e.g., `sr.crps.gaussian(mean, std, observations)`) where `observations` was not the first argument.

fix Update your function call to place `observations` as the first positional argument: `sr.crps.gaussian(observations, mean, std)`.
```
TypeError: scoringrules.crps.gaussian() got multiple values for argument 'observations'
```
cause Passing observations both positionally (as the first argument) and as a keyword argument (e.g., `observations=...`) after v0.5.0.

fix Pass `observations` either positionally or as a keyword argument, but not both. For clarity and consistency with post-v0.5.0 API, use `sr.crps.gaussian(observations, mean, std)`.
```
AttributeError: module 'scoringrules.crps' has no attribute 'some_old_function_name'
```
cause A specific scoring function or its name was changed or removed in a breaking release (e.g., v0.8.0 introduced refactoring and renaming).

fix Check the release notes for your version or the latest documentation to find the new function name or usage pattern. For example, `crps_t` was made more robust in v0.9.0, possibly implying changes.
```
ImportError: cannot import name 'crps' from 'scoringrules'
```
cause Trying to import submodules like `crps` or `logs` directly from the top-level `scoringrules` package instead of as `scoringrules.crps`.

fix Use the fully qualified import path: `import scoringrules.crps as sr_crps` or `import scoringrules as sr` and then access `sr.crps`.

Warnings

breaking The order of positional arguments for all scoring functions changed. Observations must now always be the first positional argument.
Fix: Ensure that `observations` are passed as the first argument to all scoring functions (e.g., `sr.crps.gaussian(observations, mean, std)`).
breaking Some function arguments were renamed, potentially breaking code that relied on specific keyword argument names or positional arguments that were affected by renaming.
Fix: Consult the official documentation or release notes for v0.8.0 to identify renamed arguments and update your code accordingly. For example, some arguments like `obs` might have been explicitly renamed or absorbed into new positional schemes.
deprecated Version 0.7.0 contained critical bugs. Users on this version are strongly advised to upgrade.
Fix: Upgrade to version 0.7.1 or newer (`pip install --upgrade scoringrules`).
gotcha When using Numba (which is a default dependency), the first call to a JIT-compiled scoring function can be noticeably slower due to the compilation process.
Fix: This is expected behavior. Subsequent calls to the same function with similar input shapes will be significantly faster. For benchmarking, disregard the first call.

Install

pip install scoringrules Install base package
pip install scoringrules[torch] Install with PyTorch backend support
pip install scoringrules[jax] Install with JAX backend support

Imports

scoringrules
```
import scoringrules as sr
```
crps
wrong:
```
from scoringrules import crps
```
correct:
```
import scoringrules.crps as sr_crps
```
While `from scoringrules import crps` works, it's generally recommended to import the top-level package or specific modules with aliases to avoid name clashes and for clearer structure.

gaussian

sr.crps.gaussian(observations, mean, std)

Quickstart

This quickstart calculates the Continuous Ranked Probability Score (CRPS) for a set of Gaussian probabilistic forecasts against corresponding observations. It demonstrates the basic usage pattern with `numpy` arrays and the `crps.gaussian` function, highlighting the argument order requirement.

import numpy as np
import scoringrules as sr

# Generate some synthetic data
forecast_mean = np.array([0.1, 0.2, 0.3, 0.4])
forecast_std = np.array([0.5, 0.5, 0.5, 0.5])
observations = np.array([0.1, 0.1, 0.3, 0.5])

# Calculate CRPS for a Gaussian distribution
# Observations must be the first argument since v0.5.0
crps_scores = sr.crps.gaussian(observations, forecast_mean, forecast_std)
print(f"CRPS scores: {crps_scores}")
print(f"Mean CRPS: {np.mean(crps_scores)}")

view raw JSON →