SWE-bench CLI

Warnings

• gotcha sb-cli requires Python 3.10 or newer. Attempting to use it with older Python versions will result in installation or runtime errors.
  Fix: Upgrade your Python environment to 3.10 or later (`python -m pip install --upgrade python` or use `pyenv`, `conda`).
• gotcha Submitting predictions requires an API token, which must be configured using `sb config --api-token YOUR_TOKEN`. Failure to do so will result in authentication errors when interacting with the SWE-bench API.
  Fix: Run `sb config --api-token YOUR_API_TOKEN` once to set up your authentication. The token is typically obtained from the SWE-bench website.
• gotcha The `sb submit` command relies on local file paths for both the SWE-bench instance and prediction files. Incorrect or inaccessible paths will lead to submission failures.
  Fix: Ensure that the `instance_path` and `prediction_path` arguments point to valid, existing, and readable JSON files on your filesystem.
• gotcha sb-cli is designed primarily as a command-line interface. While parts of its internal Python code can be imported, there isn't a high-level, stable, or officially documented programmatic Python API for general library use. Most interactions are expected to occur via shell commands.
  Fix: Interact with sb-cli through `subprocess` calls in Python scripts, mirroring how you would use it directly from the command line. Avoid relying on direct imports of internal modules unless specifically for testing or advanced scenarios.

Install

• pip install sb-cli Install sb-cli

Imports

• app

  from sb_cli.cli import app

  While sb-cli is primarily a command-line tool, the main Typer application object can be imported for advanced programmatic execution or testing purposes, typically using `typer.testing.CliRunner`.

Quickstart

This quickstart demonstrates how to programmatically execute the `sb submit` command using Python's `subprocess` module. Before running, ensure `sb-cli` is installed and configured with your SWE-bench API token (`sb config --api-token YOUR_TOKEN`). You also need to have started a run using `sb start` and provide valid paths to your SWE-bench instance and prediction files.

import subprocess
import os

# NOTE: This quickstart demonstrates submitting a run via the CLI.
# You must first configure your SWE-bench API token using `sb config --api-token YOUR_TOKEN`
# and start a run using `sb start` before submitting.

# Placeholder for your instance and prediction paths
instance_path = os.environ.get('SB_INSTANCE_PATH', 'path/to/your/swe_bench_instance.json')
prediction_path = os.environ.get('SB_PREDICTION_PATH', 'path/to/your/prediction.json')

if not os.path.exists(instance_path) or not os.path.exists(prediction_path):
    print(f"Warning: Placeholder paths used. Please replace '{instance_path}' and '{prediction_path}' "
          "with actual file paths for a successful submission.")
    print("You may need to run 'sb config' and 'sb start' first.")
    # Exit or mock for demonstration if files don't exist
    # For a real run, ensure these paths are valid.
    # For this example, we will just print the command without executing if paths are missing.
    command = [
        "sb",
        "submit",
        "--instance-path", instance_path,
        "--prediction-path", prediction_path,
        "--log-dir", "./sb_logs" # Optional: specify a log directory
    ]
    print(f"Mock command to execute: {' '.join(command)}")
else:
    print("Attempting to submit SWE-bench prediction...")
    command = [
        "sb",
        "submit",
        "--instance-path", instance_path,
        "--prediction-path", prediction_path,
        "--log-dir", "./sb_logs" # Optional: specify a log directory
    ]
    try:
        result = subprocess.run(command, capture_output=True, text=True, check=True)
        print("Submission successful!")
        print("STDOUT:", result.stdout)
        if result.stderr: print("STDERR:", result.stderr)
    except subprocess.CalledProcessError as e:
        print("Submission failed!")
        print("STDOUT:", e.stdout)
        print("STDERR:", e.stderr)
    except FileNotFoundError:
        print("Error: 'sb' command not found. Is sb-cli installed and in your PATH?")