{"id":3268,"library":"sb-cli","title":"SWE-bench CLI","description":"sb-cli is a command-line interface (CLI) tool designed to interact with the SWE-bench API. It enables users to submit predictions for SWE-bench instances, manage their runs, and track progress. The current version is 0.1.5, and it has an active development cadence driven by the SWE-bench community.","status":"active","version":"0.1.5","language":"en","source_language":"en","source_url":"https://github.com/swe-bench/sb-cli","tags":["cli","swe-bench","machine-learning","evaluation","tool"],"install":[{"cmd":"pip install sb-cli","lang":"bash","label":"Install sb-cli"}],"dependencies":[],"imports":[{"note":"While sb-cli is primarily a command-line tool, the main Typer application object can be imported for advanced programmatic execution or testing purposes, typically using `typer.testing.CliRunner`.","symbol":"app","correct":"from sb_cli.cli import app"}],"quickstart":{"code":"import subprocess\nimport os\n\n# NOTE: This quickstart demonstrates submitting a run via the CLI.\n# You must first configure your SWE-bench API token using `sb config --api-token YOUR_TOKEN`\n# and start a run using `sb start` before submitting.\n\n# Placeholder for your instance and prediction paths\ninstance_path = os.environ.get('SB_INSTANCE_PATH', 'path/to/your/swe_bench_instance.json')\nprediction_path = os.environ.get('SB_PREDICTION_PATH', 'path/to/your/prediction.json')\n\nif not os.path.exists(instance_path) or not os.path.exists(prediction_path):\n    print(f\"Warning: Placeholder paths used. Please replace '{instance_path}' and '{prediction_path}' \"\n          \"with actual file paths for a successful submission.\")\n    print(\"You may need to run 'sb config' and 'sb start' first.\")\n    # Exit or mock for demonstration if files don't exist\n    # For a real run, ensure these paths are valid.\n    # For this example, we will just print the command without executing if paths are missing.\n    command = [\n        \"sb\",\n        \"submit\",\n        \"--instance-path\", instance_path,\n        \"--prediction-path\", prediction_path,\n        \"--log-dir\", \"./sb_logs\" # Optional: specify a log directory\n    ]\n    print(f\"Mock command to execute: {' '.join(command)}\")\nelse:\n    print(\"Attempting to submit SWE-bench prediction...\")\n    command = [\n        \"sb\",\n        \"submit\",\n        \"--instance-path\", instance_path,\n        \"--prediction-path\", prediction_path,\n        \"--log-dir\", \"./sb_logs\" # Optional: specify a log directory\n    ]\n    try:\n        result = subprocess.run(command, capture_output=True, text=True, check=True)\n        print(\"Submission successful!\")\n        print(\"STDOUT:\", result.stdout)\n        if result.stderr: print(\"STDERR:\", result.stderr)\n    except subprocess.CalledProcessError as e:\n        print(\"Submission failed!\")\n        print(\"STDOUT:\", e.stdout)\n        print(\"STDERR:\", e.stderr)\n    except FileNotFoundError:\n        print(\"Error: 'sb' command not found. Is sb-cli installed and in your PATH?\")","lang":"python","description":"This quickstart demonstrates how to programmatically execute the `sb submit` command using Python's `subprocess` module. Before running, ensure `sb-cli` is installed and configured with your SWE-bench API token (`sb config --api-token YOUR_TOKEN`). You also need to have started a run using `sb start` and provide valid paths to your SWE-bench instance and prediction files."},"warnings":[{"fix":"Upgrade your Python environment to 3.10 or later (`python -m pip install --upgrade python` or use `pyenv`, `conda`).","message":"sb-cli requires Python 3.10 or newer. Attempting to use it with older Python versions will result in installation or runtime errors.","severity":"gotcha","affected_versions":"<0.1.0 (all versions)"},{"fix":"Run `sb config --api-token YOUR_API_TOKEN` once to set up your authentication. The token is typically obtained from the SWE-bench website.","message":"Submitting predictions requires an API token, which must be configured using `sb config --api-token YOUR_TOKEN`. Failure to do so will result in authentication errors when interacting with the SWE-bench API.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure that the `instance_path` and `prediction_path` arguments point to valid, existing, and readable JSON files on your filesystem.","message":"The `sb submit` command relies on local file paths for both the SWE-bench instance and prediction files. Incorrect or inaccessible paths will lead to submission failures.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Interact with sb-cli through `subprocess` calls in Python scripts, mirroring how you would use it directly from the command line. Avoid relying on direct imports of internal modules unless specifically for testing or advanced scenarios.","message":"sb-cli is designed primarily as a command-line interface. While parts of its internal Python code can be imported, there isn't a high-level, stable, or officially documented programmatic Python API for general library use. Most interactions are expected to occur via shell commands.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-11T00:00:00.000Z","next_check":"2026-07-10T00:00:00.000Z"}