{"id":7283,"library":"harbor","title":"Harbor (Agent Evaluation Framework)","description":"Harbor is an open-source framework, currently at version 0.3.0, designed for evaluating and optimizing AI agents and language models using sandboxed environments. It facilitates the creation and execution of benchmarks, allowing users to assess arbitrary agents and models. The library is actively developed, with its current release focusing on providing tools for robust and scalable agent evaluation. Release cadence is not explicitly stated but updates appear to be regular.","status":"active","version":"0.3.0","language":"en","source_language":"en","source_url":"https://github.com/laude-institute/harbor","tags":["AI","agents","evaluation","LLM","sandbox","benchmarking"],"install":[{"cmd":"pip install harbor","lang":"bash","label":"Using pip"},{"cmd":"uv tool install harbor","lang":"bash","label":"Using uv (recommended)"}],"dependencies":[{"reason":"Requires Python 3.12 or newer for compatibility.","package":"python","optional":false},{"reason":"Evaluations run in sandboxed Docker environments, requiring Docker to be installed and running on the system.","package":"docker","optional":false}],"imports":[{"note":"The primary interaction with the Harbor framework is often through its CLI, but modules can be imported for programmatic access to specific functionalities. The top-level `harbor` package is the entry point for most programmatic uses.","symbol":"Harbor","correct":"import harbor"}],"quickstart":{"code":"import os\nimport subprocess\n\n# Note: Harbor is primarily CLI-driven for running evaluations.\n# This example demonstrates a basic CLI interaction.\n# Ensure Docker is running and 'harbor' is installed.\n\n# Create a dummy task file for evaluation\ntask_content = \"print('Hello from Harbor evaluation!')\"\nwith open('hello_task.py', 'w') as f:\n    f.write(task_content)\n\nprint('Created hello_task.py')\n\n# Run a simple evaluation using the Harbor CLI\n# For actual evaluations, you would define an agent and environment.\n# This command is a placeholder demonstrating CLI invocation.\n# A real quickstart would involve defining a dataset and an agent.\n\ntry:\n    # Example of running a simple command, assuming a 'test' subcommand exists\n    # or a generic 'run' command without specific agent/dataset is possible.\n    # The official quickstart uses `harbor run` with datasets/environments.\n    # This generic call might not be directly runnable without setup.\n    print('Attempting to run a basic harbor CLI command...')\n    # As per documentation, a simple quickstart involves `harbor run` on a dataset.\n    # This is a simplified example. For a full eval, see official docs.\n    result = subprocess.run(\n        ['harbor', 'run', '--help'], # Or a specific dataset/agent for a real run\n        capture_output=True, text=True, check=True\n    )\n    print(\"Harbor CLI --help output:\\n\", result.stdout)\nexcept FileNotFoundError:\n    print(\"Error: 'harbor' command not found. Ensure Harbor is installed and in your PATH.\")\nexcept subprocess.CalledProcessError as e:\n    print(f\"Error running Harbor CLI: {e}\")\n    print(f\"Stdout: {e.stdout}\")\n    print(f\"Stderr: {e.stderr}\")\nfinally:\n    # Clean up dummy task file\n    if os.path.exists('hello_task.py'):\n        os.remove('hello_task.py')\n        print('Cleaned up hello_task.py')\n","lang":"python","description":"Harbor is primarily designed for command-line interaction to manage and execute agent evaluations. The quickstart typically involves using the `harbor` CLI tool to run evaluations against defined datasets and agents within Dockerized environments. This Python snippet demonstrates how you might invoke a basic `harbor` CLI command programmatically. For a full evaluation, users would define tasks, agents, and environments as described in the official Harbor documentation."},"warnings":[{"fix":"Always verify the PyPI project description and source repository to confirm you have the correct 'harbor' package for agent evaluation, not the container registry client.","message":"Harbor (the agent evaluation framework) shares a name with the Harbor Container Registry. Ensure you are installing and using the correct 'harbor' library, which is the agent evaluation framework, not the client libraries for the container registry (e.g., `harborapi`, `harbor-cli`, `harbor-api-client`).","severity":"gotcha","affected_versions":"All versions"},{"fix":"Before running any `harbor run` commands, ensure Docker Desktop or Docker Engine is installed and its daemon is active. Verify with `docker info` or `docker ps`.","message":"Running evaluations with Harbor requires Docker to be installed and running on your system, as it uses sandboxed Docker environments. Lack of a running Docker daemon will prevent evaluations from executing.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Familiarize yourself with the `harbor --help` output and the official documentation's CLI guides for running evaluations and managing components.","message":"The primary interaction pattern for Harbor is via its Command Line Interface (CLI). While it's a Python library, many core functionalities, especially running evaluations, are designed to be invoked through `harbor` CLI commands rather than direct Python imports of every component.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure you have correctly installed the agent evaluation framework: `pip install harbor` or `uv tool install harbor`.","cause":"The `harbor` package was not installed or is not accessible in the current Python environment, or a different package with a similar name (like `harborapi`) was installed instead.","error":"ModuleNotFoundError: No module named 'harbor'"},{"fix":"Install Docker Desktop (for macOS/Windows) or Docker Engine (for Linux) and ensure it's properly configured and added to your system's PATH. Verify installation with `docker --version`.","cause":"The Docker command-line client is not installed or not in your system's PATH, which is required for Harbor to provision evaluation environments.","error":"docker: command not found"},{"fix":"Verify that `pip install harbor` completed successfully. If using a virtual environment, ensure it's activated. Check your system's PATH configuration to include the directory where Python packages install their scripts (e.g., `~/.local/bin` or a virtual environment's `bin/Scripts` directory).","cause":"The `harbor` CLI tool, which is part of the Python package, is not accessible from your shell's PATH, or the installation was incomplete.","error":"Error: 'harbor' command not found. Ensure Harbor is installed and in your PATH."}]}