{"id":8442,"library":"prefect-ray","title":"Prefect-Ray","description":"Prefect-Ray provides integrations for the Prefect workflow orchestration framework with the Ray distributed execution framework. It enables Prefect flows to run tasks in parallel using Ray, either by creating a temporary local Ray instance or connecting to an existing remote one. The library is actively maintained and currently at version 0.4.5, with its own versioning separate from the main Prefect library.","status":"active","version":"0.4.5","language":"en","source_language":"en","source_url":"https://github.com/PrefectHQ/prefect/tree/main/src/integrations/prefect-ray","tags":["prefect","ray","workflow","distributed","orchestration","parallel-processing"],"install":[{"cmd":"pip install \"prefect[ray]\"","lang":"bash","label":"Recommended for full compatibility"},{"cmd":"pip install prefect-ray","lang":"bash","label":"Minimal installation"}],"dependencies":[{"reason":"Core workflow orchestration library.","package":"prefect","optional":false},{"reason":"Distributed execution framework.","package":"ray","optional":false},{"reason":"Runtime environment.","package":"Python","optional":false,"version_range":"!=3.13,>=3.9"}],"imports":[{"note":"RayTaskRunner is located in the task_runners submodule.","wrong":"from prefect_ray import RayTaskRunner","symbol":"RayTaskRunner","correct":"from prefect_ray.task_runners import RayTaskRunner"}],"quickstart":{"code":"import time\nfrom prefect import flow, task\nfrom prefect_ray.task_runners import RayTaskRunner\n\n@task\ndef shout(number: int):\n    time.sleep(0.5)\n    print(f\"#{number}\")\n\n@flow(task_runner=RayTaskRunner())\ndef count_to(highest_number: int):\n    # Tasks are submitted to Ray for parallel execution\n    shout.map(range(highest_number)).wait()\n\nif __name__ == \"__main__\":\n    count_to(10)\n    # Example of connecting to an existing Ray instance:\n    # count_to(10, task_runner=RayTaskRunner(address=\"ray://localhost:10001\"))","lang":"python","description":"This quickstart demonstrates how to define a Prefect flow and tasks, then configure the flow to use the `RayTaskRunner`. By default, a temporary local Ray instance is created. You can also specify an `address` to connect to an existing Ray cluster."},"warnings":[{"fix":"Use a compatible Python version (e.g., `python<=3.12`).","message":"Python 3.13 is not supported due to Ray's experimental support and Prefect's current lack of support. Ensure your environment uses Python 3.9 through 3.12.","severity":"breaking","affected_versions":"All versions"},{"fix":"Consult Ray documentation for ARM/M1 installation, typically involving `conda install grpcio` after `pip uninstall grpcio`.","message":"Users on Apple Silicon (M1/ARM processors) may experience issues with `ray` installation via `pip`. Manual installation of blocking components (like `grpcio`) using `conda` is often required.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Ensure `prefect-ray` and all task dependencies are installed on the Ray cluster. For remote clusters, explicitly pass `init_kwargs={'runtime_env': {'pip': ['prefect-ray', 'your-task-dep']}}` to `RayTaskRunner` for worker dependencies, and ensure the driver environment also has `prefect-ray` installed.","message":"When using `RayTaskRunner` with a remote Ray cluster (i.e., specifying the `address` argument), dependencies like `prefect-ray` and any packages used in your tasks must be installed on the Ray head node and workers. `runtime_env` might be needed for worker-specific dependencies but does not cover the driver's imports.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Configure Prefect to use a robust external database (e.g., PostgreSQL) for production or parallel workloads. Refer to Prefect's documentation for database setup.","message":"Prefect's default SQLite database is not suitable for concurrent access when using distributed task runners like `RayTaskRunner`. This can lead to database corruption or unexpected behavior.","severity":"gotcha","affected_versions":"All versions"},{"fix":"Explicitly include necessary environment variables in the `runtime_env` within `init_kwargs` for `RayTaskRunner` (e.g., `RayTaskRunner(address='...', init_kwargs={'runtime_env': {'env_vars': {'PREFECT_API_URL': 'http://your-prefect-api:4200/api'}}})`).","message":"Environment variables, such as `PREFECT_API_URL`, configured in the Prefect deployment or execution environment might not propagate correctly to Ray tasks when they run on a remote Ray cluster via `RayTaskRunner`. This can cause tasks to fail connecting to the Prefect API.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Ensure `prefect-ray` is installed on all Ray cluster nodes. If using `RayTaskRunner(address=...)`, pass `init_kwargs={'runtime_env': {'pip': ['prefect-ray']}}` to ensure workers have it.","cause":"The `prefect-ray` library is not installed or accessible in the Python environment of the Ray workers or head node when connecting to a remote Ray cluster.","error":"ModuleNotFoundError: No module named 'prefect_ray'"},{"fix":"Set `PREFECT_API_URL` explicitly in the `runtime_env` for the `RayTaskRunner`'s `init_kwargs`. For example: `RayTaskRunner(address=\"ray://your-remote-ray-head:10001\", init_kwargs={\"runtime_env\": {\"env_vars\": {\"PREFECT_API_URL\": \"http://your-prefect-api-url:4200/api\"}}})`.","cause":"Ray tasks, especially on a remote cluster, are attempting to connect to the default local Prefect API URL instead of the configured remote one, likely due to environment variable propagation issues.","error":"RuntimeError: Failed to reach API at http://127.0.0.1:4200/api/"},{"fix":"This often points to a complex interaction between Ray and Prefect's async execution. Ensure compatible versions of Ray and Prefect. Simplifying resource specifications or re-evaluating the task execution model might be necessary. Check Prefect and Ray GitHub issues for similar reports and workarounds.","cause":"This error can occur in specific scenarios involving Ray Client and how event loops are managed, particularly when resources are specified for tasks or there are concurrency conflicts.","error":"RuntimeError: There is no current event loop in thread 'ray_client_server_1'"},{"fix":"Try clearing the Prefect data directory (`~/.prefect`), uninstalling all Prefect-related packages (`prefect`, `prefect-ray`, etc.), and reinstalling them. Restarting the system can also help.","cause":"Intermittent issues with the Prefect local server becoming unreachable when `prefect-ray` is installed, sometimes after OS upgrades or reinstallation attempts. Often related to corrupted local Prefect configuration or caching.","error":"Local server randomly not working with `prefect-ray`"}]}