{"id":10021,"library":"parfive","title":"Parfive: Parallel File Downloader","description":"Parfive is an asynchronous HTTP and FTP parallel file downloader for Python. It leverages `asyncio` to efficiently download multiple files concurrently, providing features like progress bars, connection throttling, and retry mechanisms. The current version, 2.3.1, offers a robust asynchronous API for managing large-scale file transfers. Releases are made periodically to add features, address issues, and ensure compatibility with newer Python versions, maintaining an active development status.","status":"active","version":"2.3.1","language":"en","source_language":"en","source_url":"https://github.com/sunpy/parfive","tags":["download","http","ftp","async","parallel","networking","file-transfer","asyncio"],"install":[{"cmd":"pip install parfive","lang":"bash","label":"Install stable version"}],"dependencies":[{"reason":"Core HTTP client for asynchronous web requests.","package":"aiohttp"},{"reason":"Provides progress bar functionality during downloads.","package":"tqdm"},{"reason":"Used for managing and limiting concurrent tasks.","package":"asyncio-throttle"}],"imports":[{"note":"Downloader is a class directly exported by the parfive package, not a submodule.","wrong":"import parfive.Downloader","symbol":"Downloader","correct":"from parfive import Downloader"}],"quickstart":{"code":"import parfive\nimport asyncio\nimport os\n\nasync def main():\n    # Define some public URLs to download\n    urls = [\n        \"https://raw.githubusercontent.com/sunpy/parfive/main/README.md\",\n        \"https://raw.githubusercontent.com/sunpy/parfive/main/LICENSE\"\n    ]\n\n    # Create a directory for downloads if it doesn't exist\n    download_dir = \"parfive_downloads\"\n    os.makedirs(download_dir, exist_ok=True)\n\n    # Initialize the Downloader with a maximum of 5 concurrent connections\n    # and display a progress bar.\n    downloader = parfive.Downloader(max_conn=5, progress=True)\n\n    # Add URLs to the downloader, specifying the local path\n    for url in urls:\n        downloader.add_url(url, path=download_dir)\n\n    # Execute the downloads asynchronously\n    print(f\"Starting download of {len(urls)} files to '{download_dir}'...\")\n    results = await downloader.download()\n\n    # Print the paths of the downloaded files\n    print(\"Downloaded files:\")\n    for filepath in results:\n        print(f\"- {filepath}\")\n\n    # Optionally clean up the downloaded files\n    # for filepath in results:\n    #     os.remove(filepath)\n    # os.rmdir(download_dir)\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n","lang":"python","description":"This example demonstrates how to use `parfive.Downloader` to download multiple public files concurrently to a local directory. It shows how to initialize the downloader, add URLs, and await the results. The `asyncio.run()` function is used to execute the asynchronous main function."},"warnings":[{"fix":"Review the official `parfive` documentation for the 2.x API. Ensure all `parfive` operations are `await`ed and run within an `asyncio` event loop. Adapt code to handle `Results` objects.","message":"Version 2.0.0 introduced significant breaking changes, migrating the API to be fully `asyncio` native. The `Downloader` class signature and the return value of `download()` changed from a simple list of paths to a `parfive.results.Results` object which is a specialized list subclass.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Ensure your code calls `downloader.download()` with `await` and that the execution context is an `asyncio` event loop (e.g., using `asyncio.run()` for top-level script execution, or within an `await`-able function in an existing event loop).","message":"All `parfive` download operations are asynchronous and must be executed within an `asyncio` event loop. Forgetting to `await` the `downloader.download()` call or attempting to run it outside an event loop will lead to runtime errors.","severity":"gotcha","affected_versions":"All"},{"fix":"If fresh downloads are always needed, ensure target paths are unique (e.g., by including a timestamp) or explicitly delete existing files before calling `add_url`. Consider checking `overwrite=True` if that option exists or checking the `checksum`.","message":"By default, `parfive` skips downloading a file if it already exists at the target path and has the correct size. While often desired, this can lead to stale files if the remote content changes without a size change. It also means you need to manage unique filenames or explicitly remove existing files if fresh downloads are always required.","severity":"gotcha","affected_versions":"All"},{"fix":"Tune `max_conn` and `max_downloads` based on your system's capabilities, network bandwidth, and the policies of the servers you are downloading from. Start with conservative values and increase gradually if performance allows.","message":"The `max_conn` (default 5) and `max_downloads` (default 10) parameters of `Downloader` control concurrency. Setting these too high can exhaust system resources (file descriptors, network sockets) or trigger rate limits on target servers, leading to slower downloads or connection errors.","severity":"gotcha","affected_versions":"All"}],"env_vars":null,"last_verified":"2026-04-17T00:00:00.000Z","next_check":"2026-07-16T00:00:00.000Z","problems":[{"fix":"If in Jupyter, use `await` directly at the top level (if supported) or `nest_asyncio.apply()`. If within another async function, just `await` the parfive call without `asyncio.run()`.","cause":"Attempting to call `asyncio.run()` in an environment where an event loop is already active (e.g., inside a Jupyter notebook or another async function).","error":"RuntimeError: Event loop is already running"},{"fix":"Ensure you use `await downloader.download()` to properly execute the asynchronous download process and retrieve its results.","cause":"Forgetting the `await` keyword before calling an asynchronous function or method, specifically `downloader.download()`.","error":"TypeError: object asyncio.tasks.Task can't be awaited"},{"fix":"Change `parfive.Downloader` to `Downloader` after using `from parfive import Downloader` or ensure you are importing the `Downloader` class correctly.","cause":"Trying to access `parfive.Downloader` when `Downloader` is directly exported by the package and should be imported with `from parfive import Downloader`.","error":"AttributeError: module 'parfive' has no attribute 'Downloader'"}]}