HF Transfer (hf-transfer)

0.1.9 · active · verified Thu Apr 09

hf-transfer is a utility library designed to significantly speed up file transfers to and from the Hugging Face Hub. It works by optimizing the underlying transfer mechanisms used by the `huggingface_hub` library. The current version is 0.1.9, and it generally follows the release cadence of the broader Hugging Face ecosystem, with updates often tied to improvements in hub interaction.

Warnings

breaking The environment variable for enabling hf-transfer changed from `HF_HUB_ENABLE_HF_TRANSFER` to `HF_TRANSFER_ENABLED`.
Fix: Use `HF_TRANSFER_ENABLED=1` or `hf_transfer.enable_transfer()` in code. For older versions of `huggingface_hub` that might check the old variable, ensure compatibility or update `huggingface_hub`.
gotcha hf-transfer must be enabled *before* any `huggingface_hub` operations are called for acceleration to take effect.
Fix: Call `hf_transfer.enable_transfer()` at the very beginning of your script, or set the `HF_TRANSFER_ENABLED=1` environment variable before running your Python script.
gotcha hf-transfer does not provide standalone download/upload functionality; it acts as an accelerator for the `huggingface_hub` library. Therefore, `huggingface_hub` is a required dependency.
Fix: Ensure `huggingface_hub` is installed (`pip install huggingface_hub`) alongside `hf-transfer`.
gotcha Performance benefits of hf-transfer are most noticeable for large files or many small files, and can be influenced by network conditions and local storage speed. It may not provide significant speedups for very small transfers.
Fix: Evaluate performance in your specific use case. If you experience issues, consider checking network configuration, firewall settings, and disk I/O performance.

Install

pip install hf-transfer Install hf-transfer

Imports

enable_transfer
```
import hf_transfer
hf_transfer.enable_transfer()
```
The primary interaction is by calling `hf_transfer.enable_transfer()` to patch `huggingface_hub`'s behavior.

Quickstart

This quickstart demonstrates how to enable `hf_transfer` within your Python script and then use `huggingface_hub`'s `snapshot_download` function. `hf_transfer` will automatically accelerate the download operation once `hf_transfer.enable_transfer()` is called. Remember that `huggingface_hub` must also be installed for this to work.

import hf_transfer
from huggingface_hub import snapshot_download
import os

# Enable hf_transfer to accelerate subsequent Hugging Face Hub operations
hf_transfer.enable_transfer()

# Example: Download a small model from the Hugging Face Hub
# This operation will now be accelerated by hf_transfer
repo_id = "HuggingFaceH4/no_deid_test_data"
local_dir = "./no_deid_test_data_hf_transfer"

print(f"Downloading {repo_id} to {local_dir} with hf_transfer acceleration...")

try:
    snapshot_download(repo_id=repo_id, local_dir=local_dir)
    print(f"Successfully downloaded {repo_id} to {local_dir}")
except Exception as e:
    print(f"An error occurred during download: {e}")
finally:
    # Clean up downloaded files for subsequent runs if desired
    # import shutil
    # if os.path.exists(local_dir):
    #     shutil.rmtree(local_dir)
    pass

view raw JSON →