Hugging Face Xet

1.4.2 · active · verified Sat Mar 28

hf-xet is a foundational Python library that provides client technology for Xet storage, deeply integrated within the Hugging Face Hub ecosystem. It enables efficient, chunk-based deduplicated transfer of large files, such as machine learning models and datasets, to and from the Hugging Face Hub. This Rust-based library serves primarily as a backend for `huggingface_hub` and is not typically intended for direct user interaction. The project maintains an active development status, with releases often coinciding with or supporting major updates to the `huggingface_hub` library.

Warnings

gotcha hf-xet is an underlying dependency for `huggingface_hub` and is not designed for direct user interaction or import in typical Python code. All functionalities, including optimized large file transfers, are exposed through the `huggingface_hub` API.
Fix: Interact with Hugging Face Hub repositories via the `huggingface_hub` library (e.g., `huggingface_hub.snapshot_download`, `huggingface_hub.upload_file`). Ensure `huggingface_hub` version is >=0.32.0 for automatic `hf-xet` integration.
breaking Installing `hf-xet` via `pip install --no-binary hf-xet` or in non-standard environments (like Termux) could fail due to `maturin` (Rust-Python binding) build issues, particularly with older versions (e.g., 1.0.3). This might manifest as 'Failed to normalize python source path `python`' errors.
Fix: Prefer installing `hf-xet` via pre-built wheels (`pip install hf-xet`). If building from source is necessary and issues arise, check `huggingface/xet-core` GitHub issues for workarounds or try upgrading `maturin` and Python toolchains. For Termux, local cloning and installation might be required.
gotcha While `hf-xet` aims to improve performance, some users have reported occasional 503 errors, silent hangs at 90-99%, or issues resuming downloads with `xet` compared to conventional `hub` downloads, particularly in early versions or specific network conditions.
Fix: Adaptive concurrency has been enabled to address some stability issues. For persistent download issues, environment variables like `HF_XET_HIGH_PERFORMANCE=1`, `HF_XET_CHUNK_CACHE_SIZE_BYTES=0`, and specific cache path settings might help, or consider using `huggingface_hub.snapshot_download` with a valid `HF_TOKEN`.
breaking The `huggingface_hub` library (which depends on `hf-xet`) updated its minimum Python version to 3.9 from 3.8 starting with `huggingface_hub` v1.0. While `hf-xet` itself lists `>=3.8`, using it with a modern `huggingface_hub` version implies the higher Python requirement.
Fix: Ensure your Python environment is running Python 3.9 or newer when working with recent versions of `huggingface_hub` and its dependencies, including `hf-xet`.

Install

pip install hf-xet Install `hf-xet`

Imports

No direct imports for end-users
```
hf-xet is primarily an internal dependency of huggingface_hub. Users interact through huggingface_hub's API.
```
hf-xet provides Python bindings to Rust code and is designed to be used implicitly by `huggingface_hub` for optimized data transfer, rather than through direct imports in user code.

Quickstart

This quickstart demonstrates how `hf-xet` is utilized implicitly by the `huggingface_hub` library for efficient large file transfers. While `hf-xet` is not directly imported, operations like `snapshot_download` or `upload_file` from `huggingface_hub` will leverage `hf-xet`'s chunk-based deduplication and optimized transfer protocols for Xet-enabled repositories.

import os
from huggingface_hub import snapshot_download

# Ensure you have a Hugging Face token set as an environment variable (HF_TOKEN)
# For example: os.environ['HF_TOKEN'] = 'hf_YOUR_TOKEN_HERE'
# You can generate a token at: https://huggingface.co/settings/tokens

# hf-xet is used implicitly by huggingface_hub for large file transfers.
# This example downloads a small model, but the benefits of Xet become apparent
# with very large models or datasets that leverage its chunk-based deduplication.

try:
    # Downloading a model using huggingface_hub, which leverages hf-xet internally
    # for Xet-enabled repositories.
    model_path = snapshot_download(
        repo_id="google/fnet-tokenizer", # A small, example repo
        allow_patterns=["tokenizer.json"],
        local_dir="./fnet-tokenizer-local",
        token=os.environ.get('HF_TOKEN', None) # Pass token if set as env var
    )
    print(f"Model downloaded to: {model_path}")
    print("hf-xet was leveraged by huggingface_hub for efficient transfer.")
except Exception as e:
    print(f"An error occurred during download: {e}")
    print("Please ensure you have an active internet connection and a valid Hugging Face token if the repo is private or requires auth.")

view raw JSON →