DVC Data Management Subsystem

3.18.3 · active · verified Sat Apr 11

dvc-data is DVC's core data management subsystem, providing functionalities for hashing, indexing, caching, and interacting with various storage backends. As a foundational library for DVC, it receives frequent patch and minor releases to enhance performance, expand compatibility, and fix bugs.

Warnings

breaking DVC 3.0 introduced significant changes to file hashing (removed CRLF conversion) and cache storage locations. Old '.dvc' files containing deprecated keys (like 'metric', 'param', 'plot', 'cmd') will cause validation errors and older cache structures are incompatible.
Fix: Run `dvc cache migrate` to move existing DVC 2.x cache data to the 3.x format. Manually remove deprecated keys from `.dvc` files or regenerate them using `dvc add` if issues persist. Ensure consistent line endings in text files across platforms.
gotcha The `bulk_exists` method in earlier 3.x versions had issues with filesystems not implementing `ls` or when handling duplicate hashes, potentially leading to incorrect results during data existence checks.
Fix: Upgrade to `dvc-data` version 3.18.3 or newer to benefit from fixes for `bulk_exists` behavior across various filesystems and scenarios.
gotcha Earlier versions might mishandle protocol information when filtering changed metadata in `_filter_changed`, potentially leading to inaccurate detection of data changes, especially with custom protocols.
Fix: Upgrade to `dvc-data` version 3.18.3 or newer to ensure correct `protocol` propagation in metadata operations.
gotcha When working with DVC commands like `dvc pull`, `fetch`, or `push` (which rely on `dvc-data`), users might encounter '[Errno 24] Too many open files' errors, especially on macOS with S3 remotes, when using many `--jobs`.
Fix: Increase the operating system's open file descriptor limit (`ulimit -n` on UNIX-like systems) or use a lower value for the `--jobs` parameter.

Install

pip install dvc-data Basic Install
pip install "dvc-data[s3]" Install with S3 support
pip install "dvc-data[all]" Install with all cloud support

Imports

DataIndex
```
from dvc_data.index import DataIndex
```
Used for managing collections of data entries, often representing directories.
HashFile
```
from dvc_data.hashfile import HashFile
```
Represents a single file with content-addressable hashing information.
HashFileDB
```
from dvc_data.hashfile import HashFileDB
```
Manages a database of HashFile objects, similar to Git's object store.

Quickstart

This quickstart demonstrates the core functionality of `dvc-data` by creating a `DataIndex` and a `DataIndexEntry` for a local file, simulating how DVC tracks data. It uses `dvc_data.hashfile.hash.file_md5` to calculate the MD5 hash and stores it within the index entry.

import os
from dvc_data.index import DataIndex, DataIndexEntry
from dvc_data.hashfile.hash import file_md5

# Create a dummy file
with open('data.txt', 'w') as f:
    f.write('hello dvc-data world')

# Calculate its MD5 hash
md5_hash = file_md5('data.txt', os.fspath, None)

# Create a DataIndexEntry for the file
entry = DataIndexEntry(
    path='data.txt',
    hash_info={'name': 'md5', 'value': md5_hash}
)

# Add the entry to a DataIndex
index = DataIndex()
index[('data.txt',)] = entry

print(f"Created DataIndex with entry for 'data.txt': {index.has_entries}")
print(f"MD5 for 'data.txt': {index[('data.txt',)].hash_info['value']}")

# Clean up
os.remove('data.txt')

view raw JSON →