Icechunk

Warnings

• breaking Icechunk 2.0.0 and later requires Python 3.12 or higher. Support for Python 3.11 was dropped.
  Fix: Upgrade your Python environment to 3.12 or a newer compatible version.
• breaking The on-disk storage format changed with Icechunk 2.0.0. Existing repositories created with Icechunk 1.x must be migrated using the `upgrade_icechunk_repository()` function. This is an administrative operation and must be executed in isolation (no other readers/writers).
  Fix: Use `ic.upgrade_icechunk_repository(repo, dry_run=False)` to migrate your 1.x repository to the 2.0 format. Ensure no other processes are accessing the repository during migration.
• breaking Enums like `ChunkType` had their variants renamed from `UPPER_CASE` to `snake_case` (e.g., `ChunkType.INLINE` became `ChunkType.inline`).
  Fix: Update your code to use the new `snake_case` enum variant names.
• gotcha After a `writable_session.commit()` is successfully executed, that session becomes read-only. To make further changes and commit them, you must create a new `writable_session`.
  Fix: Always obtain a new `repo.writable_session()` instance for each set of modifications you intend to commit.
• gotcha Concurrent creation of an Icechunk repository in the same location from multiple processes is not safe.
  Fix: Ensure that repository creation is a singular, isolated operation. Once created, repositories can be opened concurrently.
• gotcha Icechunk's version policy allows breaking API changes to occur in minor releases (e.g., `2.0.0` to `2.1.0`), not just major versions, to align library versions with the on-disk format.
  Fix: Always review the changelog or release notes thoroughly before upgrading to any new minor version of Icechunk to understand potential breaking changes.

Install

• pip install icechunk Install with pip

Imports

• Repository

  from icechunk import Repository

  While submodules exist (e.g., `icechunk.repository`), the main classes and storage factories are often exposed directly at the top level for convenience.
• s3_storage

  from icechunk import s3_storage

  Storage factory functions are typically available directly from the top-level `icechunk` package.

Quickstart

This quickstart demonstrates how to create a local Icechunk repository, interact with it using Zarr through a writable session, commit changes, and then make further modifications requiring a new session. It also shows how to review the repository's commit history.

import icechunk as ic
import zarr
import numpy as np
import tempfile
import os

# Create a temporary directory for the local repository
temp_dir = tempfile.TemporaryDirectory()
repo_path = os.path.join(temp_dir.name, "my_icechunk_repo")

try:
    # 1. Create a new Icechunk repository on the local filesystem
    storage = ic.local_filesystem_storage(repo_path)
    repo = ic.Repository.create(storage)
    print(f"Repository created at: {repo_path}")

    # 2. Create a writable session on the 'main' branch
    session = repo.writable_session("main")

    # 3. Access the Zarr store from the session
    store = session.store # A zarr store

    # 4. Use Zarr to create a group and an array
    root = zarr.group(store=store)
    data = np.arange(1000).reshape(10, 10, 10)
    zarr_array = root.create_array(
        'my_data',
        shape=data.shape,
        dtype=data.dtype,
        chunks=(5, 5, 5)
    )
    zarr_array[:] = data

    # 5. Commit the changes
    snapshot_id = session.commit("Initial data commit")
    print(f"First commit successful with snapshot ID: {snapshot_id}")

    # A new session is required for further writes after a commit
    session_2 = repo.writable_session("main")
    store_2 = session_2.store
    zarr_array_2 = zarr.open_array(store_2, 'my_data', mode='r+')
    zarr_array_2[:5, :5, :5] = 999 # Overwrite a subset
    snapshot_id_2 = session_2.commit("Overwrite some values")
    print(f"Second commit successful with snapshot ID: {snapshot_id_2}")

    # 6. Explore version history
    print("\nRepository history:")
    for snapshot in repo.log("main"):
        print(f"  ID: {snapshot.id}, Message: {snapshot.commit_message}")

finally:
    # Clean up the temporary directory
    temp_dir.cleanup()
    print(f"\nCleaned up temporary directory: {temp_dir.name}")