Conda Package Streaming

0.12.0 · active · verified Thu Apr 16

An efficient library to read from new and old format .conda and .tar.bz2 conda packages. It enables downloading conda metadata from packages without transferring the entire file and getting metadata from local .tar.bz2 packages without reading entire files. The library, currently at version 0.12.0, uses enhanced pip lazy_wheel for `.conda` files and `tarfile.open` for `.tar.bz2` to stream data efficiently. It maintains a regular release cadence, with major updates roughly every few months.

Common errors

```
ModuleNotFoundError: No module named 'conda_package_streaming.package_streaming'
```
cause Attempting to import `package_streaming` or its functions directly from the top-level `conda_package_streaming` module, instead of as a submodule.

fix Correct the import path to `from conda_package_streaming import package_streaming` and then access functions like `package_streaming.stream_conda_info`.
```
json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
cause Attempting to `json.load()` a `tarfile.TarInfo` object directly, or an extracted file-like object that is empty or does not contain valid JSON, often due to an incorrect `member.name` check or reading the wrong file.

fix Ensure `member.name` correctly identifies a JSON file (e.g., `'info/index.json'`) and that `tar.extractfile(member)` returns a valid, non-empty file-like object containing well-formed JSON before attempting to load it.

Warnings

gotcha The behavior of `stream_conda_info` differs between `.conda` and `.tar.bz2` formats. For `.tar.bz2`, it yields all members, while for `.conda`, it yields members from the requested inner archive, allowing early termination.
Fix: Be mindful of the package format you are processing. For `.conda` files, you can `break` early once the desired metadata is found. For `.tar.bz2`, be prepared to iterate through more members or implement specific filtering.
gotcha When using `conda_reader_for_url` for `.conda` files, the returned file-like object *must* be seekable. For `.tar.bz2` files, it only needs to be readable.
Fix: Ensure your file-like objects for `.conda` are seekable if using `conda_reader_for_url`. If dealing with non-seekable streams and `.conda` packages, consider alternative streaming approaches or pre-downloading if seeking is unavoidable.
gotcha This library is optimized for *streaming metadata* and individual components without full package download. Using its URL-based APIs to extract an entire package is inefficient and against its design; prefer pre-downloading the package for full extraction.
Fix: If your use case requires the entire package content, download the `.conda` or `.tar.bz2` file first using a standard HTTP client, then use the file-based APIs provided by `conda-package-streaming` or `conda-package-handling`.

Install

pip install conda-package-streaming Pip
conda install conda-package-streaming -c conda-forge Conda (Conda-Forge channel)

Imports

stream_conda_info
wrong:
```
from conda_package_streaming import stream_conda_info
```
correct:
```
from conda_package_streaming.url import stream_conda_info
```
The primary streaming functions like `stream_conda_info` are nested within submodules like `url`, `s3`, or directly under `package_streaming` for local files, not directly under the top-level package.
stream_conda_info (for S3)
```
from conda_package_streaming.s3 import stream_conda_info
```
Use the `s3` submodule for streaming directly from S3 buckets, requiring `boto3` to be installed.
stream_conda_info (for local files)
```
from conda_package_streaming import package_streaming
# then use package_streaming.stream_conda_info(...)
```
For local file paths, `stream_conda_info` is accessed via the `package_streaming` submodule.
conda_reader_for_url
```
from conda_package_streaming.url import conda_reader_for_url
```
Provides a file-like object from a URL, used for more granular control over streaming.

Quickstart

This quickstart demonstrates how to stream package metadata (specifically `info/index.json`) from a remote `.conda` or `.tar.bz2` package URL without downloading the entire file. It iterates through the package's members and extracts the `index.json` content.

import json
from conda_package_streaming.url import stream_conda_info

# Replace with a valid .conda or .tar.bz2 URL
# For example: url = 'https://repo.anaconda.com/pkgs/main/linux-64/python-3.9.7-h62f7035_1.conda'
url = 'https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.13-h5eee18b_0.conda'

print(f"Streaming info from: {url}")

try:
    found_index_json = False
    for tar, member in stream_conda_info(url):
        if member.name == 'info/index.json':
            index_json = json.load(tar.extractfile(member))
            print("\n--- Found info/index.json ---")
            print(json.dumps(index_json, indent=2))
            found_index_json = True
            break # Stop once index.json is found
    
    if not found_index_json:
        print("\n--- info/index.json not found in package ---")

except Exception as e:
    print(f"An error occurred: {e}")

view raw JSON →