Big Data Bag Utilities

Common errors

• FileNotFoundError: [Errno 2] No such file or directory: '/path/to/bag/data/nonexistent_file.txt'

  cause You tried to add a file to the bag's payload manifest that does not exist at the specified path on the filesystem, or the `data_directory` provided to `make_bag` did not contain the expected files.
  fix Ensure all files referenced in the bag's payload manifest actually exist at the locations bdbag expects them to be, relative to the `data_directory` or when using `add_file`.

• BagValidationError: Bag is invalid

  cause The bag's integrity checks failed. Common causes include missing payload files, incorrect checksums (if files were modified after bag creation), or issues with the bag's tag files (e.g., bag-info.txt, bagit.txt).
  fix Inspect the output or logs provided by `bdbag_api.validate_bag` for specific reasons for invalidation. Missing payload files or checksum mismatches are the most frequent causes. Regenerate checksums if files were altered, or ensure all files are present.

• ImportError: cannot import name 'BDBag' from 'bdbag'

  cause This usually happens when attempting to import `BDBag` directly from the `bdbag` module's top level without specifying the correct path, or if `bdbag` itself isn't installed or is installed incorrectly.
  fix Ensure `bdbag` is correctly installed (`pip install bdbag`). The correct import statement for the `BDBag` class is `from bdbag import BDBag`. If importing high-level functions, use `from bdbag import bdbag_api`.

Warnings

• gotcha bdbag strictly enforces Python version compatibility. It requires Python versions 3.8 through 3.11. Using it with unsupported versions (e.g., Python 3.7 or Python 3.12+) can lead to `ImportError`, `ModuleNotFoundError`, or unexpected runtime errors due to dependency conflicts or syntax incompatibilities.
  Fix: Ensure your Python environment is within the supported range (3.8-3.11). Use virtual environments (e.g., `venv` or `conda`) to manage specific Python versions for your projects.
• gotcha When creating a bag, if you provide a `data_directory` argument to `bdbag_api.make_bag`, bdbag expects this directory to contain the actual data files you want to include. It will then manage moving/linking these files into the bag's internal `data/` directory. If `data_directory` is omitted, `bdbag_api.make_bag` creates an empty bag, and you'll need to manually add files using `bag.add_file()` or similar methods.
  Fix: For creating bags with existing data, always specify the `data_directory` argument pointing to the source of your payload data. For empty bags, omit `data_directory` and use bag manipulation methods.
• gotcha By default, `bdbag` uses SHA256 checksums for payload files if `checksum_algorithms` is not specified during bag creation. If you need specific checksum algorithms (e.g., MD5, SHA1), always explicitly pass them as a list to the `checksum_algorithms` parameter in `bdbag_api.make_bag`.
  Fix: Always specify `checksum_algorithms=['md5', 'sha256']` (or your preferred list) when calling `bdbag_api.make_bag` if the default SHA256 is not sufficient or you need multiple algorithms.

Install

• pip install bdbag Install bdbag

Imports

• BDBag

  from bdbag import BDBag

• bdbag_api
  wrong:

  import bdbag.api

  correct:

  from bdbag import bdbag_api

  The high-level API functions are exposed via `bdbag_api` directly from the top-level package, not `bdbag.api`.

Quickstart

This quickstart demonstrates how to create a simple bdbag, add a data file to it, and then validate its integrity. It sets up a temporary directory, writes a small file, and uses `bdbag_api.make_bag` to create the bag and `bdbag_api.validate_bag` to check its validity.

import os
import shutil
from bdbag import bdbag_api

# Define paths for the bag
bag_dir = "my_test_bag"
data_dir = os.path.join(bag_dir, "data")
test_file_path = os.path.join(data_dir, "example.txt")

# Clean up previous run if directory exists
if os.path.exists(bag_dir):
    shutil.rmtree(bag_dir)

# 1. Create data directory for the bag payload
os.makedirs(data_dir, exist_ok=True)

# 2. Create some data to put into the bag
with open(test_file_path, "w") as f:
    f.write("This is some example data for the bdbag.\n")
    f.write("It will be bagged and validated.\n")

print(f"Created test data at: {test_file_path}")

try:
    # 3. Create the bag
    # The data_directory argument tells bdbag where to find the payload files
    # and move/link them into the bag's 'data' directory.
    bag = bdbag_api.make_bag(bag_dir,
                             checksum_algorithms=['sha256'],
                             data_directory=data_dir)
    print(f"Bag created successfully at: {bag.path}")

    # 4. Validate the bag
    is_valid = bdbag_api.validate_bag(bag_dir)
    if is_valid:
        print(f"Bag '{bag_dir}' is valid.")
    else:
        print(f"Bag '{bag_dir}' is NOT valid. Check logs for details.")

except Exception as e:
    print(f"An error occurred during bag creation or validation: {e}")
finally:
    # Optional: Clean up the created bag directory
    # Uncomment the line below to remove the directory after inspection
    # if os.path.exists(bag_dir):
    #     shutil.rmtree(bag_dir)
    pass