Snakemake Storage Plugin Interface

Common errors

• ModuleNotFoundError: No module named 'humanfriendly'

  cause The `humanfriendly` package, used internally by the storage plugin interface, was not installed as a direct dependency in versions prior to v4.3.3.
  fix Install the missing dependency: `pip install humanfriendly`. Alternatively, upgrade to `snakemake-interface-storage-plugins>=4.3.3` which includes `humanfriendly` as a direct dependency.

• TypeError: Can't instantiate abstract class MyCustomStorageObject with abstract methods exists, mtime, size, download_obj, upload_obj, list_local_files, cleanup

  cause Your custom `StorageObject` class inherits from `snakemake_interface_storage_plugins.storage_object.StorageObject` but has not implemented all of its required abstract methods.
  fix Ensure that your `MyCustomStorageObject` class (and any custom `IOCacheInterface` if used) implements all abstract methods defined in the `StorageObject` base class. You must provide concrete implementations for `exists`, `mtime`, `size`, `download_obj`, `upload_obj`, `list_local_files`, and `cleanup`.

• TypeError: __init__() missing 1 required positional argument: 'checksum'

  cause You are trying to instantiate `StorageObject` or `IOCacheInterface` (or a custom class inheriting from them) without providing the `checksum` argument, which became required from v4.4.0 onwards for full functionality.
  fix Update your code to provide a `checksum` argument when initializing `StorageObject` or `IOCacheInterface` instances, or ensure your custom plugin adheres to the updated interface. For example: `MyStorageObject(query=None, protocol='my', path=url, checksum=None)` (if checksum is optional for your specific case, otherwise provide a valid checksum value).

Warnings

• breaking The `checksum` attribute was added to `IOCacheInterface` and `StorageObject` in v4.4.0. Existing custom storage plugins might need to be updated to properly handle checksum generation and validation to maintain full compatibility and leverage new Snakemake features.
  Fix: Review `IOCacheInterface` and `StorageObject` definitions. Implement checksum handling methods/attributes in your custom `StorageObject` and `IOCacheInterface` (if used) to avoid potential future compatibility issues or `TypeError`s when Snakemake expects this functionality.
• gotcha Older versions of `snakemake-interface-storage-plugins` (prior to v4.3.3) did not explicitly list `humanfriendly` as a direct dependency. If `humanfriendly` is not installed by another package, you might encounter a `ModuleNotFoundError`.
  Fix: Ensure `humanfriendly` is installed in your environment: `pip install humanfriendly`. Upgrade to version 4.3.3 or newer to have it automatically included as a dependency.
• breaking Changes in error handling, specifically around `FileOrDirectoryNotFoundError` and its conversion to `WorkflowError`, were introduced in versions 4.3.0 and 4.3.1. Custom plugins raising specific exceptions might need adjustment.
  Fix: Review how your custom plugin handles file/directory not found scenarios and other I/O errors. Ensure that custom exceptions are handled or converted appropriately to align with the interface's expected error types, especially `WorkflowError` for user-facing issues.
• gotcha Python 3.10 compatibility fixes were applied in v4.4.1. If you are using Python 3.10 with versions prior to 4.4.1, you might encounter unexpected runtime errors or incorrect behavior.
  Fix: Upgrade `snakemake-interface-storage-plugins` to v4.4.1 or higher when using Python 3.10 to ensure full compatibility: `pip install --upgrade snakemake-interface-storage-plugins`.

Install

• pip install snakemake-interface-storage-plugins Install latest version

Imports

• StorageProviderBase

  from snakemake_interface_storage_plugins.storage_provider import StorageProviderBase

• StorageObject

  from snakemake_interface_storage_plugins.storage_object import StorageObject

• StorageProviderSettingsBase

  from snakemake_interface_storage_plugins.settings import StorageProviderSettingsBase

• StoragePlugin
  wrong:

  from snakemake_interface_storage_plugins.plugin import StoragePlugin

  correct:

  from snakemake_interface_storage_plugins.registry import StoragePlugin

  The StoragePlugin class for registering plugins was moved to the 'registry' submodule.

Quickstart

This quickstart demonstrates the core components required to implement a minimal Snakemake storage plugin. You need to define a `StorageObject` (representing a file/directory on your custom storage) and a `StorageProviderBase` (which provides `StorageObject` instances). Finally, register your plugin using `StoragePlugin` to make it discoverable by Snakemake.

from snakemake_interface_storage_plugins.storage_object import StorageObject
from snakemake_interface_storage_plugins.storage_provider import StorageProviderBase
from snakemake_interface_storage_plugins.registry import StoragePlugin
from typing import Optional

# 1. Define your custom StorageObject, implementing required methods
class MyMinimalStorageObject(StorageObject):
    def exists(self) -> bool:
        # Placeholder: In a real plugin, check if the remote object exists.
        return True

    def mtime(self) -> float: return 0.0 # Last modified time
    def size(self) -> int: return 0      # Size in bytes
    def download_obj(self) -> None: pass # Download object to local path
    def upload_obj(self) -> None: pass   # Upload local object to remote
    def list_local_files(self) -> list[str]: return [] # List files in local path
    def cleanup(self) -> None: pass      # Cleanup temporary files

# 2. Define your custom StorageProvider
class MyMinimalStorageProvider(StorageProviderBase):
    def get_storage_object(self, url: str) -> StorageObject:
        # Return an instance of your custom StorageObject for the given URL.
        # 'query' and 'protocol' are typically parsed from the URL.
        return MyMinimalStorageObject(query=None, protocol=self.name, path=url)

    def example_path(self, protocol: Optional[str] = None) -> str:
        # Provide an example path for your protocol (e.g., for documentation).
        return f"{self.name}://path/to/file.txt"

    @property
    def name(self) -> str:
        # The unique name for your storage protocol (e.g., 's3', 'gcs', 'my-minimal').
        return "my-minimal"

# 3. Register your plugin with Snakemake
# This makes your plugin discoverable by Snakemake when it parses Snakefiles.
StoragePlugin(
    name="my-minimal",
    storage_provider=MyMinimalStorageProvider,
    # settings_cls=None # Optionally register a custom settings class
)

print("Minimal Snakemake storage plugin 'my-minimal' defined and registered.")
# To make this plugin active for Snakemake, save this code in a Python file
# (e.g., my_storage_plugin.py) and ensure it's on your PYTHONPATH or installed
# as part of a Snakemake extension package.