Snakemake Storage Plugin FS

1.1.3 verified Mon Apr 27 auth: no python

A Snakemake storage plugin for reading and writing from a locally mounted filesystem using rsync. Version 1.1.3 is the latest stable release, with active development and monthly releases focused on bug fixes and compatibility with Snakemake's storage plugin API.

pip install snakemake-storage-plugin-fs

Common errors

error snakemake: error: unrecognized arguments: --storage-prefix ↓

cause User passed --storage-prefix directly to Snakemake command line, but the plugin expects the prefix to be set via config file or environment variable.

fix

Set the prefix in the Snakemake configuration file (e.g., config.yaml) using the key 'storage-fs-prefix' or via the environment variable 'SNAKEMAKE_STORAGE_FS_PREFIX'.

error ModuleNotFoundError: No module named 'snakemake_storage_plugin_fs' ↓

cause The plugin is not installed, or the environment is misconfigured.

fix

Run 'pip install snakemake-storage-plugin-fs' in the correct Python environment (e.g., conda environment or virtualenv).

error rsync: command not found ↓

cause rsync is not installed on the system.

fix

Install rsync via your package manager (e.g., 'apt-get install rsync' or 'brew install rsync').

error Permission denied (publickey) ↓

cause User attempted to use the plugin with a remote SSH path, but the plugin does not support SSH authentication.

fix

Mount the remote filesystem locally (e.g., via sshfs) and use the local mount point as the remote_path.

Warnings

gotcha The plugin relies on rsync being installed on the system. Without it, file transfers will fail silently or with cryptic errors. ↓

fix Install rsync (e.g., apt-get install rsync on Debian/Ubuntu, or brew install rsync on macOS).

deprecated As of v1.0.3, the query validation API was deprecated. Old code that used `validate_query` directly will fail. Users should rely on Snakemake's standard query handling. ↓

fix Upgrade to v1.0.3 or later and remove any manual calls to deprecated validation methods.

gotcha The plugin does not support remote hosts via SSH or any network protocol; it only works with locally mounted filesystems. Users may mistakenly try to use it with remote paths via SSH, which will fail because rsync is called without SSH flags by default. ↓

fix Ensure the 'remote' path is actually a locally mounted directory (e.g., NFS, FUSE, or local disk). For remote SSH, use a different plugin (e.g., snakemake-storage-plugin-s3, snakemake-storage-plugin-gs).

gotcha Symlinks are copied without preserving their targets by default in older versions. This can lead to broken symlinks in the cache. ↓

fix Upgrade to v1.1.3 or later, which explicitly passes `--links --no-perms --no-owner` to rsync to ensure symlinks are copied correctly.

Imports

StorageProviderFS
wrong
```
import snakemake_storage_plugin_fs
```
correct
```
from snakemake_storage_plugin_fs import StorageProviderFS
```
You need to import the class, not just the module. However, users typically never import this plugin manually; Snakemake discovers and uses it automatically.

Quickstart

Basic usage of the plugin to interact with a local filesystem storage. The plugin is typically used implicitly by Snakemake when a workflow references storage queries with 'fs://' prefix.

from snakemake_storage_plugin_fs import StorageProviderFS

# Example: initialize with remote path and optional prefix
provider = StorageProviderFS(
    remote_path="/path/to/remote",
    prefix="/local/prefix"
)

# Example operations (normally triggered by Snakemake)
print(provider.list())
print(provider.stat("some_file.txt"))