dvc-task

0.40.2 · active · verified Sat Apr 11

dvc-task is an extensible Python library for queuing, running, and managing background jobs (processes) from standalone applications. It is built on Celery but leverages a pre-configured filesystem transport, eliminating the need for a full AMQP messaging server. As of April 2026, the current version is 0.40.2, with active development and somewhat frequent, though irregular, releases.

Warnings

breaking Version 0.3.0 introduced a breaking change by bumping the minimum required versions for Celery and Kombu to 5.3.0. Users on older versions of these dependencies will need to upgrade.
Fix: Upgrade Celery and Kombu to at least version 5.3.0: `pip install 'celery>=5.3.0' 'kombu>=5.3.0'`.
gotcha Users observed 'Celery Monitor Shutdowns' prior to version 0.40.1, indicating potential instability or unexpected behavior in monitoring. Version 0.40.1 includes a fix (dvc-10427) for this issue.
Fix: Upgrade to dvc-task version 0.40.1 or later to resolve Celery Monitor Shutdown issues.
gotcha While `dvc-task` is tested on Windows, its upstream dependency Celery is not officially supported on Windows. This means that certain issues specific to Windows might stem from Celery itself and could have limited official support.
Fix: Be aware of potential platform-specific issues on Windows. Refer to Celery's documentation for known Windows limitations and workarounds.
gotcha The version numbering jumped from `0.4.0` to `0.40.1`. While not explicitly stated as breaking, such a large jump in the minor version often signals significant changes or potential incompatibilities, even if release notes are brief. Users should review the changelog carefully when upgrading across this boundary.
Fix: Carefully review the changelog between `0.4.0` and `0.40.1` for unlisted breaking changes or behavioral differences, and thoroughly test your application after upgrading.

Install

pip install dvc-task Install dvc-task

Imports

FSApp
```
from dvc_task.app import FSApp
```
Main application class for a filesystem-based Celery app.
ProcessManager
```
from dvc_task.proc import ProcessManager
```
Manages background processes within tasks.

Quickstart

This quickstart demonstrates how to initialize a filesystem-based Celery application (FSApp), define a task to run a background process using ProcessManager, and then start a temporary worker to process the task. It showcases how to queue a simple 'echo' command.

from dvc_task.app import FSApp
from dvc_task.proc import ProcessManager
import time
import os

# Create a filesystem-based Celery app
# For a real application, consider a more persistent wdir
app = FSApp(wdir="./dvc-task-data")

@app.task
def run_command_task(command_args, task_name):
    manager = ProcessManager(wdir=os.path.join(app.wdir, 'processes'))
    # Run a simple command as a background process
    # The '.delay()' call queues the task
    signature = manager.run(command_args, name=task_name).delay()
    return signature.id

# Example usage:
if __name__ == "__main__":
    print("Starting dvc-task example...")
    task_id = run_command_task.delay(command_args=["echo", "Hello, dvc-task!"], task_name="greet_task")
    print(f"Task 'greet_task' queued with ID: {task_id}")

    print("Starting temporary worker...")
    # In a real scenario, the worker would run in a separate process/daemon
    # For quickstart, we run a temporary worker that exits when the queue is empty
    worker = app.TemporaryWorker()
    worker.start() # This blocks until the queue is empty or timeout

    # Clean up (optional, but good for quickstart)
    app.clean()
    print("dvc-task example finished and cleaned up.")

view raw JSON →