{"id":8005,"library":"celery-once","title":"Celery Once","description":"Celery Once is a Python library that prevents multiple execution and queuing of Celery tasks, ensuring a task runs only once. It's particularly useful for idempotent operations or tasks that should not be duplicated. The current version is 3.0.1, and it maintains an active release cadence with significant changes often introduced in major versions.","status":"active","version":"3.0.1","language":"en","source_language":"en","source_url":"https://github.com/cameronmaske/celery-once","tags":["celery","task management","concurrency","distributed tasks","once","idempotency"],"install":[{"cmd":"pip install celery-once","lang":"bash","label":"Basic installation"},{"cmd":"pip install celery-once[redis]","lang":"bash","label":"Install with Redis backend support"}],"dependencies":[{"reason":"Core dependency for defining and running tasks.","package":"celery","optional":false},{"reason":"Required for the Redis locking backend.","package":"redis","optional":true},{"reason":"Used by the Redis backend for distributed locking.","package":"python-redis-lock","optional":true},{"reason":"Required for the Msgpack locking backend.","package":"msgpack","optional":true}],"imports":[{"symbol":"QueueOnce","correct":"from celery_once import QueueOnce"}],"quickstart":{"code":"import os\nfrom celery import Celery\nfrom celery_once import QueueOnce\n\n# Configure Celery app\napp = Celery('my_app', broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')\n\n# Configure celery-once backend (e.g., Redis)\n# Set CELERY_ONCE in your Celery config, or directly in app.conf\napp.conf.update(\n    CELERY_ONCE = {\n        'backend': 'celery_once.backends.Redis', # or 'celery_once.backends.File'\n        'default_timeout': 60 * 60 # 1 hour default lock timeout\n    }\n)\n\n@app.task(base=QueueOnce)\ndef my_unique_task(arg1, arg2):\n    \"\"\"This task will only run once at a time globally.\"\"\"\n    print(f\"Executing my_unique_task with {arg1}, {arg2}\")\n    return f\"Task completed for {arg1}, {arg2}\"\n\nif __name__ == '__main__':\n    # Example usage (usually tasks are called by Celery workers)\n    # To run this, you'd typically start a Celery worker: celery -A your_module_name worker -l info\n    # And then call the task from another Python script or shell:\n    # from your_module_name import my_unique_task\n    # my_unique_task.delay('value1', 'value2')\n    # my_unique_task.delay('value1', 'value2') # This second call will be skipped if the first is still locked\n    \n    # For demonstration, manually calling (won't use Celery broker in this snippet directly)\n    print(\"To run tasks, ensure Redis is running and start a Celery worker.\\n\")\n    print(\"Example: my_unique_task.delay('hello', 123) will queue the task.\\n\")\n    print(\"If run twice quickly, the second will be skipped (depending on backend).\")\n","lang":"python","description":"This quickstart demonstrates how to integrate `celery-once` with a Celery task using the `QueueOnce` base class. It configures the Redis backend for `celery-once` and defines a task that will only execute once globally. Remember to start a Redis server and a Celery worker to see it in action."},"warnings":[{"fix":"If upgrading and using the file backend, any pending locks from prior versions will be ignored. No direct migration is possible; existing locks effectively become stale. Redis backend is unaffected.","message":"Version 3.0.0 changed the file backend's key generation by hashing and limiting key length. This makes it incompatible with existing locks from previous versions if you are using the file backend.","severity":"breaking","affected_versions":">=3.0.0"},{"fix":"It is highly recommended to remove any previous `celery-once` lock keys from Redis when upgrading from 1.x.x to 2.x.x to prevent unexpected behavior. Consult GitHub issues like #67 for more context.","message":"Version 2.0.0 fundamentally changed the Redis backend to use a SETNX-based lock (RedLock) to address race conditions. This change may not be backward compatible with existing keys stored in Redis from 1.x.x versions.","severity":"breaking","affected_versions":">=2.0.0"},{"fix":"Upgrade `celery-once` to version 2.1.1 or newer. If you must use an older version, ensure tasks using the `File` backend are short-lived or manage file descriptors carefully (though upgrading is strongly preferred).","message":"Prior to version 2.1.1, the `File` backend could lead to a 'Too many open files' error due to unclosed file descriptors, especially under heavy load or long-running processes.","severity":"gotcha","affected_versions":"<2.1.1"},{"fix":"Always explicitly define the `CELERY_ONCE` dictionary in your Celery application's configuration (`app.conf.update` or `celeryconfig.py`), specifying the `backend` and `default_timeout`.","message":"Tasks using `celery-once` might silently fail to prevent duplicate execution if the `CELERY_ONCE` configuration is missing or incorrect, defaulting to no `celery-once` backend.","severity":"gotcha","affected_versions":"All versions"}],"env_vars":null,"last_verified":"2026-04-16T00:00:00.000Z","next_check":"2026-07-15T00:00:00.000Z","problems":[{"fix":"Upgrade `celery-once` to version 3.0.0 or newer. This version introduces hashing and truncation for file backend keys, resolving the filename length issue.","cause":"When using the `File` backend with `celery-once` versions prior to 3.0.0, long or complex task arguments could generate lock filenames exceeding the operating system's filename length limits.","error":"OSError: [Errno 36] File name too long"},{"fix":"Upgrade `celery-once` to version 2.1.1 or newer. This version includes a fix to ensure file descriptors are correctly closed after use.","cause":"When using the `File` backend with `celery-once` versions prior to 2.1.1, file descriptors were not always properly closed, leading to resource exhaustion over time.","error":"Too many open files"},{"fix":"Upgrade `celery-once` to version 2.0.0 or newer. This version refactors the Redis backend to use a robust SETNX-based RedLock mechanism, which effectively mitigates race conditions. Remember to clear old Redis keys when upgrading.","cause":"Prior to `celery-once` version 2.0.0, the Redis backend implementation had known race conditions that could allow duplicate task execution under specific circumstances.","error":"Tasks are still running multiple times / Race condition when acquiring lock"},{"fix":"Ensure your task code handles exceptions gracefully. If a lock is genuinely stuck, you may need to manually clear the lock key from your chosen backend (e.g., delete the specific key from Redis or the file from the filesystem path configured for `celery-once`). Consider using a shorter `default_timeout` for locks if tasks are generally short-lived.","cause":"A task might acquire a lock but fail to release it (e.g., due to an unhandled exception before the lock release, or worker crash), leaving the task permanently locked until the timeout expires.","error":"Task always reports AlreadyQueued but never executes or completes."}]}