Celery

Warnings

• breaking All old uppercase CELERY_* config keys (CELERY_BROKER_URL, CELERY_RESULT_BACKEND, CELERY_TASK_SERIALIZER, etc.) were deprecated in Celery 4.0 and removed in Celery 5.0. Using them silently does nothing or raises errors.
  Fix: Replace all CELERY_* uppercase keys with lowercase equivalents: CELERY_BROKER_URL → broker_url, CELERY_RESULT_BACKEND → result_backend, CELERY_TASK_SERIALIZER → task_serializer. See the Celery 4.0 migration guide.
• breaking Python 3.8 support dropped in Celery 5.6.0. Minimum is now Python 3.9.
  Fix: Pin celery<5.6 for Python 3.8 environments.
• breaking SQS transport: pycurl was replaced by urllib3 in Celery 5.5, then reverted back to pycurl in 5.6 due to critical issues. Users who uninstalled pycurl after upgrading to 5.5 must reinstall it before upgrading to 5.6.
  Fix: pip install pycurl before upgrading to celery 5.6 if you use the SQS transport.
• breaking Security: broker URLs containing passwords were logged in plaintext by the delayed delivery mechanism before 5.6. Credentials visible in log files.
  Fix: Upgrade to celery>=5.6.0. Audit existing logs for exposed credentials.
• gotcha Celery requires a broker — there is no built-in broker. Without a running Redis or RabbitMQ instance, all .delay() and .apply_async() calls raise kombu.exceptions.OperationalError.
  Fix: Start Redis locally: docker run -d -p 6379:6379 redis. Then set broker='redis://localhost:6379/0' in your Celery app.
• gotcha The result backend is separate from the broker. Without a configured result_backend, result.get() blocks forever or raises NotImplementedError. Many tutorials configure the broker but forget the backend.
  Fix: Set backend= in Celery() constructor or result_backend in app.conf. For Redis: backend='redis://localhost:6379/0'.
• gotcha Task serializer defaults to json in Celery 5.x. Using pickle=True or passing non-JSON-serializable objects to tasks raises kombu.exceptions.EncodeError. Complex Python objects (datetime, custom classes) must be serialized manually.
  Fix: Convert task arguments to JSON-serializable types before calling .delay(). For datetime: pass .isoformat() string, parse inside the task.
• gotcha In notebooks and scripts using multiprocessing, Accelerator() or notebook_launcher patterns: the Celery app must be importable as a module — it cannot be defined inline in a __main__ block. Workers import the app from the module path passed to -A.
  Fix: Define the Celery app in a separate module file (e.g. tasks.py). Start worker with: celery -A tasks worker

Install

• pip install celery Core (no broker included)
• pip install celery[redis] With Redis broker/backend support
• pip install celery[rabbitmq] With RabbitMQ (librabbitmq) support
• pip install celery[sqs] With AWS SQS transport

Imports

• Celery

  from celery import Celery
  
  app = Celery(
      'myapp',
      broker='redis://localhost:6379/0',
      backend='redis://localhost:6379/0'
  )
  
  # Correct: use @app.task decorator
  @app.task
  def add(x, y):
      return x + y
  
  # Call async
  result = add.delay(4, 6)
  print(result.get(timeout=10))

  Old uppercase CELERY_* config keys were deprecated in 4.0 and fully removed in 5.0. Use lowercase keys: task_serializer, result_backend, broker_url.

Quickstart

Requires Redis running. Start worker in separate terminal: celery -A tasks worker --loglevel=info

# tasks.py
from celery import Celery

app = Celery(
    'tasks',
    broker='redis://localhost:6379/0',
    backend='redis://localhost:6379/0'
)

app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='UTC',
)

@app.task
def add(x, y):
    return x + y

@app.task(bind=True, max_retries=3)
def fetch_data(self, url):
    try:
        import requests
        return requests.get(url).json()
    except Exception as exc:
        raise self.retry(exc=exc, countdown=5)

# --- Run worker ---
# celery -A tasks worker --loglevel=info

# --- Call from client ---
# result = add.delay(4, 6)
# print(result.get(timeout=10))  # 10