How to monitor Celery Beat tasks in production

Celery Beat is the standard periodic task scheduler for Python applications using Celery. It runs as a separate process alongside your Celery workers, checks the schedule every second, and enqueues tasks into the broker when they're due.

The architecture creates a monitoring gap that's easy to miss. Celery workers and Celery Beat are independent processes. A worker failure is visible — tasks pile up in the queue, consumers are unavailable, errors surface. A Beat failure is invisible — tasks simply stop being enqueued. The queue stays empty. The workers stay idle. No error fires. No alert sounds.

If your Beat process crashes, is killed during a deployment, or loses its connection to the broker, every periodic task in your application silently stops running. Discovery typically happens when a downstream effect surfaces: missing data, stale reports, unsent notifications.

What Celery Beat doesn't monitor

Beat process failure. Celery Beat runs as a standalone process. If it crashes or is stopped, nothing re-enqueues the tasks. Celery's retry mechanism is at the worker level — Beat itself has no watchdog. You won't know Beat is down until a task that was supposed to run hasn't.

Task execution failure vs scheduling failure. Celery has built-in retry logic and dead-letter queues for task execution failures — tasks that were enqueued and then failed during execution. It has nothing equivalent for scheduling failures — tasks that should have been enqueued but weren't because Beat was down.

Silent task failure. A Celery task that completes without raising an exception is considered successful, regardless of what it accomplished. A sync that processed zero records, a report that generated zero rows, or a notification that dispatched to zero recipients all complete with SUCCESS status in Celery's result backend.

Monitoring pattern: HTTP pings from task functions

The most straightforward monitoring pattern wraps each periodic task function with start, success, and fail pings to an external monitoring service. The external service tracks whether pings arrive on schedule and fires an alert when they don't.

Step 1: Create a monitoring helper

# apps/monitoring.py
import os
import requests
import traceback
from functools import wraps
from typing import Callable, Any

API_KEY = os.environ.get("CRONTIFY_API_KEY", "")
BASE_URL = "https://api.crontify.com/api/v1/ping"
HEADERS = {"X-API-Key": API_KEY}


def ping(monitor_id: str, event: str, payload: dict | None = None) -> None:
    """Send a monitoring ping. Never raises — must not affect task execution."""
    try:
        requests.post(
            f"{BASE_URL}/{monitor_id}/{event}",
            headers=HEADERS,
            json=payload,
            timeout=5,
        )
    except Exception:
        pass


def monitored_task(monitor_id: str) -> Callable:
    """
    Decorator for Celery task functions. Sends start/success/fail pings.
    The decorated function can return a dict to attach as metadata
    on the success ping — used for silent failure detection alert rules.
    """
    def decorator(fn: Callable) -> Callable:
        @wraps(fn)
        def wrapper(*args: Any, **kwargs: Any) -> Any:
            ping(monitor_id, "start")
            try:
                result = fn(*args, **kwargs)
                meta = result if isinstance(result, dict) else {}
                ping(monitor_id, "success",
                     {"meta": meta} if meta else None)
                return result
            except Exception as exc:
                ping(monitor_id, "fail", {
                    "message": str(exc),
                    "log": traceback.format_exc(),
                })
                raise
        return wrapper
    return decorator

Step 2: Apply it to your periodic tasks

# apps/tasks.py
from celery import shared_task
from .monitoring import monitored_task


@shared_task
@monitored_task("mon_abc123")
def nightly_sync() -> dict:
    result = sync_records_from_api()
    return {
        "records_synced": result.count,
        "api_calls_made": result.api_calls,
        "duration_ms": result.duration_ms,
    }


@shared_task
@monitored_task("mon_def456")
def generate_daily_report() -> dict:
    report = build_report()
    return {
        "rows_in_report": report.row_count,
        "recipients_emailed": report.recipients,
    }

Step 3: Define the Beat schedule as normal

# celery.py or settings/celery.py
from celery.schedules import crontab

app.conf.beat_schedule = {
    "nightly-sync": {
        "task": "apps.tasks.nightly_sync",
        "schedule": crontab(hour=2, minute=0),
    },
    "daily-report": {
        "task": "apps.tasks.generate_daily_report",
        "schedule": crontab(hour=6, minute=0),
    },
}

Monitoring Beat itself — not just the tasks

Individual task pings tell you when a specific task ran or failed. They don't tell you when Beat stops entirely — because if Beat is down, no tasks are enqueued and no pings are sent.

To detect a stopped Beat process, add a dedicated heartbeat task that runs every minute:

@shared_task
def beat_heartbeat() -> None:
    """Runs every minute. Absence of this ping means Beat is down."""
    ping("mon_beat_heartbeat", "success")


# Add to beat_schedule
app.conf.beat_schedule["beat-heartbeat"] = {
    "task": "apps.tasks.beat_heartbeat",
    "schedule": 60.0,  # every minute
}

Configure the corresponding monitor in Crontify with:

• Expected schedule: * * * * * (every minute)
• Grace period: 2 minutes

If Beat goes down, no heartbeat arrives, and you get an alert within 2 minutes. This catches Beat process crashes, broker disconnections, and deployment restarts that leave Beat stopped.

Silent failure detection for Celery Beat tasks

The dict returned from your task function is sent as metadata on the success ping. In Crontify's dashboard, define alert rules against those values:

api_calls_made eq 0 → alert (upstream API was never reached)
recipients_emailed eq 0 → alert (notifications dispatched to no one)
rows_in_report eq 0 → alert (report generated no data)

A Celery task that completes with SUCCESS status in Celery's result backend but returns zero in any of these fields will trigger an external alert. This is the class of failure Celery's own monitoring — Flower, the result backend, Celery's error handlers — was never designed to catch.

django-celery-beat compatibility

If you're using django-celery-beat for database-backed schedule management, the same pattern applies. The @monitored_task decorator wraps the task function regardless of where the schedule is stored. Dynamic schedule changes (adding or removing tasks from the database) don't affect the monitoring setup — each task has a fixed monitor ID and the external monitor holds the expected schedule independently.

Frequently asked questions

What if the task is enqueued but the worker is down?

The task sits in the broker queue. When a worker comes back up, Celery picks up and executes the queued tasks. The start ping fires when the task actually runs, not when it's enqueued. This means there's a lag between the scheduled time and the start ping — configure your monitor's grace period to be longer than your typical worker restart time.

Can I monitor tasks that use Celery's eta or countdown for delayed execution?

Yes, but the external monitor needs to be configured with the expected execution time, not the time the task was enqueued. For tasks with variable execution times, consider using a broader grace period or monitoring via result inspection rather than schedule-based pings.

What's the right grace period for a Beat task?

The grace period should account for: broker latency (usually milliseconds), worker availability (time to pick up from the queue), and the task's setup time before it does any actual work. For most tasks, 5–10 minutes is appropriate. For tasks immediately after a deployment, increase the grace period to account for deployment time.

Crontify is free for up to 5 monitors — no credit card required.