Why your cron job monitoring needs recovery alerts

Most cron job monitoring is one-directional: something breaks, an alert fires. You get notified about failures. You don't get notified about recoveries.

This gap creates a specific operational problem: once an alert fires, you have no reliable signal that the issue is resolved. You're left either watching the dashboard until the job runs successfully, or sending a manual "all clear" message to your team, or — most commonly — letting the incident linger in acknowledged-but-not-resolved limbo because following up requires remembering to check.

Recovery alerts solve this. When a monitor that was in a failing state receives a healthy ping, a recovery notification fires automatically to the same channels as the original alert. You know when incidents start. You know when they end. The incident is closed without anyone having to remember to close it.

What happens without recovery alerts

Alert fatigue from repeated failures. Without recovery alerts, the only confirmation you get is continued silence — no more failure alerts means things are probably working. But silence is a weak signal. Was the issue fixed? Did the job just not run yet? Did someone pause the monitor? You don't know until you check.

Manual incident tracking. Teams without recovery alerts typically manage incident lifecycle in Slack — "issue fixed, all good now" messages posted manually by whoever deployed the fix. This is unreliable, depends on the person who fixed it remembering to post, and produces no auditable record of resolution time.

Reopened incidents. A job that was failing, appeared to recover (one successful run), and then started failing again looks like a new incident — but without recovery alerts, you never confirmed the recovery happened, so you can't distinguish "new failure" from "continuous failure."

Unknown mean time to recovery (MTTR). If you want to measure how long it takes to resolve incidents — a useful reliability metric — you need to know when incidents ended, not just when they started. Recovery timestamps enable this. Absence of recovery alerts makes MTTR calculation impossible from monitoring data alone.

How recovery alerts work

Recovery alert logic is straightforward: a monitor tracks its state (healthy, failing, or degraded). When the state transitions from failing to healthy — because a healthy ping arrives after a period of failure — the monitoring service sends a notification to the same channels as the original failure alert.

The channels should match exactly: if the failure alert went to Slack and email, the recovery alert should go to the same Slack channel and the same email. This keeps the incident thread coherent — the Slack thread that started with "❌ nightly-sync failed" closes with "✅ nightly-sync recovered."

In Crontify, recovery alerts fire automatically when a monitor returns to healthy after any failure state — missed run, hung job, failed job, or silent failure alert. No additional configuration is required. The recovery notification is sent to whatever alert channels the monitor has configured.

What counts as a recovery

Recovery requires a clean success ping — not just the absence of further failure. This distinction matters:

Not a recovery: the job hasn't run since the failure. Silence after a failure is not a healthy state — it may mean the job is still failing, or was paused, or the next scheduled time hasn't arrived.

A recovery: the job ran after the failure and completed successfully. A start ping followed by a success ping within the expected duration, with no alert rule violations, transitions the monitor from failing to healthy.

For silent failure monitors — where the alert fired because records_processed eq 0 — recovery requires a success ping where records_processed is greater than zero. A run that exits 0 but processes nothing again doesn't constitute a recovery. The alert rule that fired the original alert is the same condition that must be resolved.

Configuring recovery alerts properly

Use the same channels as failure alerts. If your failure alerts go to a #incidents Slack channel, recovery alerts should go to the same channel. Splitting them across channels creates the same tracking problem recovery alerts are designed to solve.

Don't suppress recovery alerts for noisy monitors. The temptation is to disable recovery alerts for jobs that fail frequently — to reduce notification volume. This trades alert fatigue for incident visibility: you still have the problem, you just stop being told when it's fixed. The correct fix for a noisy monitor is to tune the alert threshold, not disable recovery notifications.

Consider cooldown periods for high-frequency jobs. For a job that runs every 5 minutes, a brief failure followed by an immediate recovery can generate a failure-recovery pair every few minutes if the underlying issue is intermittent. Set a minimum failure threshold (e.g., fail 3 consecutive times before alerting) and ensure the recovery condition requires a matching number of consecutive successes. This prevents alert fatigue from transient issues while still catching real failures.

Using recovery data to measure reliability

Recovery timestamps enable reliability metrics that failure alerts alone can't provide.

Mean time to recovery (MTTR): the average time between a failure alert and the corresponding recovery. Calculated as recovery_time - failure_time averaged across incidents.

Incident frequency: how often a specific monitor transitions from healthy to failing. A monitor that has 12 incidents in a month needs attention that one with 1 incident doesn't.

Recovery rate: what percentage of failures resolve within a given time window. A monitor that typically recovers within 30 minutes but occasionally has incidents lasting hours has a different reliability profile than one with consistent long incidents.

Crontify's run history records every state transition with timestamps. You can see exactly when each incident started and when it resolved, across the full history window for your plan.

The incident lifecycle with recovery alerts

Without recovery alerts, the incident lifecycle looks like this:

1. Failure alert fires
2. Team investigates and deploys a fix
3. Team hopes the job will run successfully
4. Someone manually checks the dashboard
5. Someone manually announces the fix in Slack

With recovery alerts:

1. Failure alert fires to #incidents
2. Team investigates and deploys a fix
3. Recovery alert fires to #incidents automatically
4. Incident thread closes with a concrete timestamp

The difference is that step 5 in the first flow — the manual announcement — is often skipped or delayed. In the second flow, it's automatic and immediate.

Setting up monitors with recovery alerts

In Crontify, every monitor has recovery alerts enabled by default. When you configure Slack, email, Discord, or webhook alert channels, recovery notifications route to all of them automatically.

The only configuration decision is whether to set a consecutive-failure threshold before the initial alert fires. For high-frequency jobs (every minute or every few minutes), requiring 2–3 consecutive failures before alerting — and 2–3 consecutive successes before marking recovered — reduces noise from transient issues without hiding real incidents.

For lower-frequency jobs (hourly, daily), a threshold of 1 is usually correct: any failure is worth an immediate alert, and any successful run constitutes a recovery.

Crontify is free for up to 5 monitors — no credit card required. Recovery alerts are available on every plan, including free.