Cron job monitoring for data pipelines and ETL jobs

Data pipeline jobs have a failure profile that's different from other scheduled tasks. A web server that crashes is immediately visible — requests start failing, error rates spike, users complain. A data pipeline that fails silently keeps serving stale data, producing incomplete reports, and making incorrect calculations — sometimes for weeks — before the downstream effect surfaces.

Standard cron monitoring — heartbeat pings, exit code checks — catches the obvious failures. It misses the ones that matter most to data teams: partial loads, zero-row syncs, dropped records, and freshness violations.

This guide covers how to monitor ETL and data pipeline jobs in a way that actually catches these failures, using output metadata and alert rules to define what "success" means in data terms, not just in process terms.

The data pipeline failure modes standard monitoring misses

Zero-row extract. The source database, API, or file system returned no data. Your pipeline processed nothing, wrote nothing, and exited 0. The target table or warehouse is now stale. Heartbeat monitoring received its ping on time.

Partial load. The extract phase fetched 50,000 records. The transform phase produced 50,000 records. The load phase inserted 12,000 before hitting a database timeout or hitting a unique constraint. The remaining 38,000 were silently dropped. The job exited 0 because the partial load was considered acceptable by the code.

Schema drift. An upstream API changed a field name, type, or structure. Your transform code handled the unexpected value by defaulting to null or skipping the field. Every record loaded successfully with a null value in a column that should contain data. No exception was raised.

Stale watermark. Your incremental pipeline uses a watermark (a timestamp or ID) to fetch only new records since the last run. The watermark was not updated correctly — either due to a bug or an exception that was swallowed — so subsequent runs process the same records repeatedly, or miss new records entirely.

Duplicate records. The load phase lacked proper upsert logic. Records that were already in the target were inserted again rather than updated. The row count is higher than expected, not lower — which looks like success if you're only watching for zero.

The monitoring model for data pipelines

Standard cron monitoring tells you whether the pipeline process ran. Data pipeline monitoring needs to tell you whether the pipeline accomplished its data contract.

The data contract for a typical ETL pipeline has three components:

1. Volume. A minimum number of records was extracted, transformed, and loaded. Zero is almost never acceptable. Below-threshold is often a signal of upstream issues.
2. Freshness. The data in the target reflects reality as of a recent enough timestamp. A pipeline that loads last week's data is a failure even if it loaded something.
3. Completeness. The ratio of records successfully loaded to records extracted is within acceptable bounds. A 10% drop rate may be acceptable; a 60% drop rate is not.

Standard monitoring tools don't model any of these. You have to provide the values — and have a monitoring layer that acts on them.

Attaching pipeline metrics to monitoring pings

When your pipeline calls success(), attach the metrics that define whether the run was actually successful:

import { CrontifyMonitor } from '@crontify/sdk';

const monitor = new CrontifyMonitor({
  apiKey: process.env.CRONTIFY_API_KEY!,
  monitorId: 'your-pipeline-monitor-id',
});

await monitor.wrap(async () => {
  const startTime = Date.now();

  // Extract
  const extracted = await extractFromSource();

  // Transform
  const transformed = await transformRecords(extracted);

  // Load
  const loaded = await loadToTarget(transformed);

  // Calculate metrics
  const dropRate = extracted.count > 0
    ? ((extracted.count - loaded.successCount) / extracted.count) * 100
    : 0;

  return {
    meta: {
      records_extracted:   extracted.count,
      records_transformed: transformed.count,
      records_loaded:      loaded.successCount,
      records_failed:      loaded.failCount,
      drop_rate_pct:       Math.round(dropRate),
      watermark_advanced:  loaded.newWatermark > loaded.previousWatermark ? 1 : 0,
      duration_ms:         Date.now() - startTime,
    }
  };
});

In Crontify's dashboard, define rules against these values:

records_loaded eq 0 → alert (nothing reached the target)
drop_rate_pct gt 10 → alert (more than 10% of records failed to load)
watermark_advanced eq 0 → alert (watermark didn't advance — possible stale run)

Python ETL pipeline example

For Python-based pipelines using pandas, SQLAlchemy, or similar:

import os
import time
import requests
import traceback

MONITOR_ID = os.environ["CRONTIFY_MONITOR_ID"]
API_KEY = os.environ["CRONTIFY_API_KEY"]
BASE_URL = f"https://api.crontify.com/api/v1/ping/{MONITOR_ID}"
HEADERS = {"X-API-Key": API_KEY}


def ping(event: str, payload: dict | None = None) -> None:
    try:
        requests.post(f"{BASE_URL}/{event}", headers=HEADERS,
                      json=payload, timeout=10)
    except Exception:
        pass


def run_pipeline() -> dict:
    start = time.time()

    # Extract
    df_raw = extract_from_source()
    extracted_count = len(df_raw)

    # Transform
    df_transformed = transform(df_raw)

    # Load
    loaded_count, failed_count = load_to_warehouse(df_transformed)

    return {
        "records_extracted":   extracted_count,
        "records_loaded":      loaded_count,
        "records_failed":      failed_count,
        "drop_rate_pct":       round((failed_count / extracted_count * 100)
                                     if extracted_count else 0),
        "duration_ms":         int((time.time() - start) * 1000),
    }


ping("start")
try:
    metrics = run_pipeline()
    ping("success", {"meta": metrics})
except Exception as exc:
    ping("fail", {
        "message": str(exc),
        "log": traceback.format_exc()[:5000],
    })
    raise

Incremental pipeline freshness tracking

For incremental pipelines where the key question is "did we process new data?", track the watermark explicitly:

const previousWatermark = await db.getWatermark('orders-sync');

const newRecords = await fetchOrdersSince(previousWatermark);

if (newRecords.length === 0) {
  // Legitimate empty run or upstream issue?
  // Attach context so alert rules can distinguish
  return {
    meta: {
      records_processed: 0,
      watermark_value:   previousWatermark.toISOString(),
      watermark_age_hours: Math.round(
        (Date.now() - previousWatermark.getTime()) / 3600000
      ),
    }
  };
}

await loadRecords(newRecords);
await db.setWatermark('orders-sync', new Date());

return {
  meta: {
    records_processed:  newRecords.length,
    watermark_value:    new Date().toISOString(),
    watermark_age_hours: 0,
  }
};

Alert rule: watermark_age_hours gt 25 → fire alert (watermark is more than 25 hours old, meaning no data has been loaded in over a day).

This catches the case where the pipeline runs successfully every night but has been loading zero records because the watermark is stuck — a silent failure that would otherwise be invisible for weeks.

When to alert and when not to

Not every zero-row run is a failure. Some pipelines legitimately have nothing to process on certain runs. The goal is to distinguish expected emptiness from unexpected emptiness.

Legitimate zero-row runs: a pipeline that processes only new records will have empty runs when no new data exists. Alert on records_extracted eq 0 only if your source should always have data. If empty runs are expected, use watermark_age_hours gt N instead.

Unexpected zero-row runs: a full-refresh pipeline that replaces all records in a target table should never have a zero-row run. records_loaded eq 0 is always alertable here.

Drop rate thresholds: set your drop rate threshold based on observed historical rates. If your pipeline normally loads 99.8% of extracted records, a drop_rate_pct gt 5 rule gives you a wide safety margin. If it normally loads 100%, use records_failed gt 0.

Crontify is free for up to 5 monitors — no credit card required.

Alert rules on job output metadata are available on every plan, including free. For data pipeline jobs where exit code 0 is not a sufficient indicator of success, this is the monitoring layer that standard tools don't provide.