Monitoring Kubernetes CronJobs with HTTP pings

Kubernetes CronJob monitoring is harder than it looks. The standard approach — deploying kube-state-metrics, writing Prometheus queries, and configuring Alertmanager rules — works, but it has a fundamental weakness: everything lives inside the cluster.

If your monitoring infrastructure is inside the cluster and the cluster has problems, your monitoring may fail at exactly the moment you need it most. A node that runs out of resources, a namespace that gets misconfigured, or a control plane issue can prevent your CronJobs from running and simultaneously prevent your monitoring from alerting.

The cluster-independent alternative is push-based heartbeat monitoring: your CronJob containers send HTTP pings to an external service on each run. If the ping doesn't arrive, the external service alerts you — regardless of what's happening inside the cluster.

Why Kubernetes CronJobs fail silently

Kubernetes CronJobs have several failure modes that are easy to miss:

No Job created. The CronJob controller checks every 10 seconds. If startingDeadlineSeconds is set too low, the controller may skip a scheduled run because it missed the window. The CronJob status shows a LastScheduleTime but no corresponding Job was created.

Job created but Pod never scheduled. Insufficient node resources, missing node selectors, or PodDisruptionBudgets can prevent a Pod from being scheduled. The Job exists. The CronJob ran. Nothing executed.

ConcurrencyPolicy: Forbid skipping runs. If the previous run is still active when the next schedule fires, Kubernetes skips the new run rather than queueing it. Without external monitoring, these skips are invisible.

OOMKilled Pods. If a Pod exceeds its memory limit, it's killed silently. The Job may retry (up to backoffLimit), but if all retries are exhausted, the Job is marked failed — with no external alert unless you've configured one.

Succeeded Pods with empty output. The Pod ran, exited 0, and the Job is marked Succeeded. The container processed nothing useful. Kubernetes has no concept of output validation.

The HTTP ping pattern for Kubernetes CronJobs

The pattern is straightforward. Your container sends a start ping when it begins, a success or fail ping when it finishes. An external monitoring service tracks whether pings arrive on schedule.

Here's a complete example for a Python-based CronJob container:

# entrypoint.py
import os
import sys
import requests
import traceback

MONITOR_ID = os.environ["CRONTIFY_MONITOR_ID"]
API_KEY = os.environ["CRONTIFY_API_KEY"]
BASE_URL = f"https://api.crontify.com/api/v1/ping/{MONITOR_ID}"
HEADERS = {"X-API-Key": API_KEY}


def ping(event: str, payload: dict | None = None) -> None:
    try:
        requests.post(
            f"{BASE_URL}/{event}",
            headers=HEADERS,
            json=payload,
            timeout=10,
        )
    except Exception:
        pass  # monitoring failure must not prevent the job from running


def main() -> None:
    ping("start")
    try:
        result = run_job()
        ping("success", {
            "meta": {
                "records_processed": result["count"],
                "duration_ms": result["duration_ms"],
            }
        })
    except Exception as e:
        ping("fail", {
            "message": str(e),
            "log": traceback.format_exc(),
        })
        sys.exit(1)


if __name__ == "__main__":
    main()

For a Go-based container:

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
    "time"
)

func ping(monitorID, apiKey, event string, payload map[string]any) {
    url := fmt.Sprintf("https://api.crontify.com/api/v1/ping/%s/%s", monitorID, event)

    body, _ := json.Marshal(payload)
    req, _ := http.NewRequest("POST", url, bytes.NewReader(body))
    req.Header.Set("X-API-Key", apiKey)
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{Timeout: 10 * time.Second}
    resp, err := client.Do(req)
    if err != nil {
        return // never block the job
    }
    defer resp.Body.Close()
}

Kubernetes CronJob YAML

Inject the monitoring credentials as Kubernetes secrets and reference them as environment variables:

kubectl create secret generic crontify-credentials \
  --from-literal=api-key=ck_live_your_key \
  --from-literal=monitor-id=your-monitor-id \
  -n your-namespace

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-sync
  namespace: production
spec:
  schedule: "0 2 * * *"
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 3
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  jobTemplate:
    spec:
      backoffLimit: 0   # don't retry — let external monitoring catch failures
      activeDeadlineSeconds: 3600  # kill after 1 hour
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: sync
              image: your-registry/sync:latest
              env:
                - name: CRONTIFY_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: crontify-credentials
                      key: api-key
                - name: CRONTIFY_MONITOR_ID
                  valueFrom:
                    secretKeyRef:
                      name: crontify-credentials
                      key: monitor-id
              resources:
                requests:
                  memory: "256Mi"
                  cpu: "100m"
                limits:
                  memory: "512Mi"
                  cpu: "500m"

Note backoffLimit: 0. With external monitoring in place, retries within Kubernetes can obscure the true failure count — your monitor receives a start ping, no success ping arrives within the threshold, and an alert fires. Kubernetes retrying silently behind the scenes doesn't change the monitoring outcome.

What external monitoring catches that kube-state-metrics misses

Scheduler failures outside the cluster. If the control plane has issues that prevent the CronJob controller from running, no Jobs are created. kube-state-metrics can't scrape a metric for a Job that doesn't exist. External monitoring detects the absence of the expected ping.

Silent output failures. A Job marked Succeeded by Kubernetes processed nothing useful. kube-state-metrics sees kube_job_status_succeeded=1. External monitoring sees a metadata value of records_processed=0 and fires an alert.

Network-level job failures. A container that fails due to a DNS resolution error or a connection refused to an external service may exit 0 if the error is swallowed. External monitoring catches this via the metadata attached to the success ping.

Setting up the monitor

In Crontify, create a monitor with:

• Expected schedule: 0 2 * * * — the same cron expression as your CronJob
• Grace period: 15–30 minutes — allows time for Pod scheduling, image pulling, and init containers
• Max duration: your activeDeadlineSeconds value, or slightly above your p95 runtime
• Alert rules: records_processed eq 0 → fire alert

Crontify is free for up to 5 monitors — no credit card required.