Cron job idempotency: why it matters and how to implement it

Idempotency is the property that running an operation once produces the same result as running it multiple times. For cron jobs, it means: if your nightly sync runs twice because of a bug or a manual re-run, the data should be in the same state as if it had run once.

Most cron jobs are not idempotent. Most teams don't realise this until the first time an overlap, a retry, or a deployment restart causes a job to run twice — and the consequences are duplicate records, double-charged customers, or sent emails that shouldn't have been sent.

Idempotency is not a nice-to-have. It is the property that determines whether your job is safe to run in a real production environment.

Why cron jobs run more than once

Several common scenarios cause a job to execute more than once:

Overlap. A job that normally takes 5 minutes starts taking 20 minutes due to slow database queries. At the 15-minute mark, cron starts a new instance. Now two instances are running simultaneously, both processing the same queue.

Manual re-run. A job failed. You fix the underlying issue and manually trigger it again. If the job was partially completed before failure, the re-run starts from the beginning rather than from where it stopped.

Deployment restart. A deployment restarts the application while a scheduled job is running. The scheduler process restarts, sees the job as due, and starts a new instance — unaware that the previous one was partially complete.

Retry logic. The job failed on its first run. An automatic retry (configured in Kubernetes, Celery, or Sidekiq) starts a second run. If the first run made partial changes before failing, the second run encounters data in an intermediate state.

The idempotency patterns that work

1. Upsert instead of insert

The most common source of duplicate records is using INSERT where INSERT OR REPLACE (or ON CONFLICT DO UPDATE) would be correct.

-- Non-idempotent: creates a duplicate if run twice
INSERT INTO synced_records (external_id, data, synced_at)
VALUES ($1, $2, NOW());

-- Idempotent: updates if the record already exists
INSERT INTO synced_records (external_id, data, synced_at)
VALUES ($1, $2, NOW())
ON CONFLICT (external_id)
DO UPDATE SET
  data = EXCLUDED.data,
  synced_at = EXCLUDED.synced_at;

Use external_id — the identifier from the upstream source — as the conflict key, not an auto-generated ID. Auto-generated IDs change on each insert; external IDs are stable.

2. Track processing state explicitly

For jobs that process items from a queue or a database table, add an explicit state column and filter by it:

-- Add status tracking to the source table
ALTER TABLE pending_emails ADD COLUMN status TEXT DEFAULT 'pending';
ALTER TABLE pending_emails ADD COLUMN processed_at TIMESTAMPTZ;

-- Job queries only pending items
SELECT * FROM pending_emails
WHERE status = 'pending'
ORDER BY created_at
LIMIT 1000;

-- Mark as processed atomically (use FOR UPDATE to prevent concurrent processing)
UPDATE pending_emails
SET status = 'processing', processing_started_at = NOW()
WHERE id = ANY($1)
  AND status = 'pending';  -- guard against concurrent updates

-- After successful processing
UPDATE pending_emails
SET status = 'sent', processed_at = NOW()
WHERE id = ANY($1);

The AND status = 'pending' guard in the UPDATE is critical. Without it, a concurrent instance can pick up records already being processed by another instance.

3. Use idempotency keys for external API calls

External APIs that create resources need an idempotency key — a stable identifier that tells the API "I've already made this request; return the result of the first call rather than creating a duplicate."

// Stripe payment creation with idempotency key
await stripe.paymentIntents.create(
  {
    amount: charge.amountCents,
    currency: 'usd',
    customer: charge.stripeCustomerId,
  },
  {
    // Derived from the internal charge ID — stable across retries
    idempotencyKey: `charge-${charge.id}`,
  }
);

For APIs that don't support idempotency keys natively, track the external resource ID in your database and check for its existence before creating:

const existing = await db.externalResources.findFirst({
  where: { internalId: record.id, provider: 'stripe' }
});

if (existing) {
  return existing.externalId; // already created, return the cached ID
}

const resource = await createExternalResource(record);
await db.externalResources.create({
  data: {
    internalId: record.id,
    externalId: resource.id,
    provider: 'stripe',
  }
});

4. Distributed locking for overlap prevention

A distributed lock prevents two instances of the same job from running simultaneously. Redis is the standard implementation:

import { CrontifyMonitor } from '@crontify/sdk';

const LOCK_KEY = 'job:nightly-sync:lock';
const LOCK_TTL_MS = 30 * 60 * 1000; // 30 minutes

const monitor = new CrontifyMonitor({
  apiKey: process.env.CRONTIFY_API_KEY!,
  monitorId: 'your-monitor-id',
});

async function nightly_sync(): Promise<void> {
  const acquired = await redis.set(LOCK_KEY, '1', 'NX', 'PX', LOCK_TTL_MS);

  if (!acquired) {
    // Previous instance still running — log and exit cleanly
    console.log('Previous instance still running, skipping this execution');
    return;
  }

  try {
    await monitor.wrap(async () => {
      const result = await syncRecords();
      return { meta: { records_synced: result.count } };
    });
  } finally {
    await redis.del(LOCK_KEY);
  }
}

The TTL on the lock is the safety net: if the job crashes without releasing the lock, the TTL ensures future runs aren't permanently blocked.

Idempotency and monitoring work together

Idempotency prevents damage when a job runs multiple times. Monitoring tells you when a job runs at unexpected times — which is when idempotency is tested.

Crontify's overlap detection surfaces when a job's start ping arrives while the previous run is still active. This is the monitoring signal that tells you your idempotency implementation is being exercised in production — a job is running twice, and you should verify that the output is correct.

Alert rules on output metadata provide a second layer of verification. If a supposedly idempotent job runs twice and the second run processes zero records (because the first run already processed everything), a records_processed eq 0 rule would fire — which may or may not be the expected outcome depending on your implementation. Seeing the alert is the prompt to verify.

The combination: idempotency keeps your data correct when jobs run multiple times. External monitoring tells you when they do.

Frequently asked questions

Is idempotency the same as at-least-once vs exactly-once delivery?

Related but not identical. At-least-once delivery guarantees that a message or task will be processed, potentially multiple times. Exactly-once is the stronger guarantee that it will be processed exactly once — which is much harder to achieve and often impossible in distributed systems without coordination. Idempotent jobs are safe in an at-least-once delivery model because processing them multiple times produces the same result as once.

What about email sends — how do I make those idempotent?

Track sent emails in your database before sending:

const alreadySent = await db.sentEmails.findFirst({
  where: { userId: user.id, templateId: 'welcome', date: today() }
});

if (!alreadySent) {
  await sendEmail(user, 'welcome');
  await db.sentEmails.create({
    data: { userId: user.id, templateId: 'welcome', date: today() }
  });
}

The findFirst + create sequence should be wrapped in a database transaction or use a unique constraint to prevent concurrent writes.

My job doesn't have external IDs to use as conflict keys — how do I handle deduplication?

Derive a stable hash from the content being processed. If you're importing CSV rows, hash the row content. If you're processing API responses, hash the relevant fields. Use the hash as the conflict key. This catches exact duplicates but not near-duplicates — which is usually the right trade-off.

Crontify is free for up to 5 monitors — no credit card required.