← Back to the library
BackendApplied · 8 min

A background job runs on every replica

“After scaling to three instances, customers get three copies of the nightly email, or a worker restart processes the same job again.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

23505PostgreSQL SQLSTATE · standard

A write violated a uniqueness constraint (unique_violation).

Inspect the constraint name. A collision on an idempotency key can identify a retry, but other unique constraints can fail too; it does not prove a payment succeeded.

Official reference for 23505 (opens in new tab)Code definition reviewed

THE PRINCIPLE

Locks reduce duplicate execution; idempotency makes duplicates harmless. You need the second even when you have the first.

FIRST MOVES

  1. List where the job is triggered: in-process timers in each web replica, a CronJob, a queue consumer, a retry. Move scheduling out of web replicas or guard it.
  2. Use a lock the system already has: pg_try_advisory_xact_lock in PostgreSQL returns immediately and is released at commit or rollback, so a crash cannot leave it held. It keeps a connection and transaction open for the job's duration, so keep that work bounded.
  3. If you use a Redis lock, set it with NX and an expiry, release it only if the value is still yours, and expect it to expire mid-job.
  4. Make side effects idempotent: unique keys on outputs (one email per user per day), upserts, and provider idempotency keys.
  5. For queue consumers, treat redelivery as normal: record processed message IDs or make handlers naturally idempotent.

TOOLS: THEN → NOW

setInterval or an in-process cron in every web instanceOne scheduler (CronJob, cloud scheduler, or job framework) enqueuing work, or a database advisory lock guard
Trusting a lease to guarantee exclusivityLeases plus idempotent writes or fencing tokens checked by the resource

PATTERN SNAPSHOT

// Every replica may fire; only the one holding the lock does the work.
const NIGHTLY_INVOICES_LOCK = 842_001; // any app-wide unique bigint

await db.transaction(async (tx) => {
  const { rows } = await tx.query(
    "SELECT pg_try_advisory_xact_lock($1) AS acquired", [NIGHTLY_INVOICES_LOCK]);
  if (!rows[0].acquired) return; // another replica is running it

  // Still idempotent: UNIQUE (customer_id, period) on invoices.
  await createMissingInvoices(tx, currentPeriod());
}); // lock released automatically at COMMIT or ROLLBACK

CLOSE THE AI. EXPLAIN THIS.

Your Redis lock has a 30-second TTL and the job pauses for 40 seconds during a GC or network stall. What can happen, and what stops it from causing damage?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed