← Back to the library
BackendApplied · 8 min

A request that does minutes of work times out, then runs twice

“"Generate report" takes three minutes. Users get a 504 or 524 while the server keeps working, click again, and the report is built twice. Sometimes the gateway or client library retries on its own.”

ERROR CODES YOU MAY SEE

A code is a clue. Use its meaning and surrounding evidence to narrow the cause.

504HTTP status codes · RFC 9110 · standard

A gateway or proxy did not receive a timely response from the upstream server it needed to complete the request.

Compare the proxy's timeout with the upstream's real latency. The upstream may still finish the work after the client saw 504, so only retry idempotent requests.

Official reference for 504 (opens in new tab)Code definition reviewed
524Cloudflare 5xx errors · platform

Cloudflare connected to the origin, but the origin did not send an HTTP response before the Proxy Read Timeout (125 seconds by default).

The origin usually keeps working after the 524. Move long work to a background job with a status resource; only Enterprise zones can raise the timeout.

Official reference for 524 (opens in new tab)Code definition reviewed
23505PostgreSQL SQLSTATE · standard

A write violated a uniqueness constraint (unique_violation).

Inspect the constraint name. A collision on an idempotency key can identify a retry, but other unique constraints can fail too; it does not prove a payment succeeded.

Official reference for 23505 (opens in new tab)Code definition reviewed

THE PRINCIPLE

The request records the intent; a worker does the work. Make both the submission and the job idempotent.

FIRST MOVES

  1. Validate synchronously, then persist a job row and return 202 Accepted with Location pointing at a status resource and a Retry-After hint. Reject invalid input with 4xx before accepting anything.
  2. Require an Idempotency-Key (or derive one from the intent) with a unique constraint, so a retried POST returns the existing job instead of enqueuing another.
  3. Insert the job and its queue message atomically (transactional outbox, or a database-backed queue), otherwise a crash between the two leaves a job that never runs.
  4. Make the worker idempotent too: queues deliver at least once and visibility timeouts expire on slow jobs. Record progress and results on the job row, with heartbeats or checkpoints for long work.
  5. Report status: GET returns 200 with status and progress; when done, 303 See Other to the result. Use SSE or webhooks when polling is too slow or too chatty, offer DELETE to cancel, and expire old job records.

TOOLS: THEN → NOW

Raising gateway and proxy timeouts to ten minutes202 Accepted plus a status resource (asynchronous request-reply)
Work done inline in the web processJob queues or durable workflow engines with retries and checkpoints
Clients poll in a tight loopPolling with Retry-After, Server-Sent Events, or a signed webhook on completion

PATTERN SNAPSHOT

// POST /exports: accept once per Idempotency-Key, return 202 and a status URL.
app.post("/exports", async (req, res) => {
  const key = req.get("Idempotency-Key");
  if (!key) return res.status(400).json({ error: "Idempotency-Key required" });
  const params = exportParams.parse(req.body); // validate before accepting
  // One transaction: INSERT ... ON CONFLICT (owner_id, key) DO NOTHING, plus an outbox row.
  const job = await jobs.acceptOnce({ ownerId: req.user.id, key, params });
  if (!job.sameParams) return res.status(422).json({ error: "Key reused with other params" });
  res.status(202).location(`/exports/${job.id}`).set("Retry-After", "5")
    .json({ id: job.id, status: job.status });
});
// GET /exports/:id -> 200 { status, progress }; when done, 303 See Other to the file.

CLOSE THE AI. EXPLAIN THIS.

The client's POST times out after the server accepted the job. What must the retry carry, what does the server return for it, and why must the worker still tolerate running the same job twice?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed