← Back to the library
ArchitectureDeep dive · 9 min

The database and the queue disagree after a failure

“An order is saved but the OrderPaid event never went out, or the event went out and the transaction rolled back. The search index is missing rows nobody can explain, and a nightly resync job keeps growing.”

THE PRINCIPLE

You can't atomically write to two systems. Write once to the database, and derive everything else from what committed.

FIRST MOVES

  1. Find every dual write: code that commits to the database and also publishes, indexes, caches or calls a webhook in the same request. Publishing inside the transaction is a dual write too, because the transaction can still roll back after the publish.
  2. Use a transactional outbox: insert the event row in the same transaction as the state change, so both commit or neither does.
  3. Relay outbox rows to the broker with a polling publisher or by tailing the transaction log (CDC, for example Debezium). For search indexes and caches, CDC on the source tables is often enough.
  4. Expect at-least-once delivery: consumers dedupe on the event id; preserve ordering per aggregate by keying on the aggregate id.
  5. Monitor relay lag and outbox size, and prune or archive published rows.

TOOLS: THEN → NOW

Publish after commit and hopeTransactional outbox with a relay process
Nightly full reindex to repair driftChange data capture from the database log (Debezium, DynamoDB Streams)
Distributed transactions (2PC/XA) across DB and brokerLocal transaction plus outbox, sagas for multi-service workflows

PATTERN SNAPSHOT

-- Debezium's default outbox columns: id uuid, aggregatetype/aggregateid/type varchar, payload jsonb.
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = $1;
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
VALUES (gen_random_uuid(), 'order', $1, 'OrderPaid', $2::jsonb);
COMMIT;

-- A relay (polling publisher, or Debezium's Outbox Event Router reading the WAL)
-- publishes committed rows at least once; consumers dedupe on outbox.id.

CLOSE THE AI. EXPLAIN THIS.

Why doesn't publishing the event inside the database transaction fix the dual write, and what must consumers do once you adopt an outbox?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed