← Back to the library
System designApplied · 8 min

One bad message blocks or floods the queue

“Queue depth or consumer lag climbs, consumers look busy, and logs show the same message ID failing over and over.”

THE PRINCIPLE

Every message must eventually be acknowledged or dead-lettered; "retry forever" is an outage with a delay.

FIRST MOVES

  1. Find the repeating message ID and its receive/delivery count; capture the payload and stack trace before touching anything.
  2. SQS: set a redrive policy with maxReceiveCount high enough for transient failures, and give the DLQ longer retention than the source queue.
  3. RabbitMQ quorum queues: delivery-limit defaults to 20 since 4.0; add a dead-letter exchange. In 4.3+, basic.nack with requeue does not count toward the limit (basic.reject and crashes do), so a consumer that nacks every failure retries forever.
  4. Classic Kafka consumer groups have no broker redelivery limit: a failing record stalls its partition until you publish it to a dead-letter topic and commit past it. Share groups (production-ready since Kafka 4.2) do cap attempts via group.share.delivery.count.limit (default 5).
  5. Fail fast on permanent errors (schema, validation) instead of burning retries; keep backoff for transient ones.
  6. Size the visibility timeout to real processing time, or extend it while working, so slow messages are not redelivered as duplicates.
  7. Alarm on DLQ depth, fix the cause, then redrive. Consumers must dedupe by message or event ID.

TOOLS: THEN → NOW

Requeue on any exceptionBounded attempts plus DLQ: SQS redrive policy, RabbitMQ delivery-limit + DLX, Spring Kafka DeadLetterPublishingRecoverer
Scripts to copy messages backBuilt-in DLQ redrive (SQS console/StartMessageMoveTask) after the fix ships

PATTERN SNAPSHOT

for (const msg of batch) {
  try {
    const event = parseEvent(msg.body);           // throws PermanentError on bad data
    await processOnce(event.id, () => handle(event)); // idempotent by event ID
    await ack(msg);
  } catch (err) {
    log.warn({ id: msg.id, attempts: msg.deliveryCount, err });
    if (err instanceof PermanentError) await deadLetter(msg, err); // no retries
    // Transient: let the broker redeliver (SQS visibility timeout; RabbitMQ
    // basic.reject with requeue, since 4.3 nack is not counted) until the limit DLQs it.
  }
}

CLOSE THE AI. EXPLAIN THIS.

Why does a dead-letter queue break ordering on a FIFO queue, and when is that unacceptable?

HOW IT WORKS UNDERNEATH

SOURCES

Guide reviewed