Dead Letter Queues

Where messages go when retrying has stopped helping.

A dead-letter queue is not a fix. It is three things together, and a team that implements only the first has made their system worse:

📌 A DLQ is a quarantine + an alarm + a runbook. Without the alarm it is a silent failure absorber. Without the runbook it is a growing pile nobody knows how to clear.

This module covers the mechanics, the retention behaviour that catches everyone out, and what you actually do at 3 a.m. when the DLQ alarm fires.

1. What you will learn

• Configure a DLQ with a redrive policy and a redrive allow policy
• Explain exactly when a message moves to the DLQ — it is not when you think
• Explain the retention-timestamp difference between standard and FIFO DLQs
• Replay messages from a DLQ safely
• Write the operational runbook for a growing DLQ

2. Why a DLQ exists

Module 10 established that not deleting a message retries it. That raises an obvious question: what about a message that can never succeed?

Without a DLQ there are exactly two outcomes, and both are bad:

A DLQ provides the third option: stop trying, move it aside, and tell someone.

Why does the message have to move to a different queue? Because "failed" is not a state a message can occupy in place. SQS has no per-message status field, no way to mark something as quarantined, and no query interface to find failed messages (Module 02 §9). A separate queue is the status field — membership in it means "gave up on this one".

3. The mechanics

flowchart LR
    P["Producer"] --> MQ[("Main queue<br/>maxReceiveCount = 5")]
    MQ -->|receive| C["Consumer"]
    C -->|"success"| DEL["DeleteMessage ✅"]
    C -->|"failure — no delete"| VIS["visibility expires<br/>count++"]
    VIS --> CHECK{"count ><br/>maxReceiveCount?"}
    CHECK -->|No| MQ
    CHECK -->|Yes| DLQ[("Dead-letter queue")]
    DLQ --> ALARM["🚨 CloudWatch alarm<br/>depth > 0"]
    ALARM --> HUMAN["👤 Runbook"]

Reading the diagram. The loop on the left is Module 10's retry mechanism. The DLQ is the exit from that loop — and note that the alarm and the human are drawn as part of the design, not as optional extras.

Configuration

The redrive policy lives on the source queue and points at the DLQ's ARN:

// 1. Create the DLQ first — it must exist before it can be referenced
String dlqUrl = sqs.createQueue(r -> r
        .queueName("sqs-course-payments-dlq")
        .attributes(Map.of(
            // 14 days — you want maximum time to investigate. See §5.
            QueueAttributeName.MESSAGE_RETENTION_PERIOD, "1209600")))
    .queueUrl();

String dlqArn = sqs.getQueueAttributes(r -> r
        .queueUrl(dlqUrl)
        .attributeNames(QueueAttributeName.QUEUE_ARN))
    .attributes().get(QueueAttributeName.QUEUE_ARN);

// 2. Attach the redrive policy to the SOURCE queue
sqs.setQueueAttributes(r -> r
        .queueUrl(mainQueueUrl)
        .attributes(Map.of(
            QueueAttributeName.REDRIVE_POLICY,
            """
            {"deadLetterTargetArn":"%s","maxReceiveCount":5}
            """.formatted(dlqArn))));

CLI:

aws sqs set-queue-attributes \
  --queue-url "$MAIN_QUEUE_URL" \
  --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"'"$DLQ_ARN"'\",\"maxReceiveCount\":\"5\"}"}'

The rules

4. When the move actually happens

This is the detail that surprises people, and it matters during incidents.

⚠️ The move happens on a receive attempt, not at the moment of failure.

With maxReceiveCount = 3, a message is not moved when it fails for the third time. It is moved when SQS goes to hand it out for the fourth time and observes that the count has been exceeded.

attempt 1 → fails → count = 1
attempt 2 → fails → count = 2
attempt 3 → fails → count = 3
          ↓ visibility expires, message becomes available
SQS prepares to deliver again → count would exceed 3 → MOVED TO DLQ

The practical consequence catches people out during outages:

📌 If nothing is polling the queue, a doomed message does not move itself to the DLQ. It sits there. A circuit breaker that stops consumption (Module 10 §6) also stops messages progressing toward the DLQ — which is usually what you want, but means "the DLQ is empty" does not imply "nothing is failing".

5. The retention trap

The single most consequential DLQ detail, and it differs by queue type.

⚠️ Standard queues preserve the original enqueue timestamp when a message moves to the DLQ. FIFO queues reset it (aws-facts.md §7).

What that means concretely

STANDARD — the clock keeps running
──────────────────────────────────────────────────────────────
day 0    message sent to main queue
day 0-3  retries, failures, backoff
day 3    moves to the DLQ
         DLQ retention = 4 days (the default)
         ⚠️ message has 1 day left, not 4
day 4    DELETED — silently, evidence gone


FIFO — the clock resets
──────────────────────────────────────────────────────────────
day 0    message sent
day 3    moves to the DLQ
         DLQ retention = 4 days
         ✅ message has a full 4 days from arrival
day 7    deleted

On a standard queue, a message that spent most of its source-queue retention failing arrives at the DLQ already nearly expired. You get a DLQ alarm, an engineer looks at it the next morning, and the message is gone.

📌 Set DLQ retention to 14 days. Always. It is the maximum, it costs nothing extra (SQS bills per request, not per byte-day — Module 18), and it is the only defence against this.

A second distortion

ApproximateAgeOfOldestMessage on the DLQ means different things by type:

So on a standard DLQ, "oldest message age = 3 days" may mean it arrived three minutes ago after three days of retries. Worth knowing before you draw conclusions from it at 3 a.m.

6. The redrive allow policy

The mirror of the redrive policy. It lives on the DLQ and controls which source queues may use it.

{ "redrivePermission": "byQueue",
  "sourceQueueArns": ["arn:aws:sqs:eu-west-1:123456789012:sqs-course-payments"] }

Three options:

Why it matters: the default lets any queue in the account dump into your DLQ. On a shared DLQ you then cannot tell whose messages failed, which destroys the isolation a per-queue DLQ was supposed to give you.

📌 One DLQ per source queue. A shared DLQ means a broken notification worker and a broken payment worker fill the same alarm, and neither team knows which one is theirs (Module 20).

7. Inspecting a DLQ

Before you can act, you need to see what is in there — without consuming it.

// Read without destroying: receive, inspect, then release immediately
var response = sqs.receiveMessage(r -> r
        .queueUrl(dlqUrl)
        .maxNumberOfMessages(10)
        .waitTimeSeconds(5)
        .messageAttributeNames("All")
        .messageSystemAttributeNames(MessageSystemAttributeName.ALL));

for (Message m : response.messages()) {
    System.out.printf("messageId=%s receiveCount=%s sentAt=%s%n  body=%s%n",
            m.messageId(),
            m.attributes().get(MessageSystemAttributeName.APPROXIMATE_RECEIVE_COUNT),
            m.attributes().get(MessageSystemAttributeName.SENT_TIMESTAMP),
            m.body());

    // Release it back immediately — do NOT hold the lease while a human reads
    sqs.changeMessageVisibility(r -> r
            .queueUrl(dlqUrl)
            .receiptHandle(m.receiptHandle())
            .visibilityTimeout(0));
}

⚠️ Receiving from a DLQ makes messages invisible for the visibility timeout. If you sample 10 messages and walk away, those 10 are hidden from everyone else — including from a redrive. Always release with ChangeMessageVisibility(0).

What to look for, in order:

8. Replay — the redrive operation

Once the cause is fixed, move messages back with StartMessageMoveTask.

String taskHandle = sqs.startMessageMoveTask(r -> r
        .sourceArn(dlqArn)                  // moving FROM the DLQ
        .destinationArn(mainQueueArn)       // back to the source queue
        .maxNumberOfMessagesPerSecond(50)   // ⚠️ rate limit — see below
    ).taskHandle();

// Monitor
sqs.listMessageMoveTasks(r -> r.sourceArn(dlqArn).maxResults(1))
   .results().forEach(t -> System.out.printf(
        "status=%s moved=%d failed=%d%n",
        t.status(), t.approximateNumberOfMessagesMoved(),
        t.failureReason()));

// Abort if it is going badly
sqs.cancelMessageMoveTask(r -> r.taskHandle(taskHandle));

CLI:

aws sqs start-message-move-task \
  --source-arn "$DLQ_ARN" \
  --destination-arn "$MAIN_QUEUE_ARN" \
  --max-number-of-messages-per-second 50

⚠️ Always set maxNumberOfMessagesPerSecond. Without it, a redrive of 50,000 messages hits your consumers — and therefore your downstream — as fast as SQS can move them. If the DLQ filled because the downstream was struggling, an unlimited redrive re-creates the outage immediately.

⚠️ Redriven messages arrive with their receive count reset, so they get a full maxReceiveCount budget again. If the cause is not actually fixed, they will make the same round trip and land back in the DLQ — with more elapsed retention consumed, on a standard queue.

9. The runbook

What a DLQ alarm should trigger. The first question is the one that determines everything else.

flowchart TD
    ALARM["🚨 DLQ depth > 0"] --> COUNT{"How many,<br/>and how fast?"}

    COUNT -->|"A few, slowly"| POISON["<b>Poison messages</b><br/>Individual bad messages"]
    COUNT -->|"Thousands, in minutes"| OUTAGE["<b>Outage</b><br/>Something downstream broke"]

    POISON --> P1["Sample and inspect the bodies"]
    P1 --> P2["Is it a data problem<br/>or a code problem?"]
    P2 --> P3["Fix the producer or the consumer<br/>Deploy"]
    P3 --> P4["Redrive — volume is small,<br/>rate limit is less critical"]

    OUTAGE --> O1["<b>Check the downstream FIRST</b><br/>The DLQ is the symptom"]
    O1 --> O2["Is it still broken?"]
    O2 -->|Yes| O3["🛑 Do NOT redrive.<br/>Fix the downstream.<br/>Consider a circuit breaker<br/>to stop the DLQ growing"]
    O2 -->|No| O4["Redrive with a rate limit<br/>well below the downstream's capacity"]
    O4 --> O5["Watch downstream latency<br/>while it drains.<br/>Cancel if it degrades."]

Reading the diagram. The two branches call for almost opposite actions. Redriving during an outage repeats the failure at higher receive counts; investigating a 50,000-message spike message-by-message wastes the time you should be spending on the downstream.

The order that matters

1. Do not redrive before diagnosing. The messages will fail again, having consumed more retention.
2. Capture evidence first. Sample a few message bodies and screenshot the metric graphs. Once you fix and redrive, both are gone.
3. Fix the cause, deploy, verify on new traffic — then redrive.
4. Rate-limit the redrive below the downstream's headroom.
5. Watch while it drains. Cancel the task if downstream latency climbs.

🎯 Interview point. "Your DLQ has 50,000 messages. What do you do?" — the strongest answer starts by distinguishing poison from outage, and explicitly refuses to redrive before the cause is confirmed fixed.

10. FIFO queues and DLQs

Covered in Module 07 §8 and worth restating, because it is a genuine design decision rather than a detail.

⚠️ AWS advises against a DLQ on a FIFO queue where strict ordering must never break.

When a blocked message moves to the DLQ, the rest of its group proceeds — without that message having been processed. The group's sequence has a permanent hole, and for ledger entries or state transitions the resulting state is silently wrong.

For order-critical FIFO workloads the usual answer is: attach the DLQ, alarm aggressively, and treat every arrival as a correctness incident requiring reconciliation — not as routine failure handling.

11. Production considerations

A DLQ with no alarm is worse than no DLQ. It converts a loud failure (retry loops, error logs) into a silent one. Alarm on ApproximateNumberOfMessagesVisible > 0.

Alarm on the rate as well as the depth. Depth tells you something is wrong; the rate of growth tells you which kind of wrong, which is the first branch of the runbook.

DLQ retention at 14 days, because of §5.

One DLQ per source queue, enforced with a redrive allow policy.

Give the DLQ its own dashboard panel next to the source queue's, so the relationship is visible at a glance.

Consider a DLQ for your DLQ. If a redrive fails repeatedly, messages bounce between the two. A second-level DLQ — or simply a very long retention and a manual process — stops the loop.

Keep the correlation ID in message attributes, not the body. A DLQ message whose body you cannot parse is exactly when you most need to trace it back (Module 17).

Check maxReceiveCount × visibilityTimeout against retention. If the product approaches retention, messages expire before reaching the DLQ and you lose both the message and the evidence (Module 08 §9).

12. Common mistakes

No alarm on the DLQ. Why it's wrong: failures accumulate invisibly; the system looks healthy while work is quietly being abandoned. Instead: alarm on depth > 0, and separately on growth rate.

DLQ retention shorter than or equal to the source queue's. Why it's wrong: on standard queues the enqueue timestamp is preserved, so messages can expire almost immediately on arrival. Instead: 14 days.

Redriving before fixing the cause. Why it's wrong: the messages fail again, consuming more retention and more downstream capacity. Instead: diagnose, fix, deploy, verify on live traffic, then redrive.

Redriving at full speed. Why it's wrong: if the DLQ filled because the downstream was struggling, an unlimited redrive re-creates the outage. Instead: maxNumberOfMessagesPerSecond, set below the downstream's headroom.

Inspecting a DLQ without releasing the messages. Why it's wrong: sampled messages stay invisible for the visibility timeout, hidden even from a redrive. Instead: ChangeMessageVisibility(0) after reading.

A shared DLQ for several queues. Why it's wrong: you cannot tell whose messages failed, and one team's alarm is everyone's alarm. Instead: one DLQ per source, with a byQueue redrive allow policy.

Treating the DLQ as the fix. Why it's wrong: it is a quarantine. Nothing is repaired, retried or replayed automatically. Instead: quarantine + alarm + runbook.

Mismatched queue types. Why it's wrong: a FIFO source cannot use a standard DLQ; the configuration is rejected. Instead: match the types.

13. Real-world example

3:07 a.m. Page: payments-dlq depth > 0.

The first look

DLQ depth        : 12,847  and climbing ~50/sec
Main queue depth : 41,200  and climbing
Oldest message   : 6 minutes
Consumers        : all healthy, CPU 40%

The first branch of the runbook resolves instantly: 12,847 messages in six minutes is not a poison message. This is an outage. So the next place to look is not the DLQ.

The actual cause

The payment gateway had begun returning 503 at 03:01. Every message failed, retried five times across about 90 seconds, and landed in the DLQ. maxReceiveCount = 5 with a 20-second visibility timeout meant each message exhausted its budget in under two minutes.

The DLQ was the symptom. The gateway was the cause.

Two things went wrong that were not the gateway

1. The retry budget was far too short for the failure mode. A gateway incident lasting more than two minutes guaranteed that every in-flight message would be quarantined. These were valid payments, discarded by configuration.

2. There was no circuit breaker, so consumers kept hammering a dead gateway and kept manufacturing DLQ entries (Module 10 §6).

What they did, in order

03:09  Confirmed the gateway was down via its status page and their own metrics.
03:10  Did NOT redrive. Captured three sample message bodies and screenshotted
       the DLQ growth graph, because both would be gone after the fix.
03:11  Scaled consumers to zero. This stopped the DLQ growing — messages
       accumulated in the main queue instead, which is where they belong.
03:44  Gateway recovered.
03:47  Scaled consumers back up at 25% of normal. Watched gateway latency.
03:52  Full consumer capacity. Main queue backlog draining.
04:05  Main queue clear. Now — and only now — redrove the DLQ:

           start-message-move-task --max-number-of-messages-per-second 20

       20/sec against a gateway that handles 400/sec: deliberately
       conservative, because these messages had already failed once.
04:16  12,847 messages redriven. 3 failed again — genuinely malformed,
       from an unrelated producer bug. Those stayed in the DLQ.

The key decision was at 03:11. Scaling consumers to zero stopped the bleeding without losing anything — the messages simply waited in the main queue, which is what a queue is for.

What changed afterwards

🎯 The transferable lesson: the DLQ alarm was correct and the DLQ was the wrong place to look. A DLQ tells you that something failed, not what. Treat it as a pointer to the downstream, not as the object of investigation.

14. Interview questions

🟢 Beginner

What is a dead-letter queue? A separate queue that SQS moves messages to after they have been received more times than maxReceiveCount without being deleted. It quarantines messages that repeatedly fail, so they stop consuming capacity.

What is a redrive policy? The configuration on a source queue that names the DLQ's ARN and sets maxReceiveCount.

🟡 Intermediate

What exactly triggers the move to a DLQ? A receive attempt after the receive count has exceeded maxReceiveCount — not the failure itself. With maxReceiveCount = 3, the message moves when SQS goes to deliver it a fourth time. A consequence is that if nothing is polling, a doomed message does not move itself to the DLQ.

How do you get messages out of a DLQ? StartMessageMoveTask moves them back to the source queue or another target, with a configurable rate limit. Always set the rate limit, and only after the cause is fixed.

🔴 Advanced

Why must a DLQ's retention period be longer than the source queue's?

Because on standard queues the original enqueue timestamp is preserved when a message moves. Retention is measured from when the message was first sent, not from when it arrived in the DLQ. A message that spent three days failing in the source queue arrives at a four-day DLQ with one day left — so an engineer who looks at the alarm the next morning may find the evidence already deleted.

FIFO queues reset the timestamp instead, so a FIFO DLQ message gets its full window from arrival.

The practical rule is to set DLQ retention to 14 days regardless of type. It is the maximum and costs nothing, since SQS bills per request rather than per byte-day.

Your DLQ has 50,000 messages after a downstream outage. Describe your recovery procedure.

First, distinguish the case — 50,000 messages arriving quickly means an outage, not poison messages, so the DLQ is the symptom and the downstream is where to look.

Then, in order:

1. Confirm the downstream is actually healthy before anything else. Redriving into a still-broken dependency repeats the failure at higher receive counts.
2. Stop the bleeding. Scale consumers down or open a circuit breaker, so messages accumulate in the main queue rather than being pushed through the retry loop into the DLQ.
3. Capture evidence — sample bodies, metric screenshots — because both disappear once you fix and redrive.
4. Drain the main queue first. The DLQ messages have already waited; adding them to a live backlog doubles the pressure.
5. Redrive with a rate limit well below the downstream's spare capacity. For 50,000 messages against a 400 req/s dependency, something like 20–50/sec.
6. Watch downstream latency during the drain and cancel the move task if it degrades.
7. Expect a residue. Messages that fail again are genuinely different — real poison — and should be investigated individually.

Afterwards, the structural fixes: a circuit breaker, a retry budget sized against realistic outage durations, and an alarm on DLQ growth rate so the poison-versus-outage distinction is made by the alarm rather than by a person at 3 a.m.

Why does AWS caution against using a DLQ with a FIFO queue? Because the move breaks the ordering guarantee. A blocked message leaving for the DLQ allows the rest of its group to proceed without it ever having been processed, leaving a permanent gap in the sequence. For ledger entries or state transitions the resulting state is wrong and nothing downstream detects it. The alternative — no DLQ — preserves ordering but blocks the group until the message expires, potentially for 14 days. Both outcomes are bad; the choice is a business decision about which failure is more tolerable.

How can a message fail permanently and never reach the DLQ? Several ways. Retention expires first, if maxReceiveCount × visibilityTimeout approaches the retention period. Nothing is polling the queue, so the receive count never increments — a circuit breaker causes exactly this. A consumer catches the exception and deletes the message, which is silent data loss. Or the queue has no redrive policy at all, in which case the message simply retries until it expires.

⚫ System design

Design the complete failure-handling strategy for a payments pipeline: retries, DLQ, alarms, replay, and audit.

Classification first, because it determines everything downstream:

That first distinction — a declined card is not a failure — is the one candidates miss. Routing business outcomes into a DLQ fills it with noise and buries the real failures.

Retry budget, sized against the dependency's real behaviour: if gateway incidents typically last 10 minutes, a budget summing to 2 minutes quarantines valid payments routinely. Exponential backoff, maxReceiveCount around 8, and a circuit breaker so a prolonged outage stops consumption instead of manufacturing DLQ entries.

DLQ configuration: one per source queue, FIFO-matched if the source is FIFO, 14-day retention, byQueue redrive allow policy.

Alarms, three of them and each answering a different question:

• DLQ depth > 0 → something failed (ticket)
• DLQ growth rate > N/min → an outage is in progress (page)
• ApproximateAgeOfOldestMessage on the main queue → we are falling behind (page)

Replay as a documented, rehearsed procedure: verify the cause is fixed, drain the main queue, redrive rate-limited, monitor, expect a residue.

Audit, which payments specifically requires: every message's outcome — succeeded, declined, quarantined, replayed — written to a durable store keyed on paymentId, independent of the queue. The queue is transport; it must never be the system of record. This also gives you the reconciliation source for the ambiguous cases from Module 09 §8.

And the thing that ties it together: idempotency, because replay means re-processing. A DLQ replay three weeks after the fact must not double-charge anyone — which is why the idempotency TTL in Module 09 §9 must exceed DLQ retention.

Follow-ups to expect: "What if a redriven message fails again?" — it returns to the DLQ with a fresh receive count; a second-level DLQ or a manual quarantine stops the loop. "How do you know the DLQ is empty because nothing failed, rather than because nothing is polling?" — correlate with the source queue's receive metrics; this is why an empty DLQ is not by itself evidence of health.

15. Summary

• A DLQ is a quarantine + an alarm + a runbook. Any one alone makes things worse.
• Redrive policy on the source, naming the DLQ ARN and maxReceiveCount. Same account, Region and queue type.
• The move happens on a receive attempt, not at the moment of failure — so an unpolled queue sends nothing to its DLQ.
• Standard queues preserve the enqueue timestamp on the move; FIFO resets it. Set DLQ retention to 14 days.
• One DLQ per source queue, enforced with a byQueue redrive allow policy.
• Inspect without consuming — release with ChangeMessageVisibility(0).
• Diagnose before redriving, and always rate-limit the redrive.
• Depth tells you something failed; growth rate tells you whether it is poison or an outage.
• On FIFO, a DLQ move breaks ordering by design. Decide consciously.
• The DLQ is usually a pointer to the downstream, not the object of investigation.

← Previous: 10 — Retries · Index: Course home · Next: 12 — Long Polling →