Retries

SQS retries for free. That is exactly the problem.

There is no retry API, no backoff setting and no error classification. The default behaviour is: retry everything, at a fixed interval, immediately, until maxReceiveCount. For a downstream that is already struggling, that is close to the worst policy you could design — and it is what you get if you write no retry code at all.

This module covers how SQS retries actually work, how to classify errors so you stop retrying things that will never succeed, and how to build backoff on a service that offers none.

1. What you will learn

• Explain how SQS retries work — there is no retry mechanism, only visibility expiry
• Classify errors as transient, permanent or poison, and handle each differently
• Implement exponential backoff with jitter using ChangeMessageVisibility
• Explain retry storms and thundering herds, and how to avoid causing one
• Reason about total retry time before a message reaches the DLQ

2. There is no retry feature

Look for a retry setting in the SQS API. There isn't one. No retryPolicy, no backoffStrategy, no maxAttempts.

What exists instead:

📌 A retry is a visibility timeout expiring on a message you did not delete. That is the entire mechanism.

try {
    process(m);
    sqs.deleteMessage(...);      // success — the message is gone
} catch (Exception e) {
    log.error("failed", e);
    // Do nothing. Not deleting IS the retry.
}

That catch block contains no retry logic because none is needed. The message stays in flight until its lease expires, then becomes available again (Module 05 §5).

Three consequences follow immediately, and all three surprise people:

Why doesn't SQS have retry configuration? Because SQS never observes your failure. It sees only the absence of a delete. A "retry policy" would require SQS to know why processing failed — which would mean an API where you report failures, which is a very different service. The design keeps SQS ignorant of your semantics, which is the same reason deletion is explicit (Module 02 §6).

3. Classify before you retry

Retrying is only useful when the operation might succeed next time. Three categories:

Transient error — Simple: it might work if you try again in a moment. Technical: a failure caused by temporary conditions — timeouts, throttling, a restarting pod, a failing-over database, a 503. Example: SQLTransientConnectionException, HTTP 429, HTTP 503.

Permanent error — Simple: it will never work, however many times you try. Technical: a failure determined by the message content or by durable state — malformed JSON, a referenced entity that was deleted, a schema violation, a business-rule rejection. Example: JsonParseException, HTTP 400, a foreign key that does not exist.

Poison message — Simple: a message that reliably breaks your consumer. Technical: usually a permanent error, sometimes a message that crashes the process outright (OOM on a huge payload, a stack overflow in a parser). Example: a 900 KB payload that exhausts the heap.

Why the distinction pays

maxReceiveCount = 5, and a malformed message arrives.

Multiply by a producer bug that emits 40,000 malformed messages: 200,000 wasted downstream calls, or 40,000. During an incident, that difference is the difference between a degraded system and an outage.

In code

private boolean isRetryable(Exception e) {
    // Permanent — the message will never process. Do not retry.
    if (e instanceof JsonParseException
     || e instanceof ValidationException
     || e instanceof EntityNotFoundException) return false;

    // AWS SDK tells you directly — use it rather than guessing.
    if (e instanceof SdkException sdk) return sdk.retryable();

    // HTTP: 4xx is yours to fix, 5xx is theirs. 429 is the exception.
    if (e instanceof HttpResponseException http) {
        int s = http.statusCode();
        if (s == 429) return true;          // throttled — definitely retry, with backoff
        return s >= 500;
    }

    // Transient infrastructure
    if (e instanceof SQLTransientException
     || e instanceof TimeoutException
     || e instanceof ConnectException) return true;

    // Unknown: retry. An unrecognised bug is more likely transient than not,
    // and maxReceiveCount bounds the damage either way.
    return true;
}

📌 Default to retryable for unknown exceptions. maxReceiveCount already bounds the cost, and wrongly classifying a transient error as permanent means discarding work that would have succeeded.

Handling a permanent failure

Do not just delete it — that is silent data loss (Module 05 §11). Record it, then remove it:

catch (Exception e) {
    if (!isRetryable(e)) {
        failures.record(m.messageId(), m.body(), e);   // durable evidence
        deadLetters.send(m.body(), e.getMessage());    // explicit DLQ send
        sqs.deleteMessage(...);                        // now it is safe to delete
        return;
    }
    scheduleRetry(m, e);                                // §4
}

Sending it to the DLQ yourself — rather than letting it burn maxReceiveCount attempts — is called fast-tracking to the DLQ, and it is the main practical benefit of classification.

4. Backoff

The default retry interval is your visibility timeout, unchanged, forever. Against a downstream that is down, that is a fixed-rate load generator pointed at a struggling system.

Backoff means waiting longer after each successive failure. You build it with ChangeMessageVisibility (Module 08 §3).

flowchart TD
    RECV["Receive message<br/>ApproximateReceiveCount = n"] --> PROC["process()"]
    PROC -->|success| DEL["DeleteMessage ✅"]
    PROC -->|throws| CLASS{"isRetryable?"}

    CLASS -->|"No — permanent"| REC["Record failure<br/>+ send to DLQ<br/>+ delete"]
    CLASS -->|"Yes — transient"| CALC["delay = min(base × 2^(n-1), cap)<br/>delay = random(0, delay)"]

    CALC --> CMV["ChangeMessageVisibility(delay)<br/><i>do NOT delete</i>"]
    CMV --> WAIT["message hidden for delay"]
    WAIT -->|"delay elapses"| AVAIL["Available again"]
    AVAIL --> CHECK{"n > maxReceiveCount?"}
    CHECK -->|No| RECV
    CHECK -->|Yes| DLQ["→ DLQ 📮"]

Reading the diagram. Two paths reach the DLQ. The left one is deliberate and fast — you identified a permanent failure and quarantined it on attempt one. The right one is the slow, automatic path for transient failures that never recover.

The formula

delay = min(base × 2^(attempt - 1), cap)
delay = random(0, delay)                  ← full jitter

With base = 10s, cap = 300s:

Jitter is not optional

This is the part people skip, and it is the part that matters most.

Picture a database failing over. Four hundred messages fail within the same second. Without jitter, all four hundred retry at exactly the same moment, 10 seconds later. And again at 30 seconds. And again at 70.

You have built a synchronised load generator that hits the recovering database with a 400-request spike, repeatedly, in perfect lockstep. The spikes are what prevents recovery.

Without jitter — synchronised spikes
t=10s  ████████████████████████  400 requests
t=30s  ████████████████████████  400 requests
t=70s  ████████████████████████  400 requests

With full jitter — the same work, spread
t=0-10s   ▂▃▂▃▃▂▃▂▃▂▃▂▃▂▃▂▃▂▃▂  ~40/sec, smooth
t=10-30s  ▂▃▂▃▂▃▂▃▂▃▂▃▂▃▂▃▂▃▂▃  ~20/sec
t=30-70s  ▂▂▃▂▂▃▂▂▃▂▂▃▂▂▃▂▂▃▂▂  ~10/sec

📌 Backoff without jitter often makes things worse, because it converts continuous pressure into periodic spikes — and a spike is harder to survive than a steady load of the same average.

The implementation

private static final Duration BASE = Duration.ofSeconds(10);
private static final Duration CAP  = Duration.ofMinutes(5);

private void scheduleRetry(Message m, Exception cause) {
    int attempt = Integer.parseInt(m.attributes()
            .getOrDefault(MessageSystemAttributeName.APPROXIMATE_RECEIVE_COUNT, "1"));

    long exponential = Math.min(
            BASE.toSeconds() * (1L << Math.min(attempt - 1, 20)),   // shift, guarded
            CAP.toSeconds());

    int delay = ThreadLocalRandom.current().nextInt((int) exponential + 1);  // full jitter

    log.warn("retrying messageId={} attempt={} delaySeconds={}",
             m.messageId(), attempt, delay, cause);

    sqs.changeMessageVisibility(r -> r
            .queueUrl(queueUrl)
            .receiptHandle(m.receiptHandle())
            .visibilityTimeout(delay));
    // Deliberately no delete. The visibility change IS the retry schedule.
}

Three details:

⚠️ The maximum backoff is bounded by 12 hours, the visibility timeout ceiling (aws-facts.md §2). Backoff schedules that would exceed it silently clamp.

5. The standard-queue complication

Your carefully computed backoff is not the only thing affecting retry timing.

⚠️ On a standard queue with maxReceiveCount > 3, a message received 3 or more times without being deleted may be moved to the back of the queue (aws-facts.md §7).

So from attempt four onward, your ChangeMessageVisibility(80) means "at least 80 seconds", plus however long it takes to work back through the queue. On a queue with a large backlog that can be substantial.

Two practical consequences:

1. Never build a design that depends on precise retry timing. Past attempt three, the interval is a lower bound.
2. Total time-to-DLQ exceeds the sum of your backoff series. Any capacity planning based on "messages quarantine within N minutes" should treat N as optimistic (Module 11).

6. Retry storms and circuit breakers

Backoff smooths a single consumer's retries. It does not stop a fleet from collectively overwhelming a downstream that is genuinely down.

Consider: a downstream is returning 503 for every request. Sixty consumers are polling, receiving, failing and retrying with textbook backoff. The downstream is still receiving thousands of requests per minute — all of which will fail, all of which consume its capacity, and all of which delay its recovery.

The queue is already the correct buffer. The right response is to stop consuming rather than to keep retrying.

final class CircuitBreaker {
    private enum State { CLOSED, OPEN, HALF_OPEN }

    private volatile State state = State.CLOSED;
    private final AtomicInteger consecutiveFailures = new AtomicInteger();
    private volatile Instant openedAt;

    private static final int THRESHOLD = 20;
    private static final Duration COOLDOWN = Duration.ofSeconds(30);

    boolean shouldAttempt() {
        if (state == State.CLOSED) return true;
        if (state == State.OPEN && Instant.now().isAfter(openedAt.plus(COOLDOWN))) {
            state = State.HALF_OPEN;           // let exactly one request through
            return true;
        }
        return state == State.HALF_OPEN;
    }

    void recordSuccess() {
        consecutiveFailures.set(0);
        state = State.CLOSED;
    }

    void recordFailure() {
        if (consecutiveFailures.incrementAndGet() >= THRESHOLD) {
            state = State.OPEN;
            openedAt = Instant.now();
            log.error("circuit OPEN — pausing consumption");
        }
    }
}

In the consumer loop:

while (running) {
    if (!breaker.shouldAttempt()) {
        Thread.sleep(1_000);       // stop polling entirely; let the queue buffer
        continue;
    }
    // ... normal receive / process / delete
}

📌 When a downstream is down, the best thing a consumer can do is nothing. Let the queue do the job it exists for. Messages accumulate, the downstream recovers unimpeded, and the backlog drains afterwards.

⚠️ Not polling means messages stay Visible and the backlog grows — which is correct but will trip a naive queue-depth alarm. Alarm on ApproximateAgeOfOldestMessage and on the circuit breaker's own state, so an open circuit is visible as its own signal rather than as a mysterious backlog (Module 17).

7. The retry budget

maxReceiveCount is how many attempts a message gets before quarantine (Module 11).

Computing total retry time

With backoff, time-to-DLQ is the sum of the backoff series, not maxReceiveCount × visibilityTimeout:

base = 10s, cap = 300s, maxReceiveCount = 6

expected delays (mean of full jitter = half the exponential):
  attempt 1:  ~5s
  attempt 2: ~10s
  attempt 3: ~20s
  attempt 4: ~40s
  attempt 5: ~80s
  ────────────────
  total:    ~155s ≈ 2.6 minutes to DLQ

plus processing time per attempt
plus back-of-queue delay from attempt 4 onward  ← unbounded in principle

Two checks to run on that number:

1. Is it long enough to ride out a typical transient outage? If your database failover takes 90 seconds, a 30-second total retry budget sends every in-flight message to the DLQ during a routine event.
2. Is it far below retention? If not, messages expire instead of quarantining — you lose the message and the evidence (Module 05 §6).

8. Production considerations

Every retry re-runs your consumer, so idempotency is a hard prerequisite for any retry policy (Module 09). A retry policy without idempotency is a duplicate-effect generator.

Track redelivery rate as a first-class metric. NumberOfMessagesReceived significantly exceeding NumberOfMessagesDeleted is the earliest signal of a degrading downstream — earlier than the DLQ filling, earlier than customer complaints (Module 17).

Log the attempt number and the delay on every retry. Without it, "the queue is slow" is untraceable. With it, one grep shows you which messages are struggling and how long they have been.

Backoff interacts with the visibility timeout. ChangeMessageVisibility(delay) replaces the lease, so a delay shorter than your processing time means the message becomes available while you are still handling it. Backoff delays should always exceed normal processing time.

Retrying a permanent failure is pure cost. It multiplies error-log volume, downstream load and time-to-quarantine by maxReceiveCount, for zero chance of success.

A circuit breaker is not optional at scale. Sixty consumers with perfect per-consumer backoff still constitute a fleet-sized load on a failing downstream.

9. Common mistakes

Retrying everything. Why it's wrong: permanent failures consume the entire retry budget and the downstream's capacity for nothing. Instead: classify, and fast-track permanent failures to the DLQ.

Retrying with no backoff. Why it's wrong: the default fixed interval applies maximum pressure to a system that is already failing. Instead: exponential backoff via ChangeMessageVisibility.

Backoff without jitter. Why it's wrong: messages that failed together retry together, converting steady load into synchronised spikes. Instead: full jitter — random(0, delay).

Catching the exception and deleting the message. Why it's wrong: silent data loss. The DLQ stays empty, every metric looks healthy, and the work is simply gone. Instead: do not delete on failure. If the failure is permanent, record it explicitly first.

maxReceiveCount = 1000 "to be safe". Why it's wrong: a poison message loops for days, occupying worker slots and distorting every metric. Instead: 3–5, with a DLQ.

Backoff delay shorter than processing time. Why it's wrong: ChangeMessageVisibility replaces the lease, so the message becomes visible while you are still working on it. Instead: ensure every backoff delay exceeds your p99 processing time.

Using ChangeMessageVisibility(0) as the retry. Why it's wrong: immediate redelivery, maximum pressure, receive count racing to the DLQ. Instead: a computed backoff delay (Module 08 §8).

10. Real-world example

An inventory service. Its database fails over — a routine, planned-for event lasting about 90 seconds. Sixty consumers, ~800 messages/sec, 30-second visibility timeout, maxReceiveCount = 10.

Version 1 — no backoff, no classification

t=0s    failover begins. Every write throws SQLTransientConnectionException.
t=0-30s ~24,000 messages fail. None deleted.
t=30s   ALL 24,000 become visible simultaneously.
        60 consumers grab them and retry — against a database still failing over.
t=60s   Another ~24,000 failures join. Now ~48,000 retrying in lockstep.
t=90s   Database is back — and is immediately hit by ~48,000 queued retries
        plus 800/sec of new traffic.
t=95s   Connection pool exhausted. The database that had just recovered
        falls over again.
t=8min  Eventually stabilises. 3,100 messages reached the DLQ — all of them
        perfectly valid, discarded because maxReceiveCount was exhausted
        during a 90-second event.

The 90-second failover became an 8-minute outage. Not because of the failover — because of the retry behaviour.

Version 2 — backoff with jitter, plus classification

t=0s      failover begins.
t=0-30s   ~24,000 messages fail. Each classified as transient.
          Each gets ChangeMessageVisibility(random(0, 10s)).
t=0-40s   Retries spread smoothly across the window — ~600/sec, not a spike.
          They fail again, and back off further: 0-20s, then 0-40s.
t=90s     Database recovers. Retries are now spread across 0-80s windows,
          so the returning load is a ramp, not a wall.
t=2min    Backlog drained. The database was never re-overwhelmed.
          DLQ: 0 messages.

Same failure, same duration, same message volume. Zero messages lost, no secondary outage.

Version 3 — adding the circuit breaker

After 20 consecutive failures, consumers stop polling entirely for 30 seconds, then let one request through to test.

t=0-15s   ~12,000 messages fail. Circuit opens across the fleet.
t=15-90s  Consumers idle. Queue depth grows to ~60,000. The database
          recovers with ZERO retry traffic.
t=90s     Half-open probe succeeds. Circuit closes.
t=90-160s Backlog drains at full speed against a healthy database.

The queue depth alarm fired — correctly, and it was the only alarm. The team's dashboard showed circuit state alongside depth, so "backlog growing because we deliberately stopped consuming" was immediately distinguishable from "backlog growing because consumers are broken".

🎯 The transferable lesson: the retry policy, not the failure, determined the blast radius. Three versions, one 90-second database event, and outcomes ranging from an 8-minute outage with data in the DLQ to a 70-second blip.

11. Interview questions

🟢 Beginner

How does SQS retry a failed message? It does not, explicitly. If the consumer does not delete the message, the visibility timeout expires and the message becomes available again. Not deleting is the retry.

What is maxReceiveCount? The number of times a message may be received before SQS moves it to the dead-letter queue.

🟡 Intermediate

Where does the retry delay come from? Can you change it? It is the visibility timeout — nothing else schedules the redelivery. You change it per message with ChangeMessageVisibility, which is how backoff is implemented, since SQS has no backoff setting.

What is exponential backoff and why add jitter? Backoff means waiting progressively longer after each failure, typically doubling. Jitter randomises each delay so that messages which failed together do not retry together. Without it, backoff converts steady load into synchronised spikes against a recovering system — which is often worse than no backoff at all.

Why is deleting a message in a catch block a bug? It is silent data loss. The message is gone, the DLQ stays empty, and every metric looks healthy — so nobody discovers it until someone asks why an order was never fulfilled.

🔴 Advanced

How would you implement backoff on SQS, which has no backoff feature?

Use ChangeMessageVisibility on the failure path. Read ApproximateReceiveCount from the message as the attempt number — SQS already tracks it, so you need no state of your own. Compute min(base × 2^(attempt-1), cap), apply full jitter with random(0, delay), and set that as the new visibility timeout without deleting the message. The message is hidden for the delay and then redelivered.

Three constraints to respect: the delay must exceed your normal processing time, because ChangeMessageVisibility replaces the lease rather than adding to it; the total is bounded by the 12-hour visibility ceiling; and on standard queues, from the third receive onward the message may be moved to the back of the queue, so your delay becomes a lower bound.

How do you distinguish a transient failure from a permanent one? By the nature of the error rather than its severity. Permanent failures are determined by the message content or by durable state — malformed JSON, a schema violation, a referenced entity that does not exist, an HTTP 400. Transient failures are conditions that may pass — timeouts, HTTP 503, 429, connection refused, a database failing over. The AWS SDK exposes SdkException.retryable() directly. For unknown exceptions, default to retryable: maxReceiveCount bounds the cost, whereas wrongly classifying a transient error as permanent discards work that would have succeeded.

Your consumers are retrying so aggressively they are preventing the downstream from recovering. What do you do?

Immediate, in order of speed:

1. Open a circuit breaker — stop consuming entirely. The queue is the buffer; let it buffer. This is the fastest way to give the downstream room.
2. Scale consumers down. Fewer pollers means less pressure.
3. If neither is available quickly, raise the visibility timeout to space retries out — crude, but it works from the console and needs no deploy.

Structural, afterwards:

• Exponential backoff with jitter, so retries spread rather than spike.
• Classification, so permanent failures stop consuming the downstream's capacity.
• A circuit breaker as standing configuration, not an incident response.
• Alarm on circuit state, so "backlog growing because we stopped consuming" is distinguishable from "backlog growing because consumers are broken".

The framing worth stating explicitly: when a downstream is down, a consumer's best action is to do nothing. Retrying harder is the instinct and it is precisely backwards.

How does backoff change your time-to-DLQ calculation? Without backoff it is maxReceiveCount × visibilityTimeout. With backoff it is the sum of the backoff series — with full jitter, the expectation is roughly half the sum of the exponential series. Both are lower bounds on standard queues, because a message received three or more times may be moved to the back of the queue. The resulting number must be long enough to survive a routine transient outage and short enough to stay well below the retention period.

⚫ System design

Design a retry policy for a pipeline calling a third-party API with a 500 req/s rate limit.

The rate limit changes the shape of the problem: throttling is a transient failure that is caused by your own retry volume, so a naive policy is self-reinforcing.

1. Never exceed the limit in the first place. A client-side rate limiter (token bucket at, say, 450 req/s across the fleet) is the primary control. Retries are for what gets through anyway.

2. Classify precisely, because the third party's status codes carry most of the information:

3. Honour Retry-After when supplied. Set ChangeMessageVisibility to that value rather than to your computed backoff. The provider is telling you the answer.

4. Backoff with full jitter, base around 5 s, cap around 5 min. Jitter matters more than usual here: synchronised retries against a rate limiter guarantee another round of 429s.

5. Circuit-break on sustained 429s. If you are being throttled consistently, your steady-state rate is too high and retrying cannot fix it. Stop, let the queue buffer, and alarm — this is a capacity problem, not a transient one.

6. Control concurrency at the consumer, not just the retry path. Consumer count × per-consumer concurrency must stay under the limit (Module 14). If the autoscaler can scale you past 500 req/s, no retry policy will save you.

7. Size maxReceiveCount against their outage profile. If the provider's typical incident is 10 minutes, a retry budget summing to 3 minutes sends good messages to the DLQ routinely. Either raise the budget or accept a DLQ replay as part of the runbook (Module 11).

Follow-ups to expect: "What if the rate limit is per-tenant?" — then the limiter and possibly the queues must be per-tenant too, which is a bulkhead (Module 24). "How do you avoid the DLQ filling during a long provider outage?" — circuit-break so receive counts stop incrementing; a message that is never received never approaches maxReceiveCount.

12. Summary

• SQS has no retry API. A retry is a visibility timeout expiring on an undeleted message.
• The default interval is the visibility timeout, fixed, and identical for every kind of failure.
• Classify first: transient retries, permanent goes straight to the DLQ with a record.
• Backoff is built by you with ChangeMessageVisibility, driven by ApproximateReceiveCount.
• Full jitter is mandatory. Backoff without it converts steady load into synchronised spikes.
• A circuit breaker beats retrying when a downstream is genuinely down — let the queue buffer.
• Time-to-DLQ is the sum of the backoff series, and a lower bound on standard queues.
• Backoff delays must exceed processing time, since ChangeMessageVisibility replaces the lease.
• Never delete on failure. That is silent data loss with a healthy-looking dashboard.
• Retries require idempotency. Without it, a retry policy is a duplicate-effect generator.

← Previous: 09 — Delivery Semantics and Idempotency · Index: Course home · Next: 11 — Dead Letter Queues →