Troubleshooting

Eighteen real production scenarios, each in the same shape: SYMPTOM → POSSIBLE CAUSE → HOW TO VERIFY → FIX → PREVENTION. Written to be usable at 3 a.m. with an alarm going off, not read cover to cover.

1. What you will learn

• Diagnose the 18 most common SQS production problems from their symptoms
• Use metric combinations to narrow a cause before touching anything
• Apply the right fix, and know which fixes make things worse
• Build the preventions into the design so the scenario does not recur

2. Why this concept exists

• SQS failures present through a small set of symptoms — a growing queue, duplicate work, a filling DLQ — but each has several possible causes
• A structured method beats guessing, and guessing under pressure is how a small incident becomes a large one

3. Beginner explanation

• When something goes wrong, the queue's metrics usually tell you which part of the system is at fault
• Read them in combination, not one at a time

4. How it actually works

• The general method: identify the symptom → check the metric triad (Visible / NotVisible / Age) → check the consumer → check the downstream → check the quotas → confirm the cause before changing anything
• The 18 scenarios, each SYMPTOM → CAUSE → VERIFY → FIX → PREVENT:
• 1. Queue keeps growing
• 2. Messages processed twice
• 3. Messages disappear temporarily
• 4. Messages reappear after apparently successful processing
• 5. DLQ suddenly grows
• 6. Consumer crashes repeatedly
• 7. Unexpectedly high SQS API costs
• 8. Very high NumberOfEmptyReceives
• 9. FIFO throughput far below expectations
• 10. One MessageGroupId blocking all progress
• 11. Lambda concurrency behaving unexpectedly
• 12. Processing exceeding the visibility timeout
• 13. Deployments causing duplicate processing
• 14. Large backlog after a traffic spike
• 15. Consumer CPU-bound
• 16. Consumer I/O-bound
• 17. Database as the real bottleneck
• 18. SQS healthy, application slow
• Plus: messages silently lost (retention expiry), OverLimit errors (in-flight quota), AccessDenied after enabling KMS, receipt handle invalid

5. Diagram

• The master triage flowchart from symptom to cause
• A metric-signature table: what each combination of Visible/NotVisible/Age/Received/Deleted means

6. Step-by-step flow

• Observe: which metric is abnormal, and since when?
• Correlate: what deployed, scaled or changed at that time?
• Narrow: use the metric signature table to shortlist causes
• Verify: confirm the cause with a specific check before acting
• Act: apply the fix, with an explicit rollback
• Confirm: watch oldest-message age recover
• Prevent: add the alarm, the config change or the design change that stops a repeat

7. Configuration

• Emergency levers: raise visibility timeout, lower maxReceiveCount, scale consumers, pause the event source mapping, rate-limit a redrive
• ⚠️ PurgeQueue is irreversible and deletes everything. It is almost never the right answer during an incident

8. Production considerations

• Write the runbook before the incident. During one, you follow it; you do not author it
• Change one thing at a time — two simultaneous changes make the recovery uninterpretable
• Never redrive a DLQ before diagnosing — you will repeat the failure at higher receive counts
• Never purge to make an alarm stop. That is data loss chosen under stress
• Capture evidence first: sample a DLQ message, record metric snapshots — both are gone once you fix it
• The most common root cause in practice is not SQS at all: it is the downstream database

9. Common mistakes

• Scaling consumers as the reflex response. Why → it usually is not the bottleneck and it adds downstream pressure. Instead → diagnose first
• Purging the queue. Why → irreversible data loss. Instead → almost anything else
• Redriving without a fix. Why → the same failure, one receive-count higher. Instead → fix, then redrive at a limited rate
• Raising visibility timeout to 'stop the duplicates'. Why → it masks a processing-time problem and slows crash recovery. Instead → find why processing is slow
• Changing several settings at once. Why → no signal about what worked. Instead → one change, observe, next

10. Real-world example

• Full walkthrough of the hardest case: "consumers healthy, CPU low, no errors, queue growing" → connection pool exhaustion in a downstream service, found from the metric signature alone

11. Interview questions

• 🟡 Your SQS queue is growing. Walk me through your diagnosis.
• 🟡 Messages are being processed twice. What do you check?
• 🔴 The queue grows while consumers show low CPU and no errors. What is happening?
• 🔴 The DLQ jumped from 0 to 50,000 in five minutes. What do you do first?
• 🔴 FIFO throughput is one tenth of what you expected. Diagnose.
• ⚫ Design the runbook and alarm set that would have caught each of these before a customer did.

12. Summary

• Read Visible, NotVisible and Age together — the combination names the cause
• Verify before acting; change one thing at a time
• Never purge; never redrive before fixing
• The bottleneck is usually downstream, not SQS
• Every incident ends with a prevention, or it recurs

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• triage
• metric signature
• runbook
• PurgeQueue
• OverLimit

Diagrams to build:

• Master triage flowchart
• Metric signature table

Code samples:

• A diagnostic script: queue attributes, metric snapshot, DLQ sample, in one command
• A safe DLQ inspector that reads without consuming

Mandatory "why?" answers:

• Why can a queue grow while consumers look perfectly healthy?
• Why is purging almost never the right incident response?

← Previous: 25 — Production Architecture and System Design · Index: Course home · Next: 27 — SQS vs Kafka vs RabbitMQ →