Scaling Consumers

How to add capacity, and — more importantly — how to recognise when adding capacity will not help. The answer to "the queue is growing, add more consumers" is very often no.

1. What you will learn

• Distinguish the three levers: more consumers, more threads per consumer, more messages per call
• Explain why more consumers eventually stops improving throughput
• Design queue-depth-based autoscaling that is stable rather than oscillating
• Identify the real bottleneck before scaling anything
• Apply backpressure so a consumer fleet does not destroy its downstream

2. Why this concept exists

• SQS itself is effectively unlimited on standard queues — so the queue is almost never the bottleneck
• The bottleneck is your consumers, your database, your downstream API, or a FIFO group
• Scaling the wrong layer is the most common and most expensive response to a growing backlog

3. Beginner explanation

• More workers means more messages processed per second — up to a point
• That point is wherever the next constraint is: usually the database
• Past that point, extra workers just queue up behind each other

4. How it actually works

• Three independent levers: horizontal (more consumer processes) · vertical concurrency (more threads per process) · batch size (more messages per API call)
• Competing consumers pattern: every consumer polls the same queue; SQS distributes naturally, no partition assignment needed (unlike Kafka)
• Where scaling stops helping:
• downstream saturation (DB connection pool, third-party rate limit)
• the in-flight quota (~120,000)
• FIFO: concurrency is capped by the number of active message groups
• Lambda: the 1,250 per-ESM ceiling or account concurrency
• Autoscaling signals: ApproximateNumberOfMessagesVisible (depth) and ApproximateAgeOfOldestMessage (latency). Prefer the backlog-per-consumer ratio over raw depth
• Target-tracking on backlog per instance: acceptableLatency ÷ avgProcessingTime = messages one instance can clear → target that
• Backpressure: a bounded internal work queue plus a blocking submit is what stops a consumer from reading faster than it can process

5. Diagram

• Diagram 33 — consumer fleet / worker pool
• Diagram 34 — the autoscaling control loop (metric → alarm → scaling policy → capacity → metric)
• Diagram 35 — backpressure and the downstream bottleneck

6. Step-by-step flow

• Measure: arrival rate, processing time p50/p99, current concurrency
• Compute required concurrency from Little's Law (Module 15)
• Identify the binding constraint — consumer CPU, downstream, or quota
• Scale the constrained layer, not reflexively the consumer count
• Set a scaling policy on backlog-per-instance
• Set a maximum that protects the downstream
• Verify with the oldest-message-age metric, not with depth alone

7. Configuration

• ECS service autoscaling / EKS HPA or KEDA on a CloudWatch queue metric
• Consumer-side thread pool size and bounded work queue capacity
• MaxNumberOfMessages per call
• Lambda reserved and maximum concurrency (Module 19)
• Scale-in cooldowns long enough to avoid killing consumers mid-message

8. Production considerations

• Scaling in is the dangerous direction — terminate a consumer holding 10 in-flight messages and they all wait out the visibility timeout. Graceful shutdown (Module 22) is mandatory
• Database connection pools are usually the real ceiling. 200 consumers × 10 connections = 2,000 connections against a database that accepts 500
• Oscillation: aggressive scale-out plus a short cooldown produces a fleet that thrashes. Asymmetric policies — scale out fast, scale in slow
• Scale-to-zero saves money on idle queues but adds cold-start latency to the first message
• A FIFO queue with 4 message groups cannot use more than 4 concurrent consumers, no matter what the autoscaler does
• Cost: every idle consumer still long-polls, so an over-scaled fleet costs API requests as well as compute

9. Common mistakes

• Scaling consumers when the database is the bottleneck. Why → more contention, more timeouts, worse throughput. Instead → find the constraint first
• Autoscaling on raw queue depth. Why → depth is meaningless without processing time; 1,000 messages is trivial at 5 ms and catastrophic at 5 s. Instead → backlog per instance, or oldest-message age
• Symmetric scale-out/scale-in policies. Why → oscillation. Instead → out fast, in slow
• No graceful shutdown. Why → every deploy generates a burst of redeliveries. Instead → drain in-flight work on SIGTERM
• Ignoring the in-flight quota. Why → the fleet stalls at ~120,000 with a confusing symptom. Instead → alarm on NotVisible
• Unbounded internal queues. Why → the consumer OOMs instead of applying backpressure. Instead → bounded queue, blocking submit

10. Real-world example

• Black Friday: arrivals go from 500/s to 5,000/s. Walk the full response — what the autoscaler does, where it stops helping, and how the queue's buffering bought the time to fix the real constraint

11. Interview questions

• 🟢 How do you increase SQS consumer throughput?
• 🟡 What is the competing-consumers pattern?
• 🟡 What metric would you autoscale consumers on, and why not queue depth?
• 🔴 Why does adding more consumers eventually stop improving throughput?
• 🔴 Your queue is backing up but consumer CPU is at 15%. What is happening?
• 🔴 How does FIFO limit consumer concurrency?
• ⚫ Design an autoscaling strategy for a queue with 100× diurnal variation and a 30-second latency SLO.

12. Summary

• Three levers: consumer count, threads per consumer, messages per call
• SQS is rarely the bottleneck — the downstream usually is
• Autoscale on backlog per instance or oldest-message age, never raw depth
• Scale out fast, scale in slowly, and shut down gracefully
• FIFO concurrency is capped by active message groups

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• competing consumers
• worker pool
• backlog per instance
• target tracking
• graceful shutdown
• backpressure

Diagrams to build:

• Diagram 33 — consumer scaling
• Diagram 34 — queue-depth autoscaling loop
• Diagram 35 — backpressure

Code samples:

• A bounded worker pool with a blocking submit policy
• In-flight accounting so shutdown knows what to drain
• An ECS target-tracking scaling policy on a backlog-per-task custom metric

Mandatory "why?" answers:

• Why does adding consumers stop improving throughput?
• Why is queue depth a bad autoscaling signal on its own?
• Why is scaling in more dangerous than scaling out?

Facts to pull from _reference/aws-facts.md: in-flight ~120,000, FIFO concurrency = active groups, Lambda 1,250 per ESM

← Previous: 13 — Batch Processing · Index: Course home · Next: 15 — Performance and Throughput →