Performance and Throughput

The arithmetic. Given an arrival rate, a processing time and a latency target, how many consumers do you need — and what does the queue do when you get it wrong? This module replaces intuition with three formulas you can use in a design review.

1. What you will learn

• Apply Little's Law to size a consumer fleet
• Predict queue depth from the gap between arrival rate and service rate
• Translate a backlog into a drain time and a customer-visible latency
• Explain why '10 consumers × 10 messages per call' is not '100 messages per second'
• Find the bottleneck systematically instead of guessing

2. Why this concept exists

• "Add more consumers" is a guess. "We need 42 workers to hold p99 under 30 seconds at 5,000 msg/s" is an engineering statement
• The difference is three formulas, and interviewers at senior level expect you to reach for them

3. Beginner explanation

• If messages arrive faster than you process them, the queue grows
• How fast it grows is just the difference between the two rates
• How many workers you need is how much work arrives divided by how much one worker does

4. How it actually works

• Little's Law: L = λ × W. Concurrency required = arrival rate × processing time
• 5,000 msg/s × 0.2 s = 1,000 concurrent message slots
• Slots, not machines: 1,000 slots ÷ 50 threads per instance = 20 instances
• Queue growth: dDepth/dt = λ_arrival − λ_service. Constant, so backlog grows linearly during a deficit
• Drain time: backlog ÷ (λ_service − λ_arrival) once you are ahead
• Latency of a message ≈ queueDepth ÷ serviceRate + processingTime — queueing delay dominates under backlog
• Why 10 × 10 ≠ 100/s: the 10 messages arrive in one round trip. The rate depends on how long the round trip plus processing takes, not on the batch size. throughput = batchSize ÷ cycleTime × consumers
• 10 consumers × 10 messages, 2-second cycle → 50 msg/s, not 100
• Utilisation and the knee: as utilisation ρ → 1, queueing delay → ∞. Plan for 70–80%, not 99%
• Where latency actually goes: send → storage → poll interval → in-process queueing → processing → delete. Most designs obsess over the last one

5. Diagram

• Diagram 36 — arrival rate vs service rate and the resulting depth curve
• A latency-budget breakdown showing where the milliseconds go

6. Step-by-step flow

• State assumptions explicitly: λ, processing time p50 and p99, target latency
• Compute required concurrency with Little's Law using p99, not the mean
• Divide by per-instance concurrency to get instance count
• Add headroom for the target utilisation (÷ 0.75)
• Check every quota: in-flight, FIFO TPS, Lambda concurrency, DB connections
• Compute worst-case backlog for the peak and confirm drain time is acceptable
• Validate with a load test; the model is a starting point, not an answer

7. Configuration

• MaxNumberOfMessages, WaitTimeSeconds, consumer thread count, instance count
• Visibility timeout sized from the same p99 you used in the math (Module 08)
• Relevant ceilings → aws-facts.md §3, §4

8. Production considerations

• Use p99, not the mean. Sizing on the mean guarantees your p99 messages overrun their visibility timeout
• The model assumes a stable downstream. When the database slows by 3×, required concurrency triples — and your connection pool does not
• Queueing delay dominates under backlog. A 100,000-message backlog at 1,000 msg/s means the newest message waits 100 seconds no matter how fast processing is
• Peak vs average: size for peak, or accept a known drain time. State which you chose
• Cost is proportional to API calls, not messages (Module 18) — the throughput design and the cost design are the same design

9. Common mistakes

• Sizing from average processing time. Why → the tail is what builds backlog. Instead → p99
• Believing batch size multiplies throughput linearly. Why → throughput is bounded by cycle time. Instead → batchSize ÷ cycleTime × consumers
• Ignoring in-process queueing. Why → visibility timers started at receive, not at processing start. Instead → count the whole in-process wait
• Planning for 100% utilisation. Why → queueing delay explodes at the knee. Instead → 70–80%
• Measuring throughput on an empty queue. Why → you measured the poll loop, not the system. Instead → load test with a realistic backlog

10. Real-world example

• A telecom event pipeline: 5,000 msg/s sustained, 12,000 msg/s peak, 200 ms p50 / 800 ms p99 processing, 30-second latency SLO. Full worked sizing, including what breaks at each quota

11. Interview questions

• 🟢 If messages arrive faster than they are processed, what happens?
• 🟡 How many consumers do you need for 1,000 msg/s at 100 ms each?
• 🟡 Does doubling batch size double throughput? Explain.
• 🔴 Derive the concurrency needed for 5,000 msg/s with a 200 ms p99 and justify your headroom.
• 🔴 A queue has a 500,000-message backlog. How long to drain, and what is the p99 latency meanwhile?
• 🔴 Why does queueing delay grow non-linearly as utilisation approaches 100%?
• ⚫ Design a system for 1,000,000 events per minute. Show the math and name every quota you would hit.

12. Summary

• concurrency = arrivalRate × processingTime (Little's Law), computed on p99
• depthGrowth = arrivalRate − serviceRate; drainTime = backlog ÷ surplus
• throughput = batchSize ÷ cycleTime × consumers — batch size alone multiplies nothing
• Target 70–80% utilisation; the knee is real
• State assumptions, then load test

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• Little's Law
• arrival rate
• service rate
• queue depth
• drain time
• utilisation
• cycle time

Diagrams to build:

• Diagram 36 — arrival vs service rate and queue depth

Code samples:

• A capacity calculator (input λ, processing time, target latency → instances, with quota checks)
• A load-test harness that produces at a controlled rate and reports achieved throughput and p99

Mandatory "why?" answers:

• Why does 10 consumers × 10 messages per call not equal 100 messages/sec?
• Why does queueing delay explode as utilisation approaches 100%?
• Why size from p99 rather than the mean?

Facts to pull from _reference/aws-facts.md: standard unlimited throughput, FIFO TPS table, in-flight quota, Lambda ceilings

← Previous: 14 — Scaling Consumers · Index: Course home · Next: 16 — Security →