Production Architecture and System Design

The synthesis module: a repeatable methodology for designing an SQS-based system, and a reference architecture you can adapt. This is what you reach for in a design review — or in a system-design interview.

1. What you will learn

• Run the 20-question design methodology end to end
• Translate requirements into concrete queue configuration
• Produce a reference architecture with monitoring, security and failure handling built in
• Design multi-queue topologies: priority lanes, per-tenant isolation, staged pipelines
• Know when SQS is the wrong answer

2. Why this concept exists

• Every SQS design decision — queue type, visibility timeout, batch size, retry budget, consumer platform — is downstream of a requirement
• A methodology makes those requirements explicit before they become incidents

3. Beginner explanation

• Before you configure anything, answer some questions about what the system must do
• The answers determine the settings
• Writing them down is most of the design

4. How it actually works

• The 20 questions: what produces · what consumes · is ordering required · FIFO or standard · expected traffic · peak traffic · average message size · processing duration · acceptable latency · failure behaviour · retry budget · DLQ needed · how idempotency is achieved · how consumers scale · how backlog is monitored · behaviour during a downstream outage · security requirements · cost constraints · behaviour during deployment · behaviour during consumer crash
• Requirement → configuration mapping: ordering → queue type · processing duration p99 → visibility timeout · latency SLO → consumer concurrency (Module 15) · failure tolerance → maxReceiveCount + DLQ · traffic shape → autoscaling policy · data sensitivity → encryption and PII policy
• Reference architecture: API → (outbox) → SQS → autoscaled consumers → downstream, with DLQ, CloudWatch alarms, correlation ids, IAM roles and encryption on every edge
• Topology patterns:
• Priority lanes — separate high/low queues with separate consumer fleets (SQS has no priority field)
• Per-tenant queues — bulkheads; or fair queues when the count would explode
• Staged pipeline — queue between each processing stage, each independently scalable and independently retryable
• Fanout — SNS to many queues (Module 20)
• Claim-check — large payload in S3, reference in the message
• Multi-Region: SQS is Regional. Options are active/passive with producer failover, or dual-write with deduplication. Neither is free; state the RPO/RTO
• When SQS is the wrong tool: replay needed (Kafka) · multiple independent consumer groups over the same stream (Kafka) · sub-millisecond latency (in-memory) · complex routing topologies (RabbitMQ/EventBridge) · >14-day retention (Kafka/S3) · strict global ordering at high volume (nothing does this well)

5. Diagram

• Diagram 40 — the production reference architecture, fully annotated
• Priority-lane topology
• Staged pipeline with a DLQ per stage
• Multi-Region active/passive

6. Step-by-step flow

• Answer the 20 questions in writing
• Choose the queue type and justify it
• Compute concurrency and instance count (Module 15)
• Set visibility timeout from processing p99 (Module 08)
• Set maxReceiveCount and attach a DLQ (Module 11)
• Choose the consumer platform (Modules 19, 22, 23)
• Define the autoscaling policy and its ceiling (Module 14)
• Define alarms and dashboards (Module 17)
• Define IAM and encryption (Module 16)
• Model the cost (Module 18)
• Write the runbook and the production checklist

7. Configuration

• A worked configuration table: every attribute, its chosen value, and the requirement that produced it
• IaC templates for queue + DLQ + alarms + IAM as one unit

8. Production considerations

• Queues are cheap; misconfigured queues are not. Prefer more queues with clear ownership over one queue with mixed traffic
• Every queue needs an owner, a runbook and an alarm before it goes to production
• Design the failure path first — the happy path is the easy part and the one everyone reviews
• Capacity plan for peak, and state the drain time you accept if you do not
• Deployment is a designed behaviour, not an accident (Modules 22, 23)
• Downstream outage behaviour is the most commonly skipped question: does the queue absorb it, and for how long before retention bites?

9. Common mistakes

• Designing the happy path only. Why → production is mostly the failure path. Instead → design failure first
• One queue for mixed workloads. Why → no isolation, no independent tuning, no priority. Instead → separate queues
• Skipping the capacity math. Why → you discover the ceiling during the incident. Instead → do the arithmetic (Module 15)
• No runbook. Why → the DLQ alarm fires at 3 a.m. and nobody knows the procedure. Instead → write it with the design
• Using SQS where a log is needed. Why → no replay, 14-day cap, destructive consumption. Instead → Kafka (Module 27)
• Ignoring the deployment story. Why → duplicate processing on every release. Instead → design it

10. Real-world example

• A complete design review for an order-processing platform: the 20 answers, the resulting configuration table, the architecture diagram, the alarms, the runbook, and the cost model

11. Interview questions

• 🟡 What questions would you ask before designing an SQS-based system?
• 🔴 How do you decide the visibility timeout, maxReceiveCount and consumer count for a new service?
• 🔴 How would you implement message priority in SQS?
• 🔴 How do you handle a downstream outage lasting six hours?
• ⚫ Design a system processing 1,000,000 events per minute. Full architecture, with numbers.
• ⚫ Design an order-processing platform with three independent consumers, strict per-customer ordering, and a 30-second SLO.
• ⚫ When would you not use SQS? Give three concrete cases.

12. Summary

• Twenty questions, answered in writing, produce the configuration
• Requirement → configuration is a mapping you can state explicitly
• Separate queues for separate workloads; SQS has no priority field
• Design the failure path, the deployment path and the outage path deliberately
• Know the three or four cases where SQS is the wrong tool

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• priority lanes
• claim-check pattern
• staged pipeline
• reference architecture
• RPO
• RTO

Diagrams to build:

• Diagram 40 — production reference architecture
• Priority lanes
• Staged pipeline
• Multi-Region

Code samples:

• Complete IaC module: queue + DLQ + redrive + alarms + IAM roles + encryption
• The configuration table generated from the 20 answers

Mandatory "why?" answers:

• Why does SQS have no priority field, and what do you do instead?
• Why design the failure path before the happy path?

← Previous: 24 — Distributed Systems Concepts · Index: Course home · Next: 26 — Troubleshooting →