Course Design — AWS SQS
The design document behind the course. It exists so that 32 modules written over many sessions stay consistent, nothing in the spec gets dropped, and every diagram/lab/code sample has an owner.
Spec: ../aws-sqs-prompt.md · Template: _reference/chapter-template.md · Facts: _reference/aws-facts.md
1. Design decisions
Why 32 modules and not the spec's 28 directories
The spec gives two lists that don't fully agree: a 25-entry LEVEL 0–24 curriculum and a 28-entry
directory tree. Neither covers everything the spec's body sections demand. Four topics that the
spec treats at length had no home in the directory tree:
| Spec section with no directory | Resolution |
|---|---|
LEVEL 5 — Sending and Receiving Messages | New module 04-sending-and-receiving |
DELIVERY SEMANTICS (at-least-once, idempotency keys, transaction boundaries) | New module 09-delivery-semantics-and-idempotency |
COST | New module 18-cost |
DISTRIBUTED SYSTEM CONCEPTS | New module 24-distributed-systems-concepts |
The spec explicitly permits this: "You may adjust the number of modules if necessary, but maintain a logical progression." Every directory name from the spec's tree is preserved; the four above are inserted and the sequence renumbered. §2 below is the full traceability table proving nothing was dropped.
Ordering change: lifecycle before queue types
The spec's directory tree puts 04-standard-queue / 05-fifo-queue before 06-message-lifecycle.
This course reverses that. Standard-vs-FIFO is a comparison, and a comparison is meaningless
before the reader knows what a message's life looks like. So:
03 architecture → 04 the API surface → 05 the lifecycle → 06 standard → 07 FIFOBy module 06 the reader already knows what "at-least-once" and "in-flight" mean, so the standard/FIFO contrast lands as a genuine trade-off rather than a feature list.
Ordering change: semantics before retries
09-delivery-semantics-and-idempotency sits before 10-retries and 11-dead-letter-queues.
Retry design is unsafe to teach before the reader accepts that duplicate execution is guaranteed,
not exceptional. Idempotency is a precondition for retries, not a footnote after them.
Three-tier depth
Every hard topic is written at three explicit levels (🟢 what / 🟡 how / 🔴 what breaks). A reader preparing for an interview next week reads levels 1–2. A reader on call reads level 3. The level-3 material is the reason this is a course and not a tutorial.
2. Traceability — spec → module
Proof that every numbered LEVEL and every named spec section has a home.
Spec LEVEL | Module |
|---|---|
| 0 Prerequisites | 00 |
| 1 Messaging Fundamentals | 01 |
| 2 SQS Fundamentals | 02 |
| 3 How SQS Actually Works | 03 |
| 4 Queue Types | 06, 07 |
| 5 Sending and Receiving | 04 |
| 6 Message Lifecycle | 05 |
| 7 Reliability and Failure Handling | 09, 10 |
| 8 Visibility Timeout | 08 |
| 9 Dead Letter Queues | 11 |
| 10 Ordering and FIFO | 07 |
| 11 Scaling Consumers | 14 |
| 12 Performance and Throughput | 15 |
| 13 Long Polling | 12 |
| 14 Batch Operations | 13 |
| 15 Security | 16 |
| 16 Monitoring and Observability | 17 |
| 17 AWS Integrations | 19, 20, 21, 22 |
| 18 Advanced Architecture | 24, 25 |
| 19 Production Design | 25 |
| 20 Troubleshooting | 26 |
| 21 SQS vs Other Messaging | 27 |
| 22 Corporate Case Studies | 28 |
| 23 Hands-on Projects | 29, 30 |
| 24 Interview and System Design | 31 |
| Named spec section | Home |
|---|---|
CRITICAL CONCEPT: BATCH VS MESSAGE | 13 (§3 is the spec's highest-priority clarity requirement) |
STANDARD QUEUE | 06 |
FIFO QUEUE | 07 |
MESSAGE LIFECYCLE | 05 |
VISIBILITY TIMEOUT | 08 |
DELIVERY SEMANTICS | 09 |
RETRY AND DLQ | 10, 11 |
PERFORMANCE | 15 |
LONG POLLING | 12 |
AWS SDK EXAMPLES | 04 (introduced), then per-topic |
JAVA / SPRING BOOT | 23 |
AWS LAMBDA + SQS | 19 |
ECS / EKS / EC2 CONSUMERS | 22 |
MONITORING | 17 |
OBSERVABILITY | 17 |
SECURITY | 16 |
SQS + SNS | 20 |
OTHER AWS INTEGRATIONS | 21 |
DISTRIBUTED SYSTEM CONCEPTS | 24 |
SQS VS KAFKA VS RABBITMQ | 27 |
CORPORATE CASE STUDIES | 28 |
REAL PRODUCTION FAILURE SCENARIOS | 26 |
SYSTEM DESIGN SECTION | 25 (methodology), 31 (practice) |
NUMERICAL EXAMPLES | 15 (queueing math), reused in 14, 19, 25 |
REVISION CHEAT SHEETS | revision/ (9 files) |
HANDS-ON LABS | 29 (17 labs) |
CAPSTONE PROJECT | 30 |
INTERVIEW PREPARATION | 31 + §11 of every module |
COMMON MISCONCEPTIONS | misconceptions.md |
AWS CLI | 04 (basics), then per-topic; consolidated in revision/sqs-api-cheatsheet.md |
COST | 18 |
GLOSSARY | glossary.md |
PRODUCTION READINESS CHECKLIST | production-checklist.md |
CURRENT AWS DOCUMENTATION | _reference/aws-facts.md |
3. Module map
| # | Module | Level | Owns these concepts (defined here, linked elsewhere) |
|---|---|---|---|
| 00 | prerequisites | 🟢 | AWS account, IAM basics, CLI setup, Java 21 + SDK v2 setup, sync vs async |
| 01 | messaging-fundamentals | 🟢 | Coupling, buffering, load levelling, queue, producer, consumer, push vs pull |
| 02 | sqs-introduction | 🟢 | What SQS is, what it is not, managed-service boundary, first mental model |
| 03 | sqs-architecture | 🟡 | Distributed storage, redundant servers, why metrics say "Approximate", no broker to manage |
| 04 | sending-and-receiving | 🟢 | SendMessage, ReceiveMessage, DeleteMessage, MessageId vs ReceiptHandle, message attributes, SDK v2 client, CLI |
| 05 | message-lifecycle | 🟡 | The full state machine: stored → available → in-flight → deleted/redelivered/expired/DLQ |
| 06 | standard-queues | 🟡 | At-least-once, best-effort ordering, unlimited throughput, why duplicates occur |
| 07 | fifo-queues | 🔴 | MessageGroupId, MessageDeduplicationId, dedup interval, partitions, ordering scope, head-of-line blocking |
| 08 | visibility-timeout | 🔴 | Timeout semantics, ChangeMessageVisibility, heartbeating, the race condition |
| 09 | delivery-semantics-and-idempotency | 🔴 | At-least-once vs at-most-once vs exactly-once, idempotency keys, transaction boundaries |
| 10 | retries | 🔴 | Transient vs permanent errors, backoff, jitter, retry storms, poison messages |
| 11 | dead-letter-queues | 🔴 | maxReceiveCount, redrive policy, redrive allow policy, DLQ redrive, replay runbook |
| 12 | long-polling | 🟡 | WaitTimeSeconds, empty receives, short-polling sampling behaviour, cost and latency |
| 13 | batch-processing | 🟡 | Batch vs message vs API call vs consumer vs worker, batch APIs, partial failure |
| 14 | scaling-consumers | 🔴 | Horizontal scaling, worker pools, thread pools, queue-depth autoscaling, backpressure |
| 15 | performance-and-throughput | 🔴 | Little's Law, arrival vs service rate, concurrency math, bottleneck analysis |
| 16 | security | 🔴 | IAM identity vs resource policy, queue policies, SSE-SQS vs SSE-KMS, VPC endpoints, cross-account |
| 17 | monitoring-and-observability | 🔴 | Every CloudWatch metric, alarm design, correlation IDs, tracing, structured logs |
| 18 | cost | 🟡 | Request-based billing, 64 KB chunking, batching/long-polling savings, cost modelling |
| 19 | lambda | 🔴 | Event source mapping, who polls, scaling, ReportBatchItemFailures, when not to use Lambda |
| 20 | sns-integration | 🔴 | Fanout, filter policies, raw message delivery, per-subscriber DLQs |
| 21 | event-sources | 🔴 | S3 → SQS, EventBridge → SQS, API Gateway → SQS, Step Functions |
| 22 | ecs-eks | 🔴 | Self-managed consumers, graceful shutdown, SIGTERM, HPA/KEDA on queue depth |
| 23 | spring-boot | 🔴 | Spring Cloud AWS, @SqsListener, concurrency, error handling — and what is Spring, not SQS |
| 24 | distributed-systems-concepts | ⚫ | Which ideas are SQS-specific vs universal; eventual consistency, fault isolation |
| 25 | production-architecture | ⚫ | The 20-question design methodology, reference architecture, multi-service patterns |
| 26 | troubleshooting | ⚫ | 18 failure scenarios as SYMPTOM → CAUSE → VERIFY → FIX → PREVENT |
| 27 | sqs-vs-kafka-rabbitmq | ⚫ | Architectural comparison; partitions/offsets, exchanges/bindings; selection criteria |
| 28 | case-studies | ⚫ | 6 corporate scenarios end to end |
| 29 | hands-on-labs | 🟡 | 17 progressive labs |
| 30 | capstone | ⚫ | Order Processing Platform, production-grade |
| 31 | interview-preparation | ⚫ | Question bank, system-design drills, answer frameworks |
4. Dependency graph
flowchart TD
subgraph F["Foundations 🟢"]
M00[00 Prerequisites] --> M01[01 Messaging Fundamentals]
M01 --> M02[02 SQS Introduction]
M02 --> M03[03 SQS Architecture]
M03 --> M04[04 Sending & Receiving]
M04 --> M05[05 Message Lifecycle]
end
subgraph C["Core mechanics 🟡🔴"]
M05 --> M06[06 Standard Queues]
M05 --> M08[08 Visibility Timeout]
M06 --> M07[07 FIFO Queues]
M08 --> M09[09 Delivery Semantics & Idempotency]
M06 --> M09
M09 --> M10[10 Retries]
M10 --> M11[11 Dead Letter Queues]
end
subgraph E["Efficiency 🟡🔴"]
M04 --> M12[12 Long Polling]
M04 --> M13[13 Batch Processing]
M12 --> M14[14 Scaling Consumers]
M13 --> M14
M14 --> M15[15 Performance & Throughput]
M13 --> M18[18 Cost]
M12 --> M18
end
subgraph O["Operating 🔴"]
M11 --> M17[17 Monitoring & Observability]
M15 --> M17
M04 --> M16[16 Security]
end
subgraph I["Integrations 🔴"]
M13 --> M19[19 Lambda]
M08 --> M19
M02 --> M20[20 SNS Fanout]
M20 --> M21[21 S3 / EventBridge / API GW]
M14 --> M22[22 ECS / EKS]
M14 --> M23[23 Spring Boot]
end
subgraph P["Synthesis ⚫"]
M09 --> M24[24 Distributed Systems Concepts]
M24 --> M25[25 Production Architecture]
M17 --> M25
M16 --> M25
M25 --> M26[26 Troubleshooting]
M25 --> M27[27 SQS vs Kafka vs RabbitMQ]
M25 --> M28[28 Case Studies]
M28 --> M29[29 Hands-on Labs]
M29 --> M30[30 Capstone]
M26 --> M31[31 Interview Prep]
M30 --> M31
endCritical path (minimum to be dangerous in production): 00 → 01 → 02 → 04 → 05 → 08 → 09 → 11 → 17.
5. Diagram inventory
The spec requires 40 diagrams. Each has exactly one owning module; others link to it.
| # | Diagram | Owner | Type |
|---|---|---|---|
| 1 | Synchronous coupling — the broken system | 01 | flowchart |
| 2 | Basic SQS architecture | 02 | flowchart |
| 3 | Producer → Queue → Consumer | 02 | flowchart |
| 4 | Multiple producers | 03 | flowchart |
| 5 | Multiple consumers (competing consumers) | 03 | flowchart |
| 6 | Distributed storage across redundant servers | 03 | flowchart |
| 7 | Why metrics are "Approximate" | 03 | sequence |
| 8 | Send / Receive / Delete API round trip | 04 | sequence |
| 9 | MessageId vs ReceiptHandle | 04 | flowchart |
| 10 | Message lifecycle state machine | 05 | stateDiagram |
| 11 | Message processing — success path | 05 | sequence |
| 12 | Message processing — failure path | 05 | sequence |
| 13 | Standard queue — best-effort ordering | 06 | flowchart |
| 14 | How a duplicate is born | 06 | sequence |
| 15 | FIFO queue ordering | 07 | flowchart |
| 16 | MessageGroupId parallelism | 07 | flowchart |
| 17 | Deduplication window | 07 | timeline |
| 18 | Head-of-line blocking in a message group | 07 | sequence |
| 19 | Visibility timeout — happy path timeline | 08 | timeline |
| 20 | Visibility timeout — overrun race condition | 08 | sequence |
| 21 | Heartbeat extension via ChangeMessageVisibility | 08 | sequence |
| 22 | At-least-once → duplicate payment | 09 | sequence |
| 23 | Idempotency key guard | 09 | flowchart |
| 24 | Retry flow with backoff + jitter | 10 | flowchart |
| 25 | Retry → maxReceiveCount → DLQ | 11 | flowchart |
| 26 | Poison message isolation | 11 | flowchart |
| 27 | DLQ replay / redrive | 11 | flowchart |
| 28 | Short polling vs long polling | 12 | sequence |
| 29 | Short-polling server sampling (why you get empty responses) | 12 | flowchart |
| 30 | Batch receive — one API call, ten messages | 13 | flowchart |
| 31 | Batch send / batch delete | 13 | sequence |
| 32 | API call vs message vs batch vs worker | 13 | flowchart |
| 33 | Consumer scaling / worker pool | 14 | flowchart |
| 34 | Queue-depth-driven autoscaling loop | 14 | flowchart |
| 35 | Backpressure and the downstream bottleneck | 14 | flowchart |
| 36 | Queue depth vs arrival/service rate | 15 | flowchart |
| 37 | Monitoring architecture (metrics → alarms → dashboards) | 17 | flowchart |
| 38 | Lambda event source mapping — who polls | 19 | sequence |
| 39 | SNS → SQS fanout | 20 | flowchart |
| 40 | Production reference architecture | 25 | flowchart |
Extras beyond the required 40: S3 → SQS (21), API Gateway → SQS (21), ECS worker fleet (22), Kafka partitions/offsets (27), RabbitMQ exchanges/bindings (27), capstone architecture (30), one per case study (28).
6. Code inventory
All Java 21 + AWS SDK for Java 2.x. Each sample is owned by one module.
| Module | Samples |
|---|---|
| 00 | Credential configuration (profile, env, IAM role); SqsClient construction; Maven/Gradle deps |
| 04 | CreateQueue; SendMessage; ReceiveMessage; DeleteMessage; GetQueueAttributes; message attributes; matching AWS CLI commands |
| 07 | FIFO queue creation; send with MessageGroupId + MessageDeduplicationId; content-based dedup |
| 08 | ChangeMessageVisibility; a heartbeat extender running on a scheduled executor |
| 09 | Idempotency guard with a DB unique constraint; idempotency-key table; transaction boundary placement |
| 10 | Classify transient vs permanent; exponential backoff with full jitter via ChangeMessageVisibility |
| 11 | Redrive policy in code; DLQ inspection tool; StartMessageMoveTask replay |
| 12 | Long-polling receive loop; empty-receive handling |
| 13 | SendMessageBatch; DeleteMessageBatch; partial-failure handling of batch results |
| 14 | Worker pool with bounded queue; graceful shutdown; in-flight accounting |
| 16 | IAM producer policy; IAM consumer policy; resource-based queue policy; SSE-KMS queue creation |
| 17 | Structured log line; correlation-ID propagation through message attributes; custom CloudWatch metric |
| 19 | Lambda handler with SQSBatchResponse / ReportBatchItemFailures; ESM config (CLI + IaC) |
| 20 | SNS topic + SQS subscription + queue policy; filter policy JSON |
| 21 | S3 event notification config; EventBridge rule → SQS target; API Gateway → SQS integration |
| 22 | Containerised consumer main loop; SIGTERM handler; KEDA ScaledObject for queue depth |
| 23 | Spring Cloud AWS config; @SqsListener; concurrency settings; error handler; graceful shutdown; Micrometer metrics |
| 29 | Per-lab code |
| 30 | Full capstone service |
7. Lab inventory
29-hands-on-labs/ — each lab states prerequisites, setup, steps, expected observation, teardown,
and estimated AWS cost (nearly all fall inside the free tier).
| Lab | Title | Teaches | Depends on |
|---|---|---|---|
| 01 | Create your first queue | Console + CLI, queue attributes | 04 |
| 02 | Send and receive a message | The three-call round trip, receipt handles | 04 |
| 03 | Watch visibility timeout | Observe invisibility, then redelivery | 08 |
| 04 | Force a redelivery | Crash before delete; watch ApproximateReceiveCount climb | 05, 08 |
| 05 | Configure a DLQ | maxReceiveCount, watch a poison message land | 11 |
| 06 | FIFO ordering and groups | Ordering within a group, parallelism across groups | 07 |
| 07 | Long polling vs short polling | Compare empty receives and API call counts | 12 |
| 08 | Batch APIs | 10× fewer API calls for the same message count | 13 |
| 09 | Build a Java consumer | SDK v2 poll loop, delete on success | 04, 12 |
| 10 | Spring Boot producer + consumer | @SqsListener end to end | 23 |
| 11 | SQS + Lambda | Event source mapping, partial batch failure | 19 |
| 12 | SNS → SQS fanout | Two queues, one publish, filter policies | 20 |
| 13 | S3 → SQS | Object-created events driving workers | 21 |
| 14 | Implement idempotency | Deliberately double-deliver; prove no double effect | 09 |
| 15 | Production-style worker service | Pool, heartbeat, graceful shutdown, metrics | 14, 22 |
| 16 | Monitor with CloudWatch | Dashboard + alarms on depth and oldest-message age | 17 |
| 17 | Break it and fix it | Injected failure; diagnose from metrics alone | 26 |
8. Revision pack
revision/ — written after the course body, derived from it, never copy-pasted from it.
| File | Purpose | Read time |
|---|---|---|
sqs-one-page-summary.md | The first door. Everything essential on one page. | 2 min |
sqs-5-minute-cheatsheet.md | Compact overview of the whole topic | 5 min |
sqs-15-minute-revision.md | Diagram- and table-driven full revision | 15 min |
sqs-interview-cheatsheet.md | Q&A bank by difficulty, with answer skeletons | 20 min |
sqs-production-cheatsheet.md | On-call reference: reliability, performance, monitoring, security, ops | scan |
sqs-decision-guide.md | Decision trees: Standard vs FIFO, Lambda vs ECS, batch sizing, when not SQS | scan |
sqs-troubleshooting-cheatsheet.md | Symptom → check → cause → fix table | scan |
sqs-api-cheatsheet.md | Every API + CLI equivalent, params, caveats | scan |
sqs-configuration-cheatsheet.md | Every knob: what it controls, typical values, common mistakes | scan |
9. Build order and status
| Phase | Contents | Status |
|---|---|---|
| 0 | Design, IA, verified facts, chapter template, skeleton | ✅ Done |
| 1 | Modules 00–05 — foundations | ✅ Done |
| 2 | Modules 06–11 — core mechanics | ✅ Done |
| 3 | Modules 12–15, 18 — efficiency and cost | ⬜ |
| 4 | Modules 16–17 — security and observability | ⬜ |
| 5 | Modules 19–23 — integrations and Java/Spring | ⬜ |
| 6 | Modules 24–28 — synthesis, troubleshooting, comparison, case studies | ⬜ |
| 7 | Modules 29–31 — labs, capstone, interview prep | ⬜ |
| 8 | revision/ pack, glossary.md, misconceptions.md, production-checklist.md | ⬜ |
| 9 | Audit: spec checklist, link check, Mermaid render check, fact re-verification | ⬜ |
10. Final audit checklist
From the spec's VALIDATION BEFORE COMPLETION section. Run at phase 9.
What SQS is · why queues exist · producer · consumer · polling · message · queue · standard queue · FIFO queue · at-least-once delivery · ordering · visibility timeout · receipt handle · DeleteMessage · retry · DLQ · redrive policy · long polling · batch APIs · consumer scaling · throughput · backpressure · idempotency · Lambda · SNS · S3 · ECS · EKS · Spring Boot · IAM · encryption · CloudWatch · troubleshooting · cost · production architecture · SQS vs Kafka · SQS vs RabbitMQ · hands-on labs · capstone · interview preparation
Plus mechanical checks:
- Every internal link resolves
- Every Mermaid block renders
- No quota or price hard-coded outside
_reference/aws-facts.md - No AWS credentials in any code sample
- Every diagram is followed by explanatory prose
- Every module has all 12 template sections
- Every term defined exactly once, on first use, and linked thereafter