Learning/AWS SQS/Course design

Course Design — AWS SQS

The design document behind the course. It exists so that 32 modules written over many sessions stay consistent, nothing in the spec gets dropped, and every diagram/lab/code sample has an owner.

Spec: ../aws-sqs-prompt.md · Template: _reference/chapter-template.md · Facts: _reference/aws-facts.md

1. Design decisions

Why 32 modules and not the spec's 28 directories

The spec gives two lists that don't fully agree: a 25-entry LEVEL 0–24 curriculum and a 28-entry directory tree. Neither covers everything the spec's body sections demand. Four topics that the spec treats at length had no home in the directory tree:

Spec section with no directoryResolution
LEVEL 5 — Sending and Receiving MessagesNew module 04-sending-and-receiving
DELIVERY SEMANTICS (at-least-once, idempotency keys, transaction boundaries)New module 09-delivery-semantics-and-idempotency
COSTNew module 18-cost
DISTRIBUTED SYSTEM CONCEPTSNew module 24-distributed-systems-concepts

The spec explicitly permits this: "You may adjust the number of modules if necessary, but maintain a logical progression." Every directory name from the spec's tree is preserved; the four above are inserted and the sequence renumbered. §2 below is the full traceability table proving nothing was dropped.

Ordering change: lifecycle before queue types

The spec's directory tree puts 04-standard-queue / 05-fifo-queue before 06-message-lifecycle. This course reverses that. Standard-vs-FIFO is a comparison, and a comparison is meaningless before the reader knows what a message's life looks like. So:

03 architecture → 04 the API surface → 05 the lifecycle → 06 standard → 07 FIFO

By module 06 the reader already knows what "at-least-once" and "in-flight" mean, so the standard/FIFO contrast lands as a genuine trade-off rather than a feature list.

Ordering change: semantics before retries

09-delivery-semantics-and-idempotency sits before 10-retries and 11-dead-letter-queues. Retry design is unsafe to teach before the reader accepts that duplicate execution is guaranteed, not exceptional. Idempotency is a precondition for retries, not a footnote after them.

Three-tier depth

Every hard topic is written at three explicit levels (🟢 what / 🟡 how / 🔴 what breaks). A reader preparing for an interview next week reads levels 1–2. A reader on call reads level 3. The level-3 material is the reason this is a course and not a tutorial.

2. Traceability — spec → module

Proof that every numbered LEVEL and every named spec section has a home.

Spec LEVELModule
0 Prerequisites00
1 Messaging Fundamentals01
2 SQS Fundamentals02
3 How SQS Actually Works03
4 Queue Types06, 07
5 Sending and Receiving04
6 Message Lifecycle05
7 Reliability and Failure Handling09, 10
8 Visibility Timeout08
9 Dead Letter Queues11
10 Ordering and FIFO07
11 Scaling Consumers14
12 Performance and Throughput15
13 Long Polling12
14 Batch Operations13
15 Security16
16 Monitoring and Observability17
17 AWS Integrations19, 20, 21, 22
18 Advanced Architecture24, 25
19 Production Design25
20 Troubleshooting26
21 SQS vs Other Messaging27
22 Corporate Case Studies28
23 Hands-on Projects29, 30
24 Interview and System Design31
Named spec sectionHome
CRITICAL CONCEPT: BATCH VS MESSAGE13 (§3 is the spec's highest-priority clarity requirement)
STANDARD QUEUE06
FIFO QUEUE07
MESSAGE LIFECYCLE05
VISIBILITY TIMEOUT08
DELIVERY SEMANTICS09
RETRY AND DLQ10, 11
PERFORMANCE15
LONG POLLING12
AWS SDK EXAMPLES04 (introduced), then per-topic
JAVA / SPRING BOOT23
AWS LAMBDA + SQS19
ECS / EKS / EC2 CONSUMERS22
MONITORING17
OBSERVABILITY17
SECURITY16
SQS + SNS20
OTHER AWS INTEGRATIONS21
DISTRIBUTED SYSTEM CONCEPTS24
SQS VS KAFKA VS RABBITMQ27
CORPORATE CASE STUDIES28
REAL PRODUCTION FAILURE SCENARIOS26
SYSTEM DESIGN SECTION25 (methodology), 31 (practice)
NUMERICAL EXAMPLES15 (queueing math), reused in 14, 19, 25
REVISION CHEAT SHEETSrevision/ (9 files)
HANDS-ON LABS29 (17 labs)
CAPSTONE PROJECT30
INTERVIEW PREPARATION31 + §11 of every module
COMMON MISCONCEPTIONSmisconceptions.md
AWS CLI04 (basics), then per-topic; consolidated in revision/sqs-api-cheatsheet.md
COST18
GLOSSARYglossary.md
PRODUCTION READINESS CHECKLISTproduction-checklist.md
CURRENT AWS DOCUMENTATION_reference/aws-facts.md

3. Module map

#ModuleLevelOwns these concepts (defined here, linked elsewhere)
00prerequisites🟢AWS account, IAM basics, CLI setup, Java 21 + SDK v2 setup, sync vs async
01messaging-fundamentals🟢Coupling, buffering, load levelling, queue, producer, consumer, push vs pull
02sqs-introduction🟢What SQS is, what it is not, managed-service boundary, first mental model
03sqs-architecture🟡Distributed storage, redundant servers, why metrics say "Approximate", no broker to manage
04sending-and-receiving🟢SendMessage, ReceiveMessage, DeleteMessage, MessageId vs ReceiptHandle, message attributes, SDK v2 client, CLI
05message-lifecycle🟡The full state machine: stored → available → in-flight → deleted/redelivered/expired/DLQ
06standard-queues🟡At-least-once, best-effort ordering, unlimited throughput, why duplicates occur
07fifo-queues🔴MessageGroupId, MessageDeduplicationId, dedup interval, partitions, ordering scope, head-of-line blocking
08visibility-timeout🔴Timeout semantics, ChangeMessageVisibility, heartbeating, the race condition
09delivery-semantics-and-idempotency🔴At-least-once vs at-most-once vs exactly-once, idempotency keys, transaction boundaries
10retries🔴Transient vs permanent errors, backoff, jitter, retry storms, poison messages
11dead-letter-queues🔴maxReceiveCount, redrive policy, redrive allow policy, DLQ redrive, replay runbook
12long-polling🟡WaitTimeSeconds, empty receives, short-polling sampling behaviour, cost and latency
13batch-processing🟡Batch vs message vs API call vs consumer vs worker, batch APIs, partial failure
14scaling-consumers🔴Horizontal scaling, worker pools, thread pools, queue-depth autoscaling, backpressure
15performance-and-throughput🔴Little's Law, arrival vs service rate, concurrency math, bottleneck analysis
16security🔴IAM identity vs resource policy, queue policies, SSE-SQS vs SSE-KMS, VPC endpoints, cross-account
17monitoring-and-observability🔴Every CloudWatch metric, alarm design, correlation IDs, tracing, structured logs
18cost🟡Request-based billing, 64 KB chunking, batching/long-polling savings, cost modelling
19lambda🔴Event source mapping, who polls, scaling, ReportBatchItemFailures, when not to use Lambda
20sns-integration🔴Fanout, filter policies, raw message delivery, per-subscriber DLQs
21event-sources🔴S3 → SQS, EventBridge → SQS, API Gateway → SQS, Step Functions
22ecs-eks🔴Self-managed consumers, graceful shutdown, SIGTERM, HPA/KEDA on queue depth
23spring-boot🔴Spring Cloud AWS, @SqsListener, concurrency, error handling — and what is Spring, not SQS
24distributed-systems-conceptsWhich ideas are SQS-specific vs universal; eventual consistency, fault isolation
25production-architectureThe 20-question design methodology, reference architecture, multi-service patterns
26troubleshooting18 failure scenarios as SYMPTOM → CAUSE → VERIFY → FIX → PREVENT
27sqs-vs-kafka-rabbitmqArchitectural comparison; partitions/offsets, exchanges/bindings; selection criteria
28case-studies6 corporate scenarios end to end
29hands-on-labs🟡17 progressive labs
30capstoneOrder Processing Platform, production-grade
31interview-preparationQuestion bank, system-design drills, answer frameworks

4. Dependency graph

flowchart TD
    subgraph F["Foundations 🟢"]
        M00[00 Prerequisites] --> M01[01 Messaging Fundamentals]
        M01 --> M02[02 SQS Introduction]
        M02 --> M03[03 SQS Architecture]
        M03 --> M04[04 Sending & Receiving]
        M04 --> M05[05 Message Lifecycle]
    end

    subgraph C["Core mechanics 🟡🔴"]
        M05 --> M06[06 Standard Queues]
        M05 --> M08[08 Visibility Timeout]
        M06 --> M07[07 FIFO Queues]
        M08 --> M09[09 Delivery Semantics & Idempotency]
        M06 --> M09
        M09 --> M10[10 Retries]
        M10 --> M11[11 Dead Letter Queues]
    end

    subgraph E["Efficiency 🟡🔴"]
        M04 --> M12[12 Long Polling]
        M04 --> M13[13 Batch Processing]
        M12 --> M14[14 Scaling Consumers]
        M13 --> M14
        M14 --> M15[15 Performance & Throughput]
        M13 --> M18[18 Cost]
        M12 --> M18
    end

    subgraph O["Operating 🔴"]
        M11 --> M17[17 Monitoring & Observability]
        M15 --> M17
        M04 --> M16[16 Security]
    end

    subgraph I["Integrations 🔴"]
        M13 --> M19[19 Lambda]
        M08 --> M19
        M02 --> M20[20 SNS Fanout]
        M20 --> M21[21 S3 / EventBridge / API GW]
        M14 --> M22[22 ECS / EKS]
        M14 --> M23[23 Spring Boot]
    end

    subgraph P["Synthesis ⚫"]
        M09 --> M24[24 Distributed Systems Concepts]
        M24 --> M25[25 Production Architecture]
        M17 --> M25
        M16 --> M25
        M25 --> M26[26 Troubleshooting]
        M25 --> M27[27 SQS vs Kafka vs RabbitMQ]
        M25 --> M28[28 Case Studies]
        M28 --> M29[29 Hands-on Labs]
        M29 --> M30[30 Capstone]
        M26 --> M31[31 Interview Prep]
        M30 --> M31
    end

Critical path (minimum to be dangerous in production): 00 → 01 → 02 → 04 → 05 → 08 → 09 → 11 → 17.

5. Diagram inventory

The spec requires 40 diagrams. Each has exactly one owning module; others link to it.

#DiagramOwnerType
1Synchronous coupling — the broken system01flowchart
2Basic SQS architecture02flowchart
3Producer → Queue → Consumer02flowchart
4Multiple producers03flowchart
5Multiple consumers (competing consumers)03flowchart
6Distributed storage across redundant servers03flowchart
7Why metrics are "Approximate"03sequence
8Send / Receive / Delete API round trip04sequence
9MessageId vs ReceiptHandle04flowchart
10Message lifecycle state machine05stateDiagram
11Message processing — success path05sequence
12Message processing — failure path05sequence
13Standard queue — best-effort ordering06flowchart
14How a duplicate is born06sequence
15FIFO queue ordering07flowchart
16MessageGroupId parallelism07flowchart
17Deduplication window07timeline
18Head-of-line blocking in a message group07sequence
19Visibility timeout — happy path timeline08timeline
20Visibility timeout — overrun race condition08sequence
21Heartbeat extension via ChangeMessageVisibility08sequence
22At-least-once → duplicate payment09sequence
23Idempotency key guard09flowchart
24Retry flow with backoff + jitter10flowchart
25Retry → maxReceiveCount → DLQ11flowchart
26Poison message isolation11flowchart
27DLQ replay / redrive11flowchart
28Short polling vs long polling12sequence
29Short-polling server sampling (why you get empty responses)12flowchart
30Batch receive — one API call, ten messages13flowchart
31Batch send / batch delete13sequence
32API call vs message vs batch vs worker13flowchart
33Consumer scaling / worker pool14flowchart
34Queue-depth-driven autoscaling loop14flowchart
35Backpressure and the downstream bottleneck14flowchart
36Queue depth vs arrival/service rate15flowchart
37Monitoring architecture (metrics → alarms → dashboards)17flowchart
38Lambda event source mapping — who polls19sequence
39SNS → SQS fanout20flowchart
40Production reference architecture25flowchart

Extras beyond the required 40: S3 → SQS (21), API Gateway → SQS (21), ECS worker fleet (22), Kafka partitions/offsets (27), RabbitMQ exchanges/bindings (27), capstone architecture (30), one per case study (28).

6. Code inventory

All Java 21 + AWS SDK for Java 2.x. Each sample is owned by one module.

ModuleSamples
00Credential configuration (profile, env, IAM role); SqsClient construction; Maven/Gradle deps
04CreateQueue; SendMessage; ReceiveMessage; DeleteMessage; GetQueueAttributes; message attributes; matching AWS CLI commands
07FIFO queue creation; send with MessageGroupId + MessageDeduplicationId; content-based dedup
08ChangeMessageVisibility; a heartbeat extender running on a scheduled executor
09Idempotency guard with a DB unique constraint; idempotency-key table; transaction boundary placement
10Classify transient vs permanent; exponential backoff with full jitter via ChangeMessageVisibility
11Redrive policy in code; DLQ inspection tool; StartMessageMoveTask replay
12Long-polling receive loop; empty-receive handling
13SendMessageBatch; DeleteMessageBatch; partial-failure handling of batch results
14Worker pool with bounded queue; graceful shutdown; in-flight accounting
16IAM producer policy; IAM consumer policy; resource-based queue policy; SSE-KMS queue creation
17Structured log line; correlation-ID propagation through message attributes; custom CloudWatch metric
19Lambda handler with SQSBatchResponse / ReportBatchItemFailures; ESM config (CLI + IaC)
20SNS topic + SQS subscription + queue policy; filter policy JSON
21S3 event notification config; EventBridge rule → SQS target; API Gateway → SQS integration
22Containerised consumer main loop; SIGTERM handler; KEDA ScaledObject for queue depth
23Spring Cloud AWS config; @SqsListener; concurrency settings; error handler; graceful shutdown; Micrometer metrics
29Per-lab code
30Full capstone service

7. Lab inventory

29-hands-on-labs/ — each lab states prerequisites, setup, steps, expected observation, teardown, and estimated AWS cost (nearly all fall inside the free tier).

LabTitleTeachesDepends on
01Create your first queueConsole + CLI, queue attributes04
02Send and receive a messageThe three-call round trip, receipt handles04
03Watch visibility timeoutObserve invisibility, then redelivery08
04Force a redeliveryCrash before delete; watch ApproximateReceiveCount climb05, 08
05Configure a DLQmaxReceiveCount, watch a poison message land11
06FIFO ordering and groupsOrdering within a group, parallelism across groups07
07Long polling vs short pollingCompare empty receives and API call counts12
08Batch APIs10× fewer API calls for the same message count13
09Build a Java consumerSDK v2 poll loop, delete on success04, 12
10Spring Boot producer + consumer@SqsListener end to end23
11SQS + LambdaEvent source mapping, partial batch failure19
12SNS → SQS fanoutTwo queues, one publish, filter policies20
13S3 → SQSObject-created events driving workers21
14Implement idempotencyDeliberately double-deliver; prove no double effect09
15Production-style worker servicePool, heartbeat, graceful shutdown, metrics14, 22
16Monitor with CloudWatchDashboard + alarms on depth and oldest-message age17
17Break it and fix itInjected failure; diagnose from metrics alone26

8. Revision pack

revision/ — written after the course body, derived from it, never copy-pasted from it.

FilePurposeRead time
sqs-one-page-summary.mdThe first door. Everything essential on one page.2 min
sqs-5-minute-cheatsheet.mdCompact overview of the whole topic5 min
sqs-15-minute-revision.mdDiagram- and table-driven full revision15 min
sqs-interview-cheatsheet.mdQ&A bank by difficulty, with answer skeletons20 min
sqs-production-cheatsheet.mdOn-call reference: reliability, performance, monitoring, security, opsscan
sqs-decision-guide.mdDecision trees: Standard vs FIFO, Lambda vs ECS, batch sizing, when not SQSscan
sqs-troubleshooting-cheatsheet.mdSymptom → check → cause → fix tablescan
sqs-api-cheatsheet.mdEvery API + CLI equivalent, params, caveatsscan
sqs-configuration-cheatsheet.mdEvery knob: what it controls, typical values, common mistakesscan

9. Build order and status

PhaseContentsStatus
0Design, IA, verified facts, chapter template, skeletonDone
1Modules 00–05 — foundationsDone
2Modules 06–11 — core mechanicsDone
3Modules 12–15, 18 — efficiency and cost
4Modules 16–17 — security and observability
5Modules 19–23 — integrations and Java/Spring
6Modules 24–28 — synthesis, troubleshooting, comparison, case studies
7Modules 29–31 — labs, capstone, interview prep
8revision/ pack, glossary.md, misconceptions.md, production-checklist.md
9Audit: spec checklist, link check, Mermaid render check, fact re-verification

10. Final audit checklist

From the spec's VALIDATION BEFORE COMPLETION section. Run at phase 9.

What SQS is · why queues exist · producer · consumer · polling · message · queue · standard queue · FIFO queue · at-least-once delivery · ordering · visibility timeout · receipt handle · DeleteMessage · retry · DLQ · redrive policy · long polling · batch APIs · consumer scaling · throughput · backpressure · idempotency · Lambda · SNS · S3 · ECS · EKS · Spring Boot · IAM · encryption · CloudWatch · troubleshooting · cost · production architecture · SQS vs Kafka · SQS vs RabbitMQ · hands-on labs · capstone · interview preparation

Plus mechanical checks:

  • Every internal link resolves
  • Every Mermaid block renders
  • No quota or price hard-coded outside _reference/aws-facts.md
  • No AWS credentials in any code sample
  • Every diagram is followed by explanatory prose
  • Every module has all 12 template sections
  • Every term defined exactly once, on first use, and linked thereafter