Distributed Systems Concepts

Stepping back from the API. SQS has been teaching distributed-systems principles the whole way through — this module names them, separates the universal from the AWS-specific, and gives you the vocabulary that senior design discussions actually run on.

1. What you will learn

• Separate what is an SQS behaviour from what is a property of all distributed systems
• Explain the patterns SQS enables: event-driven architecture, CQRS, outbox, saga
• Explain eventual consistency in terms a product manager can act on
• Reason about fault isolation, bulkheads and graceful degradation
• Apply the vocabulary in a design discussion

2. Why this concept exists

• An engineer who has only learned SQS can operate a queue; an engineer who has learned the principles can evaluate Kafka, Pub/Sub or Service Bus next week
• Interviewers at senior level test the principles, using SQS as the prop

3. Beginner explanation

• Most of what makes SQS feel strange is not SQS — it is what happens when computers talk over a network they cannot trust
• Once you see that, the rules stop feeling arbitrary

4. How it actually works

• Universal (true of any distributed queue): at-least-once delivery, the impossibility of exactly-once delivery, the need for idempotency, backpressure, retry with backoff, poison messages, eventual consistency, partial failure, the Two Generals problem
• SQS-specific: visibility timeout as the liveness mechanism, receipt handles, maxReceiveCount redrive, MessageGroupId semantics, the 5-minute dedup window, the approximate metrics
• Decoupling, three kinds: temporal (different times), spatial (no addresses), and failure (one side's outage is the other's queue growth)
• Load levelling converts a throughput problem into a latency problem — an explicit trade, not a free win
• Bulkheads: separate queues per tenant or per workload so one cannot starve another. Fair queues are a lighter-weight version of the same idea
• Eventual consistency: the queue is the inconsistency window. Its length is your backlog divided by your service rate — a number you can measure and promise
• Event-driven architecture: events (things that happened) vs commands (things requested). Queues carry both; the distinction changes ownership, retry semantics and schema evolution
• Outbox pattern: write the business change and the outgoing message in one database transaction, then relay to SQS — the standard fix for 'committed the order but failed to send the message'
• Saga pattern: a sequence of local transactions with compensating actions, coordinated over queues, where a distributed transaction is impossible
• CQRS: queues carrying writes to a read-model projector
• Graceful degradation: shed load, queue it, or reject it — three different product decisions

5. Diagram

• Venn diagram: universal distributed-systems concepts vs SQS-specific mechanisms
• The outbox pattern: transaction boundary, relay, queue
• A saga with compensating transactions over queues
• Bulkheads: per-tenant queues vs one shared queue

6. Step-by-step flow

• Identify the consistency requirement of the business operation
• Decide whether the message is an event or a command
• Decide where the transaction boundary is — and whether you need an outbox
• Decide the isolation boundary — one queue, or one per tenant/workload
• Decide the degradation behaviour under overload
• Decide how inconsistency is surfaced to the user

7. Configuration

• No configuration — this module is design vocabulary
• It feeds directly into Module 25's design methodology

8. Production considerations

• The dual-write problem is everywhere: any code that writes to a database and then sends a message can fail between the two. The outbox is the answer; 'we'll just retry' is not
• Eventual consistency must be a product decision, not an implementation detail discovered by a customer. Quantify the window and put it in the SLO
• Bulkheads pay for themselves on the first noisy-neighbour incident — and cost you queue sprawl in the meantime
• Sagas are expensive. Compensating transactions are business logic and they are hard to test. Use them when a distributed transaction is genuinely impossible, not by default
• Every pattern here has a failure mode; teach the failure mode alongside the pattern

9. Common mistakes

• Treating at-least-once as an SQS quirk. Why → it is inherent to networked messaging. Instead → design for it in any system
• Dual-writing to a database and a queue. Why → no atomicity; you will lose messages or create phantom ones. Instead → outbox
• Selling eventual consistency as 'instant'. Why → the support tickets arrive anyway. Instead → measure and publish the window
• One shared queue for all tenants. Why → one tenant's backlog is everyone's outage. Instead → bulkheads or fair queues
• Sagas as a default architecture. Why → compensation logic is where the bugs live. Instead → reserve them for genuine cross-service transactions

10. Real-world example

• An order service that commits the order then fails to publish OrderPlaced: the incident, the missing-inventory-update symptom, and the outbox implementation that fixes it

11. Interview questions

• 🟡 What does eventual consistency mean for a queue-based system?
• 🟡 What is the difference between an event and a command?
• 🔴 Which SQS behaviours are universal to distributed systems and which are AWS-specific?
• 🔴 What is the dual-write problem and how does the outbox pattern solve it?
• 🔴 When would you use a saga instead of a distributed transaction?
• ⚫ Design a multi-tenant event platform with per-tenant isolation and a published consistency SLO.

12. Summary

• At-least-once, idempotency, backpressure and partial failure are universal; visibility timeout and receipt handles are SQS
• Decoupling is temporal, spatial and failure-wise
• The outbox pattern solves the dual-write problem
• Eventual consistency has a measurable window — measure it and publish it
• Bulkheads and fair queues prevent one tenant from becoming everyone's outage

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• eventual consistency
• dual-write problem
• outbox pattern
• saga
• CQRS
• bulkhead
• fault isolation
• graceful degradation
• event vs command

Diagrams to build:

• Universal vs SQS-specific
• Outbox pattern
• Saga with compensation
• Bulkheads

Code samples:

• An outbox table plus a relay that reads it and sends to SQS
• A compensating transaction sketch for a two-step saga

Mandatory "why?" answers:

• Why can't you atomically write to a database and a queue?
• Why is at-least-once not an AWS limitation?

← Previous: 23 — SQS with Java and Spring Boot · Index: Course home · Next: 25 — Production Architecture and System Design →