SQS vs Kafka vs RabbitMQ

A real architectural comparison, not a feature table. "SQS is simple, Kafka is fast, RabbitMQ is flexible" is a slogan. The actual difference is in the storage model — and once you see that, every other difference follows.

1. What you will learn

• Explain the core abstraction each system provides
• Explain Kafka's topics, partitions, offsets and consumer groups
• Explain RabbitMQ's exchanges, queues, bindings and acknowledgements
• Compare on 15 dimensions with evidence rather than folklore
• Choose between them for a given workload and defend the choice

2. Why this concept exists

• You will be asked to justify SQS against alternatives in design reviews and interviews
• And occasionally SQS genuinely is the wrong choice — recognising that is part of knowing it well

3. Beginner explanation

• SQS is a queue: take a message, it's gone
• Kafka is a log: read a message, it stays; you remember where you were
• RabbitMQ is a router: messages go through rules that decide which queues they land in

4. How it actually works

• The root difference — the storage model:
• SQS: distributed queue, destructive consumption, no replay, ≤ 14-day retention
• Kafka: append-only partitioned log, non-destructive reads, consumers track offsets, replay is native, retention by time or size (or infinite with tiered storage)
• RabbitMQ: in-memory/disk queues fed by exchanges; destructive consumption; routing is the differentiator
• Kafka concepts: topic → partitions → ordered offsets. Ordering is per partition; the partition key determines both ordering scope and parallelism — structurally the same idea as FIFO's MessageGroupId, which is a useful bridge for readers. Consumer groups allow many independent readers over the same data
• RabbitMQ concepts: producer → exchange (direct, topic, fanout, headers) → binding → queue → consumer; manual ack/nack, prefetch, TTL, per-queue DLX
• Comparison dimensions: primary abstraction · push/pull · storage · ordering · delivery semantics · replay · retention · consumer model · scaling · throughput · operational burden · infrastructure ownership · failure handling · acknowledgement · cloud integration · cost shape
• Cost shape matters: SQS is usage-priced with no floor; MSK/RabbitMQ have an hourly floor plus operational headcount. For spiky low-average workloads SQS wins on total cost even when its per-message price is higher
• Managed variants: MSK / MSK Serverless, Amazon MQ (RabbitMQ/ActiveMQ) — these change the operational-burden column substantially and should not be ignored in the comparison

5. Diagram

• Three storage models side by side: queue vs log vs exchange-routed queue
• Kafka topic with partitions, offsets and two consumer groups
• RabbitMQ exchange types and bindings
• A decision tree for choosing between them

6. Step-by-step flow

• Ask: does anything need to re-read the data later? → Kafka
• Ask: do multiple independent consumers need the same stream? → Kafka, or SNS+SQS fanout
• Ask: is the routing logic complex and dynamic? → RabbitMQ or EventBridge
• Ask: is the team's operational capacity small? → SQS
• Ask: is the workload spiky with a low average? → SQS on cost
• Ask: is sustained throughput very high with a stable rate? → Kafka on cost
• Otherwise → SQS

7. Configuration

• Not applicable — comparison module
• Feeds revision/sqs-decision-guide.md

8. Production considerations

• Replay is the decisive capability. If you might need to reprocess last month's events after a bug fix, SQS cannot do it and no configuration will change that
• Kafka's operational cost is real even managed: partition planning, consumer group rebalancing, offset management, schema evolution
• SNS + SQS fanout covers many 'we need Kafka' requirements — multiple independent consumers, without the operational load. It does not cover replay
• Do not run Kafka to get ordering if MessageGroupId on a FIFO queue would do — and do not use FIFO if a partition key over Kafka is what you actually need at volume
• Hybrid is normal and fine: Kafka for the event backbone, SQS for task queues at the edges

9. Common mistakes

• "SQS and Kafka are interchangeable." Why → destructive vs non-destructive consumption is a fundamental difference. Instead → choose on replay and consumer model
• Choosing Kafka for throughput you do not have. Why → operational cost with no benefit. Instead → measure first
• Choosing SQS when replay is a requirement. Why → it simply cannot. Instead → Kafka, or persist events separately
• Ignoring total cost of ownership. Why → per-message price is the smallest term; engineers are the largest. Instead → compare TCO
• Comparing SQS to self-hosted Kafka only. Why → MSK Serverless changes the comparison. Instead → compare like with like

10. Real-world example

• A company moving from RabbitMQ on EC2 to SQS: what got simpler, what they lost (routing flexibility, in-place priority), and how they replaced each capability

11. Interview questions

• 🟡 What is the difference between SQS and Kafka?
• 🟡 What are Kafka partitions and consumer groups?
• 🔴 Why can Kafka replay messages and SQS cannot? Explain from the storage model.
• 🔴 How is a Kafka partition key similar to a FIFO MessageGroupId?
• 🔴 When would you choose RabbitMQ over both?
• ⚫ Your company runs Kafka and someone proposes moving a task queue to SQS. Evaluate.

12. Summary

• Queue (SQS) vs log (Kafka) vs router (RabbitMQ) — the storage model explains everything else
• Replay and multiple independent consumer groups are Kafka's decisive advantages
• Complex routing is RabbitMQ's
• Zero operational burden and usage-shaped cost are SQS's
• SNS + SQS covers much of the 'we need Kafka' space, minus replay

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• topic
• partition
• offset
• consumer group
• exchange
• binding
• prefetch
• dead letter exchange
• destructive consumption

Diagrams to build:

• Three storage models
• Kafka partitions and consumer groups
• RabbitMQ exchanges and bindings
• Selection decision tree

Mandatory "why?" answers:

• Why can Kafka replay and SQS cannot?
• Why is a Kafka partition key structurally like a FIFO MessageGroupId?

← Previous: 26 — Troubleshooting · Index: Course home · Next: 28 — Corporate Case Studies →