SNS + SQS Fanout

One event, many independent consumers, each with its own queue, its own retry budget and its own DLQ. The fanout pattern is the most common production SQS topology, and the reason is simple: it is the only way to add a consumer without touching the producer.

1. What you will learn

• Explain why SNS and SQS are used together rather than either alone
• Build a fanout: one topic, many queues, many consumers
• Use filter policies to route without producer changes
• Explain raw message delivery and when you need it
• Reason about failure isolation across subscribers

2. Why this concept exists

• One queue = one logical consumer group. A second consumer that needs the same messages competes for them rather than getting its own copy
• SNS alone has no durable buffer — a subscriber that is down misses the message (HTTP subscribers retry, but the guarantees are weaker)
• SNS + SQS gives fanout and durability: each subscriber gets its own copy in its own buffer

3. Beginner explanation

• SNS is a broadcaster; SQS is a mailbox
• Publish once to the topic; every subscribed queue gets its own copy
• Each consumer reads its own mailbox at its own pace and can fail without affecting the others

4. How it actually works

• Topic → subscription → queue. Each subscribed queue receives an independent copy
• The queue policy must allow sns.amazonaws.com to sqs:SendMessage, conditioned on aws:SourceArn = the topic ARN (Module 16)
• Message envelope: by default SNS wraps the payload in JSON metadata. Raw message delivery strips it so the consumer sees exactly what was published
• ⚠️ Raw message delivery changes the parsing contract — enabling it later breaks existing consumers
• Filter policies are evaluated on message attributes (or, with MessageBody scope, on the body). A subscriber receives only matching messages — routing logic that lives in the subscription, not the producer
• Failure isolation: a broken consumer fills only its own queue and its own DLQ
• SNS-level DLQ: a subscription can have its own DLQ for delivery failures (topic → queue), distinct from the SQS-level DLQ (queue → consumer)
• FIFO topics → FIFO queues preserve ordering end to end; standard topics cannot feed FIFO queues
• Cross-account fanout is common and needs the queue policy
• Alternative: EventBridge offers richer content-based routing and a schema registry; SNS is simpler and cheaper for pure fanout (Module 21)

5. Diagram

• Diagram 39 — SNS topic fanning out to three queues and three consumers
• Competing consumers on one queue vs fanout — the diagram that makes the difference obvious
• Filter policy routing

6. Step-by-step flow

• Create the topic
• Create one queue per consumer
• Attach a queue policy allowing the topic to send, conditioned on aws:SourceArn
• Subscribe each queue to the topic
• Enable raw message delivery if the consumer wants the bare payload
• Attach filter policies where a subscriber wants a subset
• Publish once; SNS delivers a copy to every matching subscription
• Each consumer polls its own queue with its own retry and DLQ settings

7. Configuration

• Queue policy with Service: sns.amazonaws.com and aws:SourceArn
• RawMessageDelivery
• FilterPolicy and FilterPolicyScope (MessageAttributes or MessageBody)
• Subscription redrive policy (the SNS-level DLQ)
• Per-queue visibility, retention and redrive — each subscriber tunes independently

8. Production considerations

• Adding a consumer must never require a producer change. If it does, your topology is wrong
• Each queue needs its own DLQ and its own alarm — a shared DLQ destroys the isolation you built the fanout for
• Filter policies save money: an unmatched message is never delivered, so it is never received, processed or billed
• Decide the envelope format on day one; switching raw delivery on or off later is a breaking change for every consumer of that queue
• Watch for duplicate fanout: a consumer subscribed both directly to the topic and via another service's queue will process twice
• SNS delivery to SQS is at-least-once too — the duplicates compose

9. Common mistakes

• Two consumers on one queue expecting both to see every message. Why → they compete; each message goes to one of them. Instead → fanout
• Missing queue policy. Why → SNS cannot deliver and the failure is silent unless you look at SNS metrics. Instead → queue policy with aws:SourceArn
• Enabling raw delivery on a live queue. Why → consumers parsing the envelope break instantly. Instead → new subscription, migrate, retire
• A shared DLQ for all subscribers. Why → you cannot tell whose message failed. Instead → one DLQ per queue
• Filtering in the consumer instead of the subscription. Why → you pay to deliver, receive and discard. Instead → filter policy
• Forgetting SNS retries. Why → SNS→SQS delivery failures retry and can duplicate. Instead → idempotency everywhere

10. Real-world example

• OrderPlaced fanned out to inventory, notifications and analytics. Then the requirement "add a fraud-check consumer" arrives — and the producer is not touched

11. Interview questions

• 🟢 What is SNS? How does it differ from SQS?
• 🟡 Why use SNS and SQS together instead of either alone?
• 🟡 What is a filter policy?
• 🔴 What happens if two consumers poll the same queue expecting the same messages?
• 🔴 What is raw message delivery and why is changing it a breaking change?
• 🔴 How do you isolate failures between fanout subscribers?
• ⚫ Design an event-driven architecture where new consumers can be added without redeploying producers.

12. Summary

• One queue = one consumer group; fanout = one queue per consumer group
• SNS gives the broadcast, SQS gives the durable per-subscriber buffer
• Filter policies move routing out of the producer
• Each queue owns its DLQ, alarms and tuning
• Raw message delivery is a contract decision, made once

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• SNS
• topic
• subscription
• fanout
• filter policy
• raw message delivery
• subscription DLQ

Diagrams to build:

• Diagram 39 — SNS to SQS fanout
• Competing consumers vs fanout
• Filter policy routing

Code samples:

• Creating a topic, queues and subscriptions (Java + CLI + IaC)
• The queue policy granting SNS with aws:SourceArn
• A filter policy JSON with attribute matching
• Consumer handling both enveloped and raw payloads during a migration

Mandatory "why?" answers:

• Why can't two services just share one queue?
• Why does SNS alone not give durability?
• Why does each subscriber need its own DLQ?

← Previous: 19 — SQS and AWS Lambda · Index: Course home · Next: 21 — Event Sources: S3, EventBridge, API Gateway →