How SQS Actually Works

This module exists to answer one question properly:

Why is every SQS metric called Approximate?

The answer turns out to explain duplicate delivery, out-of-order delivery, and empty responses on a non-empty queue — all at once, from a single design decision. Once you have it, most of the rest of the course becomes deducible rather than memorised.

A note on sources. AWS does not publish SQS's internals in detail. Everything stated as fact below is documented AWS behaviour, linked in aws-facts.md. Where a diagram illustrates a model consistent with that documented behaviour rather than a confirmed implementation, it says so.

1. What you will learn

• Describe how SQS distributes messages across servers within a Region
• Explain why ApproximateNumberOfMessages cannot be exact
• Explain why ReceiveMessage can return nothing while the queue is not empty
• Explain why standard queues can deliver out of order and can duplicate
• Trace how multiple producers and multiple consumers share one queue

2. The one design decision

A queue could be implemented as a single ordered list on a single machine. That design gives you, for free:

• Exact counts — you can just measure the list
• Strict ordering — there is one list, in one order
• No duplicates — one copy, one delete

It also gives you, unavoidably:

• A throughput ceiling — one machine's worth
• A durability problem — one disk
• An availability problem — one host to lose

AWS chose the other trade. SQS distributes each queue's messages redundantly across multiple servers within a Region, with no single coordinator. That buys effectively unlimited throughput and high durability, and it costs exactness in all three of the properties above.

📌 Remember this. Approximate counts, best-effort ordering, and occasional duplicates are not bugs and not limitations AWS intends to fix. They are the visible surface of "distributed, with no global coordinator."

3. The storage model

flowchart TB
    subgraph PRODUCERS["Producers"]
        P1["Producer A"]
        P2["Producer B"]
        P3["Producer C"]
    end

    FE{{"SQS front end<br/>(request routing)"}}

    subgraph STORAGE["☁️ SQS storage — multiple servers, one Region"]
        S1[("Server 1<br/>m1 m4 m7")]
        S2[("Server 2<br/>m1 m2 m5")]
        S3[("Server 3<br/>m2 m3 m6")]
        S4[("Server 4<br/>m3 m4 m8")]
    end

    P1 --> FE
    P2 --> FE
    P3 --> FE
    FE --> S1
    FE --> S2
    FE --> S3
    FE --> S4

Reading the diagram. Two things to notice.

1. Each message appears on more than one server. m1 is on servers 1 and 2; m2 on 2 and 3. That redundancy is what makes the message survive a host failure — and, as §6 shows, it is exactly what makes duplicates possible.
2. No server holds the whole queue. There is no single place that knows the queue's complete contents. Any question about "the queue" must be answered by asking several servers and combining their answers.

(Illustrative model. AWS documents that messages are stored redundantly on multiple servers; the specific placement shown is for explanation.)

4. Multiple producers, multiple consumers

Neither side coordinates with the other, or with itself.

flowchart LR
    subgraph P["Producers — no coordination"]
        PA["Order Service<br/>× 12 instances"]
        PB["Admin tool"]
        PC["Batch importer"]
    end

    Q[("SQS Queue")]

    subgraph C["Consumers — competing consumers"]
        CA["Worker 1"]
        CB["Worker 2"]
        CC["Worker 3"]
        CD["Worker N"]
    end

    PA --> Q
    PB --> Q
    PC --> Q
    Q -.->|"each message to<br/>exactly one worker"| CA
    Q -.-> CB
    Q -.-> CC
    Q -.-> CD

Reading the diagram. The consumer side is the competing consumers pattern: every worker polls the same queue, and each message goes to exactly one of them. There is no partition assignment, no group membership protocol, no rebalancing — which is a meaningful operational difference from Kafka (Module 27), where adding a consumer triggers a rebalance.

Term: competing consumers — Simple: many workers share one queue, and each message is handled by one of them. Technical: a pattern where N consumers poll a single queue, with the queue distributing messages among them, giving horizontal scaling without partition assignment. Example: 40 containers polling sqs-course-payments.

📌 Scaling consumers in SQS is literally just starting more processes. Nothing needs to be told.

5. Why metrics say "Approximate"

Now the central question.

Ask "how many messages are in the queue?" There is no single place that knows. The front end must poll the storage servers and add up what they report.

sequenceDiagram
    autonumber
    participant You
    participant FE as SQS front end
    participant S1 as Server 1
    participant S2 as Server 2
    participant S3 as Server 3
    participant P as A producer

    You->>FE: GetQueueAttributes(ApproximateNumberOfMessages)
    FE->>S1: count?
    S1-->>FE: 40
    FE->>S2: count?
    P->>S3: SendMessage (arrives mid-query)
    S2-->>FE: 35
    FE->>S3: count?
    S3-->>FE: 26
    FE-->>You: ~101
    Note over You,S3: The three counts were taken at three<br/>different instants. A producer wrote during<br/>the query. A consumer may have deleted.<br/>No instant existed at which the answer was 101.

Reading the diagram. The number you get back is not a slightly-stale snapshot of a true value. There was no single moment at which the queue contained 101 messages. The figure is a sum of independent observations taken at slightly different times, in a system that never stops changing.

Why are the metrics called "Approximate"? Because computing an exact count would require freezing every storage server simultaneously — a distributed lock across the whole queue, taken on every metric read. That would destroy the throughput the distribution was there to provide. AWS chose to name the compromise honestly rather than hide it behind a number that looks exact and is not.

What this means in practice

The metrics are published at one-minute granularity, which compounds the point: even an exact count would be up to a minute old by the time you saw it.

⚠️ Common mistake. Autoscaling logic with a condition like "scale to zero when ApproximateNumberOfMessages == 0". Approximate counts flicker, and a scale-to-zero on a transient zero will terminate workers holding in-flight messages. Module 14 covers doing this properly.

6. Why duplicates happen

Redundancy is the cause. Follow one message.

sequenceDiagram
    autonumber
    participant C as Consumer
    participant FE as SQS front end
    participant SA as Server A (has m1)
    participant SB as Server B (has m1)

    Note over SA,SB: m1 is stored on both servers

    C->>FE: ReceiveMessage
    FE->>SA: any messages?
    SA-->>FE: m1
    FE-->>C: m1 + ReceiptHandle

    C->>C: process m1 ✅
    C->>FE: DeleteMessage(handle)
    FE->>SA: delete m1
    SA-->>FE: deleted
    FE-xSB: delete m1 (does not arrive —<br/>server busy / partitioned)
    FE-->>C: 200 OK

    Note over C: Consumer believes it is done.<br/>It is right — for Server A.

    C->>FE: ReceiveMessage (later)
    FE->>SB: any messages?
    SB-->>FE: m1 (still has it)
    FE-->>C: m1 again 🔁

Reading the diagram. Nothing malfunctioned. The consumer processed correctly and deleted correctly. One replica did not get the delete in time and served the message again.

Why can't AWS just make deletes reach every replica before returning? They could — by waiting for every replica to acknowledge. That turns every delete into a synchronous fan-out whose latency is governed by the slowest replica, and whose availability is governed by the least-available one. SQS would become slower and less available in exchange for removing duplicates that a correctly-written consumer already tolerates. AWS chose availability.

This is a design trade, not an accident, and it is why Module 09 treats idempotency as mandatory rather than defensive.

📌 Duplicates are rare in absolute terms and certain in aggregate. At a million messages a day, "rare" happens every day. Design for it.

There is a second, more common source of duplicate processing — a consumer crashing after doing the work and before deleting — which is Module 08's subject. Both produce the same symptom; only one is about storage.

7. Why ordering is best-effort

There is no global sequencer. Messages sent as m1, m2, m3 are written to different servers, and consumers poll servers independently. Nothing guarantees a consumer sees m1 before m2.

flowchart LR
    subgraph SENT["Sent, in order"]
        direction TB
        A1["m1"] --> A2["m2"] --> A3["m3"] --> A4["m4"]
    end
    SENT --> Q[("Distributed across<br/>storage servers")]
    Q --> RECV
    subgraph RECV["Received — a legal outcome"]
        direction TB
        B1["m2"] --> B2["m1"] --> B3["m4"] --> B4["m3"]
    end

Reading the diagram. "Best-effort" means SQS makes a genuine attempt — messages usually arrive roughly in order, and at low volume in a test environment they will look perfectly ordered. That is precisely the trap: development traffic hides the behaviour that production reveals.

There is also a second, explicitly documented reordering mechanism that surprises people:

⚠️ On a standard queue with a redrive policy where maxReceiveCount > 3, a message that has been received 3 or more times without being deleted is moved to the back of the queue (aws-facts.md §7).

So a message that keeps failing does not retry promptly in place — it goes to the end of the line. This matters twice over: it makes retry latency unpredictable (Module 10), and it distorts ApproximateAgeOfOldestMessage, which then reports the age of the next message under that threshold rather than the genuinely oldest one.

Why does SQS move a repeatedly-failed message to the back? Because a poison message at the head of the queue would otherwise be re-served immediately, over and over, consuming receive capacity that healthy messages need. Moving it back is head-of-line-blocking avoidance for standard queues.

FIFO queues solve ordering differently — with partitions and a per-group sequencer, paid for in throughput. That is Module 07.

8. Why an empty response does not mean an empty queue

The third consequence, and the one that costs people the most money.

flowchart TB
    C["Consumer:<br/>ReceiveMessage<br/>(short polling)"] --> FE{{"SQS front end"}}
    FE -->|"samples a subset"| S1[("Server 1<br/>❌ empty")]
    FE -->|"samples a subset"| S2[("Server 2<br/>❌ empty")]
    FE -.->|"not queried<br/>this time"| S3[("Server 3<br/>✅ has 4 messages")]
    FE -.->|"not queried"| S4[("Server 4<br/>✅ has 2 messages")]
    FE ==>|"returns: 0 messages"| C

Reading the diagram. With short polling, SQS samples a subset of servers and returns immediately with whatever that subset had. Six messages exist. The consumer was told zero.

This is documented behaviour, not speculation: short polling queries a subset of servers based on a weighted random distribution, while long polling queries all of them.

Why would SQS ever sample instead of asking everyone? Because asking every server on every receive makes each call as slow as the slowest server, on a call that happens millions of times a second across the service. Sampling keeps ReceiveMessage fast. Long polling makes the opposite trade — query everything, but hold the request open so the cost is amortised over a wait rather than paid per call. That is why long polling both reduces empty responses and lowers latency, which sounds contradictory until you see this diagram.

Two immediate consequences:

• Never conclude "the queue is empty" from one empty response. Emptiness is a sustained condition, not a single observation.
• Turn on long polling. ReceiveMessageWaitTimeSeconds = 20, on every queue. Module 12 quantifies what leaving it off costs.

9. The full picture

Every surprising SQS behaviour, traced to its cause:

flowchart TD
    ROOT["<b>Distributed storage,<br/>redundant, no global coordinator</b>"]

    ROOT --> C1["Counts must be summed<br/>from many servers"]
    ROOT --> C2["Deletes propagate<br/>asynchronously"]
    ROOT --> C3["No global sequencer"]
    ROOT --> C4["Receives sample servers<br/>(short polling)"]

    C1 --> E1["📊 <b>Approximate</b> metrics"]
    C2 --> E2["🔁 <b>Duplicate</b> delivery"]
    C3 --> E3["🔀 <b>Best-effort</b> ordering"]
    C4 --> E4["📭 <b>Empty</b> receives on a<br/>non-empty queue"]

    E1 --> F1["Alarm on trends and<br/>oldest-message age"]
    E2 --> F2["Make consumers idempotent"]
    E3 --> F3["Order-independent design,<br/>or FIFO"]
    E4 --> F4["Enable long polling"]

Reading the diagram. Read it top to bottom as cause → behaviour → what you do about it. Every one of the four "what you do about it" boxes is a whole module later in the course. If you can reconstruct this diagram, you can derive most of SQS's rules rather than memorising them.

10. What is exact, and what is not

Not everything about a queue is approximate. The distinction matters when you are debugging.

📌 Rule of thumb: anything about configuration is exact; anything about aggregate state is approximate. AWS prefixes the approximate ones — so the naming is doing your documentation for you.

Note that ApproximateReceiveCount is approximate too. It is reliable enough to drive maxReceiveCount and excellent for debugging, but it is not a counter you should do arithmetic with.

11. Production considerations

Never build logic on an exact count. Alarm on sustained conditions and on rates of change. Module 17 shows which metrics to combine.

Design consumers to be idempotent and order-independent from day one. Retrofitting idempotency into a system that assumed exactly-once is expensive — it means auditing every side effect in every handler. Building it in from the start costs one extra database write per message.

Low-volume testing hides all of this. Duplicates, reordering and empty receives all become visible at production volume and are effectively invisible at ten messages a minute. Deliberately test for them (Lab 04, Lab 14) rather than waiting to discover them.

You cannot partition an SQS queue, and you do not need to. Unlike Kafka, there is no partition count to choose, no repartitioning operation, and no upper bound you provision against. A standard queue absorbs whatever you throw at it (aws-facts.md §4). The corollary is that you also cannot pin related messages to the same consumer — for that you need FIFO message groups (Module 07).

The in-flight quota is a real ceiling. Roughly 120,000 messages may be received-but-not-deleted at once. A consumer that receives and never deletes will silently approach it, and the symptom differs by polling mode: short polling returns an OverLimit error, long polling returns nothing at all (aws-facts.md §3). A queue that has mysteriously stopped delivering, with no errors, is a classic presentation of this.

12. Common mistakes

Alarming on ApproximateNumberOfMessages == 0. Why it's wrong: approximate counts flicker; the alarm will flap and the scale-in it triggers will kill busy workers. Instead: alarm on sustained depth or on ApproximateAgeOfOldestMessage.

Concluding the queue is empty from one empty receive. Why it's wrong: short polling sampled a subset of servers that happened to have nothing. Instead: long polling, and treat emptiness as a sustained condition across several polls.

Assuming messages sent in order arrive in order on a standard queue. Why it's wrong: no global sequencer, plus the explicit back-of-queue move after 3 receives. Instead: FIFO, or encode ordering in your data rather than relying on delivery.

Treating a duplicate as an incident. Why it's wrong: it is expected behaviour at scale, and paging on it trains people to ignore pages. Instead: count duplicates as a metric; alarm on the rate rising, which signals a real problem upstream.

Ignoring ApproximateNumberOfMessagesNotVisible. Why it's wrong: a rising, non-falling in-flight count means messages are being received and never deleted — and at ~120,000 the queue stops delivering. Instead: alarm on it approaching the quota (Module 17).

13. Real-world example

An ops engineer opens a ticket:

"The SQS console says the queue has 0 messages, but our consumer just processed one. Is the console broken? Also, we saw the same order processed twice this morning."

Walk it end to end.

"The console says 0." ApproximateNumberOfMessages was summed across storage servers at one-minute granularity. Messages arriving and being consumed continuously means the number is a reading of a moving system, never a snapshot of a still one. Zero means "approximately zero", which at low volume is entirely consistent with a message existing.

"But my consumer got one." Long polling queries all servers and returns the instant anything is available. The consumer's view is strictly more current than the metric's.

"The same order was processed twice." Two candidate causes, and they are distinguished by one metric:

They logged ApproximateReceiveCount on every receive — which is why the diagnosis took minutes instead of days. It was 2: a consumer had been OOM-killed mid-processing. The replication case is much rarer.

The resolution. Nothing about SQS was changed. They added an idempotency guard keyed on orderId (Module 09), fixed the memory limit, and replaced their depth-based alarm with one on oldest-message age.

🎯 Interview point. "How would you tell whether a duplicate came from redelivery or from replication?" — ApproximateReceiveCount. It is a small answer that demonstrates you have actually debugged this.

14. Interview questions

🟢 Beginner

Why does SQS scale without you configuring anything? Messages are distributed across many servers within a Region, and AWS adds capacity automatically. There is no partition count, no cluster size, and no provisioned throughput on a standard queue.

🟡 Intermediate

Why are SQS metrics called "Approximate"? Because no single server holds the whole queue. The count is a sum of independent per-server observations taken at slightly different instants, in a system that is changing throughout. There was never a moment at which the reported number was exactly true. Making it exact would require a distributed freeze on every read.

Why can ReceiveMessage return zero messages when the queue is not empty? Short polling samples a subset of storage servers and returns immediately. If the sampled subset happens to hold nothing, you get an empty response even though other servers have messages. Long polling queries all servers and waits, which is why it both reduces empty responses and lowers delivery latency.

What is the competing-consumers pattern? Many consumers poll one queue and each message is delivered to exactly one of them. It gives horizontal scaling with no partition assignment, no group membership and no rebalancing — you scale by starting more processes.

🔴 Advanced

Explain how SQS's storage model causes both duplicates and out-of-order delivery. Could AWS fix one without the other?

Both come from the same root — redundant storage with no global coordinator — but through different mechanisms.

Duplicates: a message lives on several servers. A DeleteMessage must propagate to all of them. If it does not reach one in time, that replica can serve the message again. Fixing this means making deletes synchronous across all replicas, which makes every delete as slow as the slowest replica and as available as the least available one.

Reordering: there is no global sequencer, so nothing establishes a total order across servers. Fixing this means introducing a sequencer, which is a coordination point and therefore a throughput ceiling.

So yes, they are separable in principle — the fixes are different — but both fixes cost the same thing: coordination, and therefore throughput and availability. AWS does offer both fixes, bundled, in FIFO queues: send-side deduplication over a 5-minute window, plus per-message-group ordering. The price is exactly what you would predict — a throughput limit that is per-partition rather than unlimited (Module 07).

If duplicates come from replication, why does making my consumer idempotent help? The duplicate already happened. Because the goal was never exactly-once delivery — that is unachievable. The goal is exactly-once effect. Idempotency moves the deduplication to the only place that has enough context to do it correctly: the consumer, which knows the business meaning of the message. Module 09 develops this.

⚫ System design

Design a queue-backed system where approximate counts are unacceptable for a business decision — for example, "close the batch when all 50,000 records are processed." How do you get an exact number?

The key insight to lead with: do not ask the queue. The queue's counts are approximate by construction, and no amount of polling makes them exact. Move the counting to a system that can be exact.

A workable design:

1. Assign a batch id when the work is created, and record the expected count in a database row: {batchId, expected: 50000, completed: 0}.
2. Each message carries the batch id. On successful processing, the consumer increments completed in the same transaction as the work itself — which requires idempotency so a redelivery does not double-count (Module 09). A processed-records table keyed on (batchId, recordId) gives both properties at once.
3. Completion is a database condition, completed == expected, not a queue condition. Detect it with a conditional update that fires exactly once.
4. Handle the failure tail. Messages that reach the DLQ never increment the counter, so the batch would hang forever. Either count DLQ arrivals as terminal failures against the same row, or give the batch a deadline after which it is reconciled by a sweeper.

Follow-ups a good candidate anticipates:

• "What if a consumer processes but crashes before committing?" The transaction rolls back and the message is redelivered — which is why the increment must be inside the transaction, not after it.
• "What about the DLQ path?" Covered above. This is the part most candidates miss, and it is the one that causes a stuck batch in production.
• "Could you just use the queue depth?" No — and being able to say precisely why is the point of this module.

15. Summary

• SQS stores messages redundantly across multiple servers within a Region, with no global coordinator.
• That single decision produces four visible behaviours:
  • Approximate metrics — counts are summed from independent, slightly-stale observations
  • Duplicate delivery — a delete may not reach every replica in time
  • Best-effort ordering — no global sequencer, plus an explicit back-of-queue move after 3 receives
  • Empty receives — short polling samples a subset of servers
• Each has a standard response: alarm on trends, be idempotent, design order-independently, enable long polling.
• Configuration is exact; aggregate state is approximate. AWS's naming tells you which is which.
• Competing consumers scale by simply starting more processes — no partitions, no rebalancing.
• The in-flight quota (~120,000) is a real ceiling, and it presents differently under short and long polling.

← Previous: 02 — Introduction to SQS · Index: Course home · Next: 04 — Sending and Receiving Messages →