Standard Queues

The default queue type, and the one you should use unless you can prove otherwise.

Module 03 explained the mechanism — distributed storage with no global coordinator. This module turns that mechanism into a contract: three guarantees, stated precisely, with the fine print. Knowing exactly what you are promised is what lets you design against it instead of hoping.

1. What you will learn

• State the three guarantees of a standard queue precisely
• Distinguish the two entirely different causes of duplicate processing
• Explain what "best-effort ordering" permits and forbids
• Read ApproximateNumberOfMessagesVisible and NotVisible correctly, and know when the second one is an emergency
• Decide when a standard queue is the right choice — and push back when someone reaches for FIFO

2. The contract

A standard queue promises exactly three things.

Read the right-hand column twice. Every production surprise in this module lives there.

📌 The contract in one sentence: you will get every message, possibly more than once, roughly in order.

3. Guarantee 1 — unlimited throughput

AWS documents standard queues as supporting "a very high, nearly unlimited number of API calls per second, per action" (aws-facts.md §4). There is no published per-queue ceiling, no partition count to choose, and no capacity to provision.

This is genuinely unusual and worth appreciating. Compare:

Why can SQS do this when Kafka cannot? Because Kafka's partition is the unit of ordering, and ordering requires a single writer per partition. SQS standard gave up ordering, so it has no such unit, and no reason to expose one. Unlimited scale is what best-effort ordering bought. FIFO takes the ordering back, and the partition limit reappears immediately (Module 07).

But there is a ceiling, and it is yours

The queue scales. Three things around it do not:

1. Your consumers — the usual actual bottleneck (Module 14)
2. Your downstream — database connections, third-party rate limits
3. The in-flight quota — see §7, and it catches people out

4. Guarantee 2 — at-least-once delivery

At-least-once delivery — Simple: you will definitely get the message, and sometimes you will get it twice. Technical: SQS guarantees each message is delivered a minimum of one time, with no guaranteed upper bound on deliveries. Example: a payment message delivered twice because the first consumer crashed before deleting.

Two causes, and they are not the same

People treat "duplicates" as one phenomenon. There are two, with different frequencies, different signatures and different fixes. Being able to tell them apart is a genuine diagnostic skill.

flowchart TB
    START["A message is processed twice"]
    START --> Q{"ApproximateReceiveCount<br/>on the second delivery?"}

    Q -->|"1"| A["<b>Cause A — replication</b><br/>A delete did not reach every replica<br/>before one of them re-served it"]
    Q -->|"2 or more"| B["<b>Cause B — redelivery</b><br/>Visibility timeout expired without a delete:<br/>crash, overrun, or a failed deploy"]

    A --> AF["<b>Frequency:</b> rare<br/><b>Fix:</b> idempotency — nothing else can help<br/><b>Module:</b> 03 §6"]
    B --> BF["<b>Frequency:</b> common, and mostly<br/>under your control<br/><b>Fix:</b> idempotency, plus size the<br/>timeout, plus graceful shutdown<br/><b>Modules:</b> 08, 22"]

Reading the diagram. One field distinguishes them, and it is free: ApproximateReceiveCount, which you should already be logging on every receive (Module 04 §9).

Cause A — replication. The message lives on several servers. A DeleteMessage propagates to them asynchronously. If one replica does not receive it before serving the message again, you get a second delivery with a receive count of 1 — because from that replica's point of view, this is the first time. Rare. Nothing you configure affects it.

Cause B — redelivery. The visibility timeout expired without a delete. The consumer crashed, or processing took longer than the timeout, or a deploy killed the worker mid-message. The receive count is 2 or more, because SQS knows this is a second handout. Common, and largely within your control.

🎯 Interview point. "How would you tell whether a duplicate came from redelivery or from replication?" — ApproximateReceiveCount. It is a two-word answer that signals you have actually debugged this rather than read about it.

The important consequence

Both causes lead to the same place:

📌 Your consumer must be safe to run twice. Not "should be". Must. This is not defensive programming, it is a requirement of the contract you accepted by using a queue.

At a million messages a day, "rare" happens daily. Module 09 is entirely about making it harmless.

5. Guarantee 3 — best-effort ordering

flowchart LR
    subgraph S["Producer sends"]
        direction TB
        A1["m1"] --> A2["m2"] --> A3["m3"] --> A4["m4"] --> A5["m5"]
    end

    S --> Q[("Standard queue<br/>no global sequencer")]
    Q --> R

    subgraph R["A consumer may see"]
        direction TB
        B1["m2"] --> B2["m1"] --> B3["m5"] --> B4["m3"] --> B5["m4"]
    end

Reading the diagram. The right-hand order is not a malfunction — it is a permitted outcome. SQS makes a genuine attempt at order, and usually succeeds. "Usually" is the problem.

Why this is more dangerous than it sounds

Best-effort ordering is a trap for exactly the reason it is well-behaved: at low volume, with one consumer, on a developer's laptop, messages arrive in order every single time. The bug ships because the test environment cannot produce it.

Then production has forty consumers, and:

• Two consumers poll simultaneously and finish in the wrong order
• One consumer is briefly slower than another
• A message fails and is retried, arriving long after its successors
• And the explicitly documented case below

⚠️ On a standard queue with maxReceiveCount > 3, a message received 3 or more times without being deleted is moved to the back of the queue (aws-facts.md §7).

So a struggling message does not merely arrive late — it is deliberately sent to the end of the line (Module 05 §8).

Designing around it

The rule: never let delivery order carry meaning. Put the ordering information in the data.

The right-hand column is worth internalising, because it is the same technique that makes messages idempotent. Order-independence and idempotency are two faces of the same design: both come from writing absolute state rather than relative changes.

📌 If a message says "add 5", order and duplicates both break you. If it says "set to 47", neither can.

6. Reading the metrics

Four counters describe a standard queue's state. Their combination is the diagnosis.

All are approximate, at one-minute granularity (Module 03 §5). Alarm on sustained conditions, never on a single datapoint or an exact value.

The signatures

Visible and NotVisible read together tell you which part of the system is at fault:

📌 NotVisible rising while Visible stays flat is the one to learn. It means work is going in and not coming out — and it is invisible to anyone watching only queue depth. Module 17 builds this into a full triage table.

7. The in-flight quota

The ceiling that surprises people, because it presents as nothing happening at all.

In-flight messages — messages received by a consumer but not yet deleted. A standard queue supports approximately 120,000 (aws-facts.md §3).

120,000 sounds enormous. It is reachable, and the way you reach it is almost always a bug: consumers that receive messages and neither delete them nor crash — a hung handler, a swallowed exception with no delete, an in-process retry loop that never terminates.

What happens at the quota depends on your polling mode, and this is a genuinely confusing asymmetry:

Since you should be using long polling everywhere (Module 12), the realistic failure looks like: the queue has a large backlog, consumers are running and polling happily, and no messages are being delivered. There is no error anywhere.

📌 The detector is an alarm on ApproximateNumberOfMessagesNotVisible approaching 120,000. Without it, this failure is nearly undiagnosable from the outside.

Why does a quota exist at all? Every in-flight message is state SQS must track — which handle was issued, when its lease expires, how many times it has been received. That tracking is bounded. The quota is where AWS drew the line, and it is raisable via Service Quotas if you have a genuine need.

8. Fair queues

A newer capability that solves a real multi-tenant problem without FIFO's cost.

The problem. One queue, many tenants. Tenant A submits 500,000 messages at 09:00. Tenants B through Z now wait behind them — their messages are fine, just at the back of a very long line. One noisy neighbour has degraded everyone.

The classic fixes, both with real costs:

• A queue per tenant — true isolation (a bulkhead, Module 24), but queue sprawl and per-tenant operational overhead once you have thousands
• FIFO with MessageGroupId = tenantId — gives fairness, but drags in FIFO's throughput limits and ordering semantics you did not ask for

Fair queues let you supply a MessageGroupId on a standard queue purely to signal tenancy. SQS uses it to mitigate starvation between groups, and you keep standard's unlimited throughput (aws-facts.md §6).

sqs.sendMessage(r -> r
        .queueUrl(queueUrl)
        .messageBody(body)
        .messageGroupId(tenantId));   // on a STANDARD queue — fairness, not ordering

Two things to know:

⚠️ MessageGroupId on a standard queue does not give you ordering. It is a fairness hint. If you find yourself hoping it orders things, you want FIFO (Module 07).

⚠️ It is billed at both fair-queue and standard rates (aws-facts.md §9). Worth modelling before enabling it fleet-wide (Module 18).

AWS recommends including a MessageGroupId on all messages when using fair queues — a queue where some messages are grouped and others are not has no consistent notion of fairness.

9. Choosing standard

The decision, stated plainly:

flowchart TD
    Q1{"Does the ORDER of messages<br/>change the correctness<br/>of the result?"}
    Q1 -->|No| STD["✅ <b>Standard</b>"]
    Q1 -->|Yes| Q2{"Can you make processing<br/>order-independent?<br/>(versions, upserts,<br/>absolute state)"}
    Q2 -->|Yes| STD
    Q2 -->|No| Q3{"Does ordering apply<br/>globally, or per entity?"}
    Q3 -->|"Per entity<br/>(account, customer, order)"| FIFO["⚠️ <b>FIFO</b><br/>MessageGroupId = that entity<br/>Module 07 — mind the throughput"]
    Q3 -->|"Globally, at volume"| HARD["🛑 <b>Rethink</b><br/>Global ordering means serial<br/>processing. Nothing does this<br/>well at scale"]

    STD --> IDEM["Either way:<br/><b>idempotency is still required</b>"]
    FIFO --> IDEM

Reading the diagram. Note where every path ends. Choosing FIFO does not remove the need for idempotency — a point Module 07 devotes a section to, because the AWS documentation's phrase "exactly-once processing" persuades people otherwise.

🎯 Interview point. The strongest answer to "standard or FIFO?" starts by questioning the premise: most workloads that think they need ordering need per-entity ordering, and many need none at all once processing is made order-independent. Reaching for FIFO reflexively costs throughput you cannot get back.

10. Production considerations

Assume duplicates from day one. Build the idempotency guard before the feature ships. Retrofitting it means auditing every side effect in every handler — expensive, and done under time pressure after an incident.

Assume reordering. If step B depends on step A, encode that in the data, not in the delivery. Version numbers, upserts and absolute state are the tools.

Alarm on in-flight approaching the quota. The long-polling failure mode — no messages, no errors — is otherwise close to undiagnosable.

Watch the Received vs Deleted ratio. NumberOfMessagesReceived significantly exceeding NumberOfMessagesDeleted over a window means messages are being redelivered. It is the earliest warning that a downstream is degrading — earlier than depth, earlier than the DLQ filling (Module 17).

Fair queues matter for shared multi-tenant queues. If one customer can flood a queue that serves all customers, you have a noisy-neighbour problem whether or not you have noticed it yet.

A standard queue is almost never the bottleneck. When throughput is disappointing, look at your consumers and your downstream first (Module 15). The queue scaling is the one part of the system you did not have to size.

11. Common mistakes

Choosing FIFO to avoid duplicates. Why it's wrong: FIFO deduplicates sends within a 5-minute window. It does nothing about a consumer crashing after processing and before deleting — which is the common cause. Instead: standard plus idempotency. If you also need ordering, FIFO plus idempotency.

Relying on order because it works in development. Why it's wrong: low volume with one consumer hides reordering completely. Instead: design order-independently, and test with concurrent consumers.

Reading NotVisible as "stuck". Why it's wrong: in-flight work is normal — it is what a busy queue looks like. Instead: the shape matters. Rising and never falling is the problem; rising and falling is health.

Ignoring the in-flight quota. Why it's wrong: under long polling you hit it silently and delivery simply stops. Instead: alarm on NotVisible approaching 120,000.

Using MessageGroupId on a standard queue and expecting ordering. Why it's wrong: it is a fairness mechanism, not an ordering one. Instead: FIFO if you need ordering.

Treating every duplicate as an incident. Why it's wrong: duplicates are contractually expected, and paging on expected behaviour trains people to ignore pages. Instead: count them; alarm on the rate rising, which signals a real upstream problem.

12. Real-world example

An analytics event pipeline. 20,000 events/sec sustained, 60,000/sec at peak. Each event is a user interaction; each is written to a data warehouse.

Why standard is obviously right here:

• Ordering is meaningless. Two clicks arriving in either order produce the same analytics.
• FIFO could not do it anyway. Even in high-throughput mode, 60,000/sec would need careful partition design and is region-dependent (Module 07). Standard has no such conversation.
• The volume makes duplicates a daily certainty — 20,000/sec is 1.7 billion events a day.

What idempotency looks like when the workload is naturally suited to it:

// Each event carries a producer-generated eventId.
// The warehouse write is an upsert keyed on it.
//
//   INSERT INTO events (event_id, user_id, type, occurred_at, payload)
//   VALUES (?, ?, ?, ?, ?)
//   ON CONFLICT (event_id) DO NOTHING
//
// A duplicate delivery is a no-op at the database level.
// No lookup, no lock, no extra round trip — the constraint does the work.

This is the cheapest form of idempotency: the unique constraint strategy (Module 09). It costs one index and no application logic, and it works precisely because the event carries an identity the producer generated.

The bug they shipped anyway. A "sessionise these events" feature assumed events for a session arrived in order, and computed session duration as lastSeen - firstSeen by tracking a running value. Out-of-order arrival produced negative durations for roughly 0.3% of sessions — invisible in staging, obvious in a dashboard a month later.

The fix was two lines, and it is the §5 table in action: instead of tracking a running value, compute MIN(occurred_at) and MAX(occurred_at) over the stored events. Absolute rather than relative. Order-independent and duplicate-safe at the same time — one change bought both.

13. Interview questions

🟢 Beginner

What are the characteristics of a standard SQS queue? Nearly unlimited throughput, at-least-once delivery, and best-effort ordering. It is the default type and requires no capacity planning.

What does at-least-once delivery mean? Every message is delivered a minimum of one time, with no guaranteed maximum. Duplicates are normal and expected.

🟡 Intermediate

Why can a standard queue deliver duplicates? Describe the mechanism. Two distinct mechanisms. First, replication: messages are stored on multiple servers and a DeleteMessage propagates asynchronously, so a replica that misses the delete can serve the message again — this shows a receive count of 1. Second, and far more common, redelivery: the visibility timeout expired without a delete because the consumer crashed or overran, so SQS hands the message out again — receive count 2 or more.

What does "best-effort ordering" mean in practice? SQS attempts to preserve send order but guarantees nothing. In practice it is usually correct at low volume and frequently wrong at high concurrency — which makes it a trap, because development environments do not reproduce it. There is also an explicit reordering rule: after three receives without deletion, a message may be moved to the back of the queue.

What is the difference between Visible and NotVisible? Visible is the backlog — messages waiting to be picked up. NotVisible is in-flight work — received but not yet deleted. Read together they identify the failure class: Visible rising with NotVisible flat means consumers are not polling; both rising means consumers are too slow; NotVisible rising and never falling means messages are being received and never resolved.

🔴 Advanced

Your consumer is idempotent but the queue still causes problems when messages arrive out of order. What kind of workload is this, and what do you do?

Idempotency and order-independence are different properties, and this workload has one but not the other. Idempotency means processing the same message twice is safe. Order-independence means processing different messages in any order is safe. A handler that does balance = X is both. A handler that applies a sequence of state transitions can be perfectly idempotent per message and still reach the wrong final state if the transitions arrive reordered.

Three options, in order of preference:

1. Make it order-independent. Version each message and reject any version lower than the one you already hold; or restructure to write absolute state rather than deltas. Cheapest, and usually possible.
2. Buffer and sort. Accumulate messages for an entity over a short window, sort by a producer-assigned sequence number, then apply. Adds latency and a stateful component.
3. Move to FIFO with MessageGroupId = the entity. Correct, but you inherit the throughput limits and head-of-line blocking (Module 07). Choose this when ordering is genuinely intrinsic — a ledger, a state machine.

The diagnostic question to ask first: is ordering required globally, or only within one entity? Almost always the latter, which is what makes option 1 or 3 viable.

Why does a standard queue have no throughput limit when FIFO does? Because a throughput limit on a distributed queue comes from coordination, and coordination is what ordering requires. FIFO must serialise within a message group, which means a single writer per partition, which is a bounded rate. Standard gave up ordering and therefore needs no such serialisation point. The unlimited throughput is not a bonus feature — it is the direct payment for best-effort ordering.

⚫ System design

You are told to use FIFO for an events pipeline at 50,000 msg/s. Push back with numbers.

Start with the throughput arithmetic, because it may settle the question outright (aws-facts.md §4):

• FIFO without high-throughput mode: 300 TPS per action per partition; 3,000 messages/sec with batching. 50,000/sec is 16× over. Not close.
• With high-throughput mode: feasible in the largest Regions (up to 70,000 TPS non-batched), but it is region-dependent — in the smallest Regions the default is 2,400 TPS, so a design that works in us-east-1 fails in eu-west-2. That alone makes it a fragile choice for a multi-Region service.
• It also requires many message groups. Throughput scales with partitions, which derive from group ids. A design with few groups cannot reach the headline number regardless of mode.

Then question the requirement itself, which is the part that matters:

• Is ordering needed globally, or per entity? If per-user, MessageGroupId = userId gives the parallelism — but then ask whether per-user ordering is genuinely required, or whether the processing can be made order-independent (§5), which costs nothing.
• What breaks if two events are processed out of order? Frequently the honest answer is "nothing" — the requirement was inherited rather than derived.
• What does FIFO cost you? Head-of-line blocking (one failing message stalls its entire group), a DLQ that breaks the ordering guarantee you paid for, and lower throughput.

The recommendation: standard, with order-independent processing and idempotency on eventId. If some subset genuinely requires ordering, route that subset to a FIFO queue and leave the other 49,000/sec on standard. Splitting by requirement is almost always better than applying the strictest requirement to everything.

Follow-ups to expect: "What if we later need ordering?" — the migration is a new queue and a producer change, since the name must end .fifo; plan for it but do not pay for it now. "How do you know ordering isn't needed?" — write down the specific pair of events whose inversion would produce a wrong answer. If nobody can name one, you have your answer.

14. Summary

• The contract: unlimited throughput, at-least-once delivery, best-effort ordering.
• Two causes of duplicates, distinguished by ApproximateReceiveCount: replication (count 1, rare) and redelivery (count ≥ 2, common).
• Unlimited throughput is the payment for giving up ordering. FIFO buys ordering back and the limits return.
• Best-effort ordering is a trap because development environments never reproduce the failure. Put ordering in the data, not the delivery.
• Order-independence and idempotency are the same design: write absolute state, not relative changes.
• Visible and NotVisible read together name the failure class. NotVisible rising and never falling is the one to learn.
• The in-flight quota (~120,000) fails silently under long polling. Alarm on it.
• Fair queues give multi-tenant fairness via MessageGroupId without FIFO's throughput cost — fairness, not ordering, and billed at both rates.
• Choosing FIFO never removes the need for idempotency.

← Previous: 05 — The Message Lifecycle · Index: Course home · Next: 07 — FIFO Queues →