Delivery Semantics and Idempotency

Five modules have now ended with the same sentence: your consumer must be safe to run twice. This module is where that debt is paid.

It makes two claims, and the distinction between them is the entire subject:

📌 Exactly-once delivery is impossible. Not difficult — impossible, in any distributed system, for reasons that have nothing to do with AWS.

📌 Exactly-once effect is entirely achievable. It is ordinary engineering, and it costs about one extra database write per message.

Teams that conflate the two either waste effort chasing a guarantee that cannot exist, or assume they have one and ship double-charges.

1. What you will learn

• Distinguish at-most-once, at-least-once and exactly-once, and say which SQS provides
• Explain why exactly-once delivery is impossible, and why exactly-once effect is not
• Implement idempotency four different ways and choose between them
• Place transaction boundaries so a crash cannot produce a partial effect
• Handle the hard case: a side effect at a third party you do not control
• Explain why SQS deduplication is not idempotency

2. The three semantics

At-most-once — Simple: you might lose it, you will never see it twice. Technical: the receiver acknowledges before processing. A crash after acknowledgement loses the message. Example: fire-and-forget metrics, where a dropped datapoint is irrelevant.

At-least-once — Simple: you will definitely get it, sometimes twice. Technical: the receiver acknowledges after processing. A crash before acknowledgement causes redelivery. Example: SQS.

Exactly-once — Simple: precisely one delivery, guaranteed. Technical: would require atomic agreement between two independent systems about whether a message was processed. Not achievable in the delivery layer. Example: nothing. This is the point.

Notice that the first two differ by one line of code moved:

// At-most-once — acknowledge first
delete(msg);
process(msg);        // crash here → message lost forever

// At-least-once — acknowledge last
process(msg);        // crash here → message redelivered
delete(msg);

There is no third ordering. You acknowledge before the work or after it, and each choice picks a failure mode. SQS chose at-least-once, because losing work is almost always worse than repeating it.

🎯 Interview point. "Why does SQS give at-least-once rather than exactly-once?" The answer is not "AWS didn't implement it" — it is that the only alternative orderings produce at-most-once, and exactly-once is not available at this layer at all.

3. Why exactly-once delivery is impossible

Not an AWS limitation. A property of networks.

sequenceDiagram
    autonumber
    participant C as Consumer
    participant S as SQS

    C->>C: process message ✅ (work is done)
    C->>S: DeleteMessage
    Note over C,S: 🌩️ the response never arrives

    Note over C: The consumer now knows:<br/>"I sent a delete."<br/>It does NOT know whether<br/>SQS received it.

Reading the diagram. The consumer has two choices and no way to choose correctly:

• Retry the delete. If the original succeeded, harmless. If SQS never got it — fine. But if the consumer crashes now, the message is redelivered and the work happens twice.
• Assume success. If the delete was actually lost, the message is redelivered anyway.

There is no third option, and no amount of additional messaging fixes it — every acknowledgement of an acknowledgement has the same problem. This is the Two Generals problem: two parties communicating over a lossy channel can never reach certain common knowledge about a shared fact.

Why can't SQS just use a transaction? Because a transaction requires a single coordinator with authority over both participants. SQS cannot participate in your database's transaction, and your database cannot participate in SQS's. Distributed transaction protocols (two-phase commit) exist, are available in neither, and would trade the problem for a blocking coordinator that fails in worse ways.

What is achievable

Move the deduplication to where the information lives.

SQS cannot know that paymentId = P123 means the same real-world event as a message it delivered four minutes ago. Your code can. That asymmetry of knowledge is why the responsibility lives where it does — it is delegation, not an unfixed bug.

4. The failure, concretely

sequenceDiagram
    autonumber
    participant S as SQS
    participant C as Consumer
    participant G as Payment gateway
    participant D as Database

    S-->>C: msg: charge P123 £29.99 (count=1)
    C->>G: charge card
    G-->>C: ✅ charged £29.99
    C->>D: INSERT payment P123
    D-->>C: ok
    Note over C: 💥 OOM-killed before DeleteMessage

    Note over S: lease expires

    S-->>C: msg: charge P123 £29.99 (count=2)
    C->>G: charge card
    G-->>C: ✅ charged £29.99 AGAIN ⚠️
    Note over G: Customer has paid £59.98<br/>for a £29.99 order

Reading the diagram. Nothing malfunctioned. SQS behaved exactly as specified, the gateway behaved exactly as specified, and the consumer's only sin was dying at an inconvenient moment — which processes do.

📌 The window between "the effect happened" and "the message was deleted" cannot be closed. It can only be made harmless.

5. Idempotency

Idempotent — Simple: running it twice has the same result as running it once. Technical: an operation where f(f(x)) = f(x) — applying it repeatedly produces no additional change beyond the first application. Example: status = 'SHIPPED' is idempotent. attempts = attempts + 1 is not.

Choosing the key

Before any strategy, you need an identifier. Getting this wrong invalidates everything downstream.

The rule: the key must come from the message, and be identical across redeliveries.

📌 Prefer a business key. MessageId protects against redelivery; a business key protects against redelivery and a producer that sent the same intent twice — which happens on producer retries, at-least-once upstream hops, and SNS fanout duplicates (Module 20).

6. Four strategies

Strategy 1 — natural idempotency (free)

Make the operation a set, not a change.

// ❌ Not idempotent — each delivery adds
order.setAttempts(order.getAttempts() + 1);
inventory.decrement(sku, qty);

// ✅ Idempotent — each delivery converges on the same state
order.setStatus(SHIPPED);
inventory.setQuantity(sku, absoluteQuantityFromMessage);

This is the same principle that produces order-independence (Module 06 §5): write absolute state, not relative changes. One design decision buys safety against both duplicates and reordering.

Free when applicable. Not always applicable — you cannot "set" a payment.

Strategy 2 — unique constraint (cheapest real guard)

Let the database refuse the duplicate.

void handle(PaymentCommand cmd) {
    try {
        // UNIQUE(payment_id) — the constraint IS the guard
        jdbc.update("""
            INSERT INTO payments (payment_id, order_id, amount_cents, status)
            VALUES (?, ?, ?, 'PENDING')
            """, cmd.paymentId(), cmd.orderId(), cmd.amountCents());

    } catch (DuplicateKeyException e) {
        // Someone already claimed this paymentId — this is a replay.
        log.info("replay ignored paymentId={}", cmd.paymentId());
        return;                              // caller deletes the message
    }

    charge(cmd);                             // ⚠️ see §7 — this ordering has a subtle hole
}

Why this is good: no extra table, no read-then-write race, and the atomicity is enforced by the database rather than by your code being careful. The check and the claim are a single operation.

Postgres variant when you simply want "first write wins":

INSERT INTO events (event_id, user_id, payload)
VALUES (?, ?, ?)
ON CONFLICT (event_id) DO NOTHING

Strategy 3 — idempotency-key table (most general)

When the effect spans several tables, or you need to return the original result on a replay.

@Transactional                                   // ← one transaction for BOTH
void handle(PaymentCommand cmd) {
    try {
        idempotency.insert(cmd.paymentId(), Instant.now().plus(TTL));
    } catch (DuplicateKeyException e) {
        log.info("replay ignored paymentId={}", cmd.paymentId());
        return;
    }

    // Every effect below commits atomically with the idempotency record above.
    payments.record(cmd);
    ledger.debit(cmd.accountId(), cmd.amountCents());
    outbox.enqueue(new PaymentRecorded(cmd.paymentId()));
}

📌 The @Transactional boundary is the whole strategy. Both the idempotency record and every effect commit together, or neither does. §7 explains what breaks without it.

Strategy 4 — conditional write (DynamoDB and friends)

dynamo.putItem(PutItemRequest.builder()
        .tableName("payments")
        .item(Map.of(
            "paymentId", AttributeValue.fromS(cmd.paymentId()),
            "amount",    AttributeValue.fromN(String.valueOf(cmd.amountCents())),
            "status",    AttributeValue.fromS("RECORDED")))
        .conditionExpression("attribute_not_exists(paymentId)")   // ← the guard
        .build());
// throws ConditionalCheckFailedException on a replay

The same idea as a unique constraint, in a store that has no constraints. An optimistic-concurrency version column achieves the same thing in SQL.

Choosing

7. Transaction boundaries — where it actually goes wrong

Most idempotency bugs are not missing guards. They are correctly written guards on the wrong side of a commit.

The broken version

// ❌ TWO transactions — the guard has a hole exactly as wide as the gap
void handle(PaymentCommand cmd) {
    if (idempotency.exists(cmd.paymentId())) return;     // transaction 1: read
    charge(cmd);                                          // the effect
    idempotency.insert(cmd.paymentId());                  // transaction 2: write
}

Two independent failures:

Race. Two consumers receive the same message (visibility overrun, Module 08 §5). Both execute exists before either executes insert. Both see false. Both charge.

Crash. The consumer charges the card and dies before the insert. The record was never written, so the redelivery sees false and charges again. The guard is present and useless.

The working version

// ✅ ONE transaction — the claim and the effect are inseparable
@Transactional
void handle(PaymentCommand cmd) {
    try {
        idempotency.insert(cmd.paymentId(), ...);   // claim — fails if already taken
    } catch (DuplicateKeyException e) {
        return;
    }
    recordPayment(cmd);                              // effect, same transaction
}

The insert is simultaneously the check and the claim — one atomic operation, so there is no window between them. A crash anywhere rolls the whole thing back, and the redelivery starts cleanly.

📌 The rule: the idempotency record and the effect must commit atomically. If they are in two transactions, you have reintroduced the window you were trying to close.

8. The hard case: external side effects

Strategy 3 works because both the guard and the effect live in one database. A payment gateway does not.

@Transactional
void handle(PaymentCommand cmd) {
    idempotency.insert(cmd.paymentId());
    gateway.charge(cmd);        // ⚠️ NOT part of your transaction.
                                //    If the transaction rolls back afterwards,
                                //    the money has still moved.
}

You cannot roll back a third party. Three techniques, in order of preference.

Technique 1 — use the provider's idempotency key (always, if available)

Every serious payment API supports this: Stripe, Adyen, Braintree, PayPal.

gateway.charge(ChargeRequest.builder()
        .amount(cmd.amountCents())
        .currency(cmd.currency())
        .idempotencyKey(cmd.paymentId())   // ← the provider deduplicates
        .build());

The provider returns the original result for a repeated key rather than charging again. You have pushed the guard to the only party that can enforce it authoritatively.

📌 If your external dependency supports idempotency keys, use them. This is the correct answer and everything below is a workaround for providers that do not.

Technique 2 — record intent before, outcome after

void handle(PaymentCommand cmd) {
    // 1. Durably record that we are ABOUT to act. Commits before anything external.
    var attempt = attempts.begin(cmd.paymentId());     // status = IN_PROGRESS
    if (attempt.isAlreadyComplete()) return;           // replay of finished work

    if (attempt.isAlreadyInProgress()) {
        // A previous attempt started and did not finish. We do not know whether
        // the charge went through. Do NOT retry blindly.
        throw new NeedsReconciliationException(cmd.paymentId());
    }

    // 2. External call
    var result = gateway.charge(cmd);

    // 3. Durably record the outcome
    attempts.complete(cmd.paymentId(), result.transactionId());
}

This does not eliminate the ambiguity — it converts an invisible ambiguity into a visible one. An IN_PROGRESS record older than some threshold is a known-uncertain case, surfaced for reconciliation, rather than a double charge nobody noticed.

Technique 3 — reconcile

For a gateway with neither idempotency keys nor a lookup API, query it before acting:

if (gateway.findChargeByReference(cmd.paymentId()).isPresent()) return;

⚠️ This is a read-then-write race — two consumers can both find nothing. It reduces the duplicate rate substantially but does not eliminate it. Combine with technique 2, and accept that a reconciliation process is part of the design.

🎯 Interview point. "How do you prevent a duplicate payment?" A strong answer names the layers: the provider's idempotency key first; a local atomic guard second; explicit intent/outcome records for the ambiguous window; and a reconciliation process for what remains. A weak answer says "check if we already processed it", which is the broken §7 code.

9. TTL — the detail that gets skipped

Idempotency records cannot live forever and cannot be short-lived.

TTL must exceed the longest possible redelivery window:

    maxReceiveCount × visibilityTimeout     (all retries)
  + message retention period                (it could sit in the queue)
  + DLQ retention                           (if you replay from the DLQ — Module 11)
  + a safety margin

Worked:

maxReceiveCount     = 5
visibilityTimeout   = 5 min       →  25 min of retries
retention           = 4 days
DLQ retention       = 14 days     →  replay is possible up to 18 days later
                                     ───────────────────────────────────
TTL                                = 30 days

⚠️ Too short and a DLQ replay three weeks later sails straight past an expired guard and charges the customer again — the exact failure the guard existed to prevent, occurring at the moment of an incident recovery.

⚠️ No TTL at all and the table grows without bound. At 10,000 messages/sec that is 864 million rows a day.

DynamoDB TTL, a Postgres partition drop, or a scheduled DELETE WHERE expires_at < now() all work. Pick one deliberately rather than discovering the problem later.

10. Production considerations

Duplicates are a metric, not an alarm. They are contractually expected. Count them; alarm on the rate rising, which signals a real upstream problem.

Log ApproximateReceiveCount on every receive. A shift in its distribution is your earliest warning that processing time has crept past the visibility timeout.

Idempotency costs one write per message. Budget it in the throughput math (Module 15) — at 10,000 msg/sec that is 10,000 extra writes per second, and it may well be your binding constraint.

Test it deliberately. Deliver the same message twice in an integration test and assert one effect (Lab 14). Idempotency that has never been exercised is a hypothesis.

Multi-system effects are the genuinely hard case. When one message must update a database and call a third party and publish an event, no single transaction covers it. That is the outbox pattern and sagas (Module 24).

Idempotency does not give you order-independence. They are different properties from a similar design principle (Module 06 §13). A handler can be perfectly idempotent and still reach a wrong final state if messages arrive reordered.

11. Common mistakes

Assuming FIFO removes the need for idempotency. Why it's wrong: FIFO deduplicates sends within 5 minutes. Consumer redelivery after a crash is untouched (Module 07 §3). Instead: idempotency on every queue type.

Check-then-act across two transactions. Why it's wrong: two consumers both see "not processed"; or a crash between the effect and the record leaves no evidence. Instead: one atomic operation — a unique constraint or a conditional write.

Generating the idempotency key at receive time. Why it's wrong: a redelivery generates a different key, so the guard never matches. Instead: derive it from the message, preferably from a business identifier.

Using the receipt handle as the key. Why it's wrong: it changes on every receive by design. Instead: MessageId, or better, a business key.

Idempotency records with no TTL, or too short a TTL. Why it's wrong: unbounded growth, or late duplicates escaping — including on DLQ replay, which is exactly when you can least afford it. Instead: TTL longer than retries + retention + DLQ retention.

Treating every duplicate as an incident. Why it's wrong: expected behaviour; paging on it trains people to ignore pages. Instead: count them, alarm on the rate.

Putting an external call inside a database transaction. Why it's wrong: the transaction can roll back; the external effect cannot. It also holds a database connection for the duration of a network call. Instead: the provider's idempotency key, plus intent/outcome records.

12. Real-world example

The double-charge, worked through three versions.

Version 1 — the incident

void handle(PaymentCommand cmd) {
    var result = gateway.charge(cmd.cardToken(), cmd.amountCents());
    payments.insert(cmd.paymentId(), result.transactionId(), "COMPLETED");
}

A deploy at 14:20 rolled the consumer fleet with no graceful shutdown (Module 22). Forty-one consumers were terminated mid-charge. Each had already called the gateway; none had deleted its message.

At 14:25 the leases expired and all 41 messages were redelivered. 41 customers were charged twice, totalling £1,847. The support cost exceeded the refunds.

What the logs showed: nothing unusual. Every gateway call returned 200. Every database insert succeeded. The only signal was ApproximateReceiveCount = 2, which they were not logging.

Version 2 — the guard, still broken

void handle(PaymentCommand cmd) {
    if (payments.existsById(cmd.paymentId())) return;      // ← transaction 1
    var result = gateway.charge(cmd.cardToken(), cmd.amountCents());
    payments.insert(cmd.paymentId(), result.transactionId(), "COMPLETED");   // ← transaction 2
}

Better — it survives a slow redelivery. It does not survive the case that actually happened: the crash occurs between the gateway call and the insert, so the redelivery finds no record.

It also fails under visibility overrun, where two consumers run existsById concurrently and both see false.

This version is the most dangerous of the three, because it looks correct and passes every obvious test.

Version 3 — correct

@Transactional
public void handle(PaymentCommand cmd) {

    // 1. Claim atomically. Insert IS the check — no window.
    try {
        payments.insertPending(cmd.paymentId(), cmd.orderId(), cmd.amountCents());
    } catch (DuplicateKeyException e) {
        var existing = payments.findById(cmd.paymentId());
        if (existing.status() == IN_PROGRESS && existing.olderThan(RECONCILE_AFTER)) {
            // A previous attempt reached the gateway and never came back.
            // We must not guess. Surface it.
            throw new NeedsReconciliationException(cmd.paymentId());
        }
        log.info("replay ignored paymentId={} receiveCount={}",
                 cmd.paymentId(), cmd.receiveCount());
        return;                                   // caller deletes the message
    }

    // 2. External call carries the provider's idempotency key.
    //    Even if we somehow arrive here twice, the gateway charges once.
    var result = gateway.charge(ChargeRequest.builder()
            .cardToken(cmd.cardToken())
            .amountCents(cmd.amountCents())
            .idempotencyKey(cmd.paymentId())      // ← the real protection
            .build());

    // 3. Record the outcome, same transaction as the claim.
    payments.complete(cmd.paymentId(), result.transactionId());
}

Three layers of defence, and it is worth naming why each exists:

What else changed alongside the code, because the code was never the whole fix:

• Graceful shutdown (Module 22) — removing the cause, not just the symptom
• ApproximateReceiveCount on every log line — so the next occurrence is visible in minutes
• A duplicate-delivery integration test in CI
• Idempotency TTL set to 30 days, computed as in §9

🎯 The transferable lesson: the incident was caused by a deploy, not by SQS. Idempotency made it harmless; graceful shutdown made it rare. You want both, and a team that fixes only one has fixed half of it.

13. Interview questions

🟢 Beginner

What does at-least-once delivery mean? Every message is delivered a minimum of one time, with no guaranteed maximum. Duplicates are normal and expected.

What is idempotency? An operation is idempotent if performing it twice has the same effect as performing it once. status = 'SHIPPED' is idempotent; attempts = attempts + 1 is not.

🟡 Intermediate

Why can SQS deliver the same message twice? Two mechanisms. Redelivery: a consumer received the message and did not delete it before the visibility timeout expired — usually a crash, an overrun, or a deploy. And replication: a delete that did not reach every storage replica before one of them re-served the message. They are distinguished by ApproximateReceiveCount.

What is at-most-once, and when would you want it? Acknowledging before processing, so a crash loses the message rather than repeating it. Appropriate only when loss is cheaper than duplication — fire-and-forget metrics, for instance. SQS does not offer it directly; you would implement it by deleting before processing, which is almost always a bug rather than a choice.

🔴 Advanced

Why is exactly-once delivery impossible? Is exactly-once processing impossible too?

Exactly-once delivery is impossible because of the Two Generals problem. After a consumer sends DeleteMessage, a lost response leaves it unable to determine whether the delete was received. It must either retry (risking a duplicate if it crashes) or assume success (risking a redelivery if the delete was lost). No additional round of acknowledgement helps, because the acknowledgement has the same problem.

Exactly-once effect is achievable, and the distinction matters. SQS cannot know that two messages represent the same real-world event; your consumer can, because it understands the business meaning. Moving deduplication to the consumer — via an idempotency key committed atomically with the effect — produces exactly-once outcomes on top of at-least-once delivery.

So: the guarantee cannot exist in the delivery layer, and does not need to.

How would you prevent a duplicate payment? Walk through the transaction boundaries.

Layered, in order:

1. The provider's idempotency key. Pass a stable business identifier as the gateway's idempotency key. The provider then deduplicates authoritatively — it is the only party that can.
2. A local atomic claim. Insert a row keyed on paymentId in the same transaction as recording the payment. The insert is both the check and the claim, so there is no window between them, and a DuplicateKeyException means replay.
3. Explicit intent/outcome records for the ambiguous case. If an attempt reached IN_PROGRESS and never completed, you genuinely do not know whether the charge went through. Surface it for reconciliation rather than guessing.

The critical boundary is that the idempotency record and the effect commit together. Two separate transactions reintroduce exactly the window you were closing — and the external call must sit outside the transaction entirely, since you cannot roll back a third party.

Your idempotency check and your effect are in separate transactions. What can go wrong?

Two distinct failures. A race: two consumers holding the same message — which happens whenever processing overruns the visibility timeout — both run the check before either runs the write, both see "not processed", both act. And a crash: the consumer performs the effect and dies before writing the idempotency record, so the redelivery finds no evidence and performs the effect again.

The fix is to make the check and the claim one atomic operation — a unique-constraint insert, an ON CONFLICT DO NOTHING, or a conditional write — committed in the same transaction as the effect.

Why is FIFO deduplication not idempotency? It is a 5-minute window on the send side: a producer retrying SendMessage with the same deduplication id does not create a second message. It provides nothing on the receive side. A consumer that processes and then crashes before deleting will receive the message again on a FIFO queue exactly as on a standard one. Different halves of the system, different problems.

⚫ System design

Design an order pipeline where the same message may be delivered three times and the customer must be charged exactly once.

1. Establish the identity. A producer-generated paymentId, created once when the order is placed and carried in the message body. Not MessageId — that would not catch a producer retry creating two messages for one order.

2. Choose the guard by effect location.

3. Get the boundaries right.

BEGIN
  INSERT payments(payment_id, status=PENDING)   -- claims; throws on replay
  INSERT outbox(PaymentInitiated)
COMMIT
  gateway.charge(idempotencyKey = payment_id)   -- outside the transaction
BEGIN
  UPDATE payments SET status=COMPLETED, txn_id=…
COMMIT

The external call sits between two transactions deliberately — it cannot be rolled back, so it must not be inside one.

4. Handle the ambiguous window. A PENDING row older than a threshold means an attempt reached the gateway and never returned. Do not retry blindly. Reconcile against the gateway, or surface it to an operator.

5. TTL the records longer than retries + retention + DLQ retention — around 30 days for typical settings — so a DLQ replay weeks later still hits the guard.

6. Reduce the rate as well as the harm. Idempotency makes duplicates safe; these make them rare: graceful shutdown on deploy, a visibility timeout sized from the p99, and a maxReceiveCount low enough that poison messages quarantine quickly.

7. Prove it. An integration test that delivers the same message three times and asserts exactly one gateway call and one ledger entry. Plus a duplicate-rate metric in production, alarmed on the rate rather than on any single occurrence.

Follow-ups to expect: "What if the gateway has no idempotency key?" — intent/outcome records plus a reconciliation job, and accept a small ambiguous set. "What if the effects span three services?" — that is a saga with compensating transactions, and the compensations are business logic (Module 24). "How do you test the crash window?" — inject a failure between the effect and the delete; that is Lab 14.

14. Summary

• SQS gives at-least-once. At-most-once and at-least-once differ only by whether you acknowledge before or after the work; there is no third ordering.
• Exactly-once delivery is impossible — the Two Generals problem, not an AWS limitation.
• Exactly-once effect is routine. Deduplication belongs in the consumer, which is the only party that knows the business meaning.
• The key must come from the message. Prefer a business identifier over MessageId; never the receipt handle.
• Four strategies: natural idempotency, unique constraint, idempotency-key table, conditional write.
• The idempotency record and the effect must commit atomically. Two transactions reopen the window.
• External side effects need the provider's idempotency key, plus intent/outcome records for the ambiguous case.
• TTL the records longer than retries + retention + DLQ retention.
• FIFO deduplication is not idempotency — different half of the system.
• Idempotency makes duplicates harmless; graceful shutdown and a well-sized timeout make them rare. You want both.

← Previous: 08 — Visibility Timeout · Index: Course home · Next: 10 — Retries →