SQS and AWS Lambda

The integration everyone reaches for first. The key fact, which the word "trigger" actively obscures: the Lambda service polls SQS on your behalf. SQS still does not push. Everything surprising about this integration follows from that one sentence.

1. What you will learn

• Explain precisely which component polls the queue
• Configure an event source mapping and predict its scaling behaviour
• Implement partial batch failure with ReportBatchItemFailures and know why the default is dangerous
• Size visibility timeout relative to function timeout
• Decide when Lambda is the wrong consumer

2. Why this concept exists

• Lambda removes the poll loop, the worker pool, the scaling policy and the servers
• In exchange you accept its batching semantics, its concurrency model and its failure handling — none of which are obvious, and all of which have sharp edges

3. Beginner explanation

• You point Lambda at a queue and your function runs whenever messages arrive
• AWS does the polling for you
• If your function succeeds, the messages are deleted automatically

4. How it actually works

• The Lambda service runs pollers that call ReceiveMessage, then invoke your function synchronously with a batch, then call DeleteMessage for the batch on success
• Default scaling: starts at 5 concurrent invocations, scales by up to +300/minute, ceiling 1,250 per event source mapping. Idle scales back to 5, optimising to 2
• MaximumConcurrency (2–1,000) caps an event source; distinct from function reserved concurrency
• Provisioned mode: dedicated pollers, min 2–200 / max 2–10,000, scales up to 1,000 concurrency/minute, up to 100,000 concurrent invokes. Mutually exclusive with MaximumConcurrency
• Batching: default batch size 10, MaximumBatchingWindowInSeconds up to 5 minutes; invocation fires on window expiry, payload-size limit, or batch size. ⚠️ On a low-traffic queue Lambda may wait up to 20 s regardless of a smaller window
• Failure, by default: the function throwing makes the entire batch visible again — messages that succeeded are reprocessed. This is the single biggest Lambda+SQS footgun
• ReportBatchItemFailures: return {"batchItemFailures": [{"itemIdentifier": "<messageId>"}]} so only the listed messages return. Requires the ESM function-response type to be set
• Visibility timeout must be ≥ function timeout; AWS guidance is at least 6×
• DLQ: configure it on the queue, not on the function — a function-level DLQ does not apply to SQS event sources
• FIFO: batches may span groups, order is preserved within a group, concurrency = min(active groups, MaximumConcurrency)
• Poller ReceiveMessage calls are billed SQS requests even when empty (Module 18)

5. Diagram

• Diagram 38 — sequence showing the Lambda service polling, invoking, and deleting (emphasising who does what)
• Batch failure: default whole-batch return vs ReportBatchItemFailures
• Scaling curve: 5 → +300/min → 1,250

6. Step-by-step flow

• Create the queue and the function
• Create an event source mapping (queue ARN → function)
• Lambda's pollers begin long-polling the queue
• Messages appear; a poller receives up to BatchSize
• Lambda invokes the function synchronously with the batch
• Success → Lambda deletes the whole batch
• Failure → whole batch returns, unless ReportBatchItemFailures narrows it
• After maxReceiveCount, the queue's redrive policy moves messages to the DLQ

7. Configuration

• BatchSize, MaximumBatchingWindowInSeconds, FunctionResponseTypes: [ReportBatchItemFailures]
• ScalingConfig.MaximumConcurrency or ProvisionedPollerConfig — not both
• Function timeout, and queue visibility timeout ≥ 6× that
• Event filtering to discard uninteresting messages before invocation (and before paying for it)
• All numbers → aws-facts.md §8

8. Production considerations

• Always enable ReportBatchItemFailures. Without it, one bad message in a batch of 10 causes 9 needless reprocessings — and with a non-idempotent consumer, 9 duplicate side effects
• Lambda can overwhelm your database. 1,250 concurrent functions × 1 connection each is 1,250 connections. Use MaximumConcurrency, or RDS Proxy, or don't use Lambda
• Throttling looks like failure: if the account concurrency limit is hit, invocations are throttled, messages return, receive counts climb, and the DLQ fills with perfectly good messages
• Cost crossover: at sustained high throughput, an always-on ECS consumer is usually cheaper than Lambda. Lambda wins on spiky and low-volume workloads
• Long-running work does not fit — the 15-minute function ceiling is hard. Use ECS or Step Functions
• Event filtering can cut invocation count and cost substantially on queues with mixed traffic

9. Common mistakes

• Believing SQS pushes to Lambda. Why → it explains nothing about scaling or batching. Instead → the Lambda service polls
• Not enabling ReportBatchItemFailures. Why → whole-batch reprocessing. Instead → enable it and return the failed ids
• Visibility timeout equal to function timeout. Why → no headroom for retries within the window. Instead → ≥ 6×
• DLQ configured on the function. Why → it does not apply to SQS event sources. Instead → redrive policy on the queue
• Ignoring downstream connection limits. Why → 1,250 concurrent functions. Instead → cap concurrency
• Using Lambda for 10-minute jobs. Why → 15-minute hard ceiling, and retries re-run the whole thing. Instead → ECS/EKS

10. Real-world example

• An image-processing pipeline: S3 → SQS → Lambda. Show the batch failure bug in its natural habitat, then the fix, then the point at which the workload outgrows Lambda

11. Interview questions

• 🟢 How do you trigger a Lambda function from SQS?
• 🟡 Which component polls the queue — SQS or Lambda?
• 🟡 What happens by default when one message in a batch fails?
• 🔴 How does Lambda scale for an SQS event source? Give the actual numbers.
• 🔴 Why must visibility timeout exceed function timeout, and by how much?
• 🔴 Your Lambda consumers are exhausting the database connection pool. What are your options?
• ⚫ When would you choose ECS over Lambda for an SQS consumer? Argue with cost and operational numbers.

12. Summary

• The Lambda service polls; SQS never pushes
• 5 → +300/min → 1,250 concurrency per ESM; provisioned mode goes much further
• Enable ReportBatchItemFailures — the default is whole-batch retry
• Visibility timeout ≥ 6× function timeout; DLQ goes on the queue
• Lambda suits spiky and short work; sustained high throughput usually belongs on ECS

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• event source mapping
• ReportBatchItemFailures
• MaximumBatchingWindowInSeconds
• provisioned mode
• MaximumConcurrency
• partial batch response

Diagrams to build:

• Diagram 38 — who polls
• Partial batch failure comparison
• Lambda scaling curve

Code samples:

• Java Lambda handler returning SQSBatchResponse with failed item identifiers
• Creating an ESM with batch size, window and FunctionResponseTypes (CLI + IaC)
• Configuring MaximumConcurrency and provisioned mode
• An event filter pattern

Mandatory "why?" answers:

• Why does the whole batch retry when one message fails?
• Why must the visibility timeout be several times the function timeout?
• Why does 'SQS triggers Lambda' misdescribe what happens?

Facts to pull from _reference/aws-facts.md: 5 initial / +300 per min / 1,250 max, MaximumConcurrency 2–1,000, provisioned mode limits, batch window up to 5 min, 20s low-traffic caveat, 6x visibility guidance

← Previous: 18 — Cost · Index: Course home · Next: 20 — SNS + SQS Fanout →