Event Sources: S3, EventBridge, API Gateway

The other ways messages get into a queue without a producer you wrote. Each integration is covered the same way: why use it, the architecture, the message flow, what happens on failure, and the operational gotcha that is not in the getting-started guide.

1. What you will learn

• Wire S3 event notifications to a queue and handle their delivery semantics
• Route EventBridge events to SQS and explain when EventBridge beats SNS
• Build an API Gateway → SQS integration with no Lambda in between
• Use SQS with Step Functions for long-running and callback patterns
• For each: predict the failure and retry behaviour

2. Why this concept exists

• Not every producer is code you control. Object uploads, AWS service events and raw HTTP requests all need to reach a queue
• Each source has different delivery guarantees and different failure behaviour — and assuming they are all the same is how events get silently lost

3. Beginner explanation

• Things that happen in AWS can drop a message into your queue automatically
• A file lands in S3 → a message appears. A rule matches an event → a message appears
• Your consumer does not care where it came from

4. How it actually works

• S3 → SQS: event notification config on the bucket; queue policy must allow s3.amazonaws.com with aws:SourceArn = bucket ARN. ⚠️ S3 notifications are designed for at-least-once delivery and, for some event types, are not guaranteed — use S3 Inventory or EventBridge for completeness-critical workflows. Event types (s3:ObjectCreated:*, Put vs CompleteMultipartUpload) matter more than people expect
• EventBridge → SQS: rule with an event pattern and an SQS target; richer content-based matching than SNS filter policies, plus archive and replay, schema registry, and third-party SaaS sources. Target failures go to an EventBridge DLQ
• When EventBridge over SNS: complex routing, replay, cross-account event buses, AWS-service events. When SNS over EventBridge: simple fanout, higher throughput, lower cost
• API Gateway → SQS: an AWS service integration — no Lambda, no cold start, no compute cost. Requires an IAM role for API Gateway and a request mapping template. Great for high-volume ingest endpoints that only need to enqueue
• Step Functions + SQS: SendMessage as a task state, and the .waitForTaskToken callback pattern where the workflow pauses until a consumer returns the token — the standard way to put human approval or a long external job inside a state machine
• Lambda → SQS (function as producer) is just SendMessage; note the pitfall of sending after a partial failure
• Every one of these is at-least-once. Idempotency (Module 09) applies to all of them

5. Diagram

• S3 → SQS → worker fleet
• EventBridge rule routing one event to several targets including SQS
• API Gateway → SQS direct integration (contrast with API GW → Lambda → SQS)
• Step Functions waitForTaskToken callback via SQS

6. Step-by-step flow

• S3: create queue → queue policy for s3.amazonaws.com with aws:SourceArn → bucket notification config → upload → message arrives with bucket/key, not the object body
• EventBridge: create rule with event pattern → add SQS target → EventBridge assumes a role or uses the queue policy → matching events are enqueued
• API Gateway: create the AWS integration → IAM role with sqs:SendMessage → mapping template converting the HTTP body → deploy → POST enqueues directly
• Step Functions: task state with waitForTaskToken → message carries the token → consumer calls SendTaskSuccess/SendTaskFailure

7. Configuration

• S3 notification configuration and event-type selection
• EventBridge event pattern, target, and target DLQ
• API Gateway integration request mapping template and Content-Type handling
• Queue policies for every service principal, always with source conditions

8. Production considerations

• S3 messages carry a reference, not the object. Your consumer must fetch from S3 — and handle the object having been deleted or replaced since
• S3 notification delivery is not guaranteed for every event type. For workflows where missing a file is unacceptable, reconcile against S3 Inventory or use EventBridge
• A large multipart upload emits CompleteMultipartUpload, not Put — filtering on the wrong event type silently drops big files
• API Gateway → SQS hides errors unless the mapping template surfaces them; test the failure path explicitly
• EventBridge has its own throttles and a 256 KB event limit — smaller than SQS's message limit
• Every integration needs its own monitoring: S3 has no 'notification failed' metric you will notice by accident

9. Common mistakes

• Expecting the S3 object body in the message. Why → only the reference is sent. Instead → fetch it
• Subscribing to s3:ObjectCreated:Put only. Why → multipart uploads emit a different event. Instead → s3:ObjectCreated:*
• Assuming S3 notifications are guaranteed. Why → they are not, for all event types. Instead → reconcile for critical flows
• Missing the queue policy for the service principal. Why → silent non-delivery. Instead → policy with aws:SourceArn
• Using EventBridge where SNS suffices. Why → higher cost and lower throughput for plain fanout. Instead → match the tool to the routing complexity
• API Gateway → Lambda → SQS when the Lambda does nothing. Why → cold starts and cost for a pass-through. Instead → direct integration

10. Real-world example

• A document-processing pipeline: upload to S3 → notification → SQS → worker fleet fetches and converts. Include the multipart-upload bug and its fix

11. Interview questions

• 🟢 How can S3 trigger processing through SQS?
• 🟡 What does an S3 event notification message actually contain?
• 🟡 When would you use EventBridge instead of SNS?
• 🔴 Why might an S3 upload not produce an SQS message?
• 🔴 How do you build an HTTP endpoint that enqueues to SQS with no compute?
• ⚫ Design an ingestion pipeline receiving files from 200 partners with per-partner isolation and guaranteed completeness.

12. Summary

• S3, EventBridge and API Gateway can all produce into a queue without code
• All of them need a queue policy with a source condition
• S3 sends a reference and does not guarantee every notification
• EventBridge for complex routing and replay; SNS for plain fanout
• API Gateway → SQS removes a whole Lambda from the ingest path

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• S3 event notification
• EventBridge rule
• event pattern
• service integration
• waitForTaskToken

Diagrams to build:

• S3 → SQS
• EventBridge → SQS
• API Gateway → SQS
• Step Functions callback

Code samples:

• S3 bucket notification configuration plus the matching queue policy
• EventBridge rule with an event pattern and an SQS target
• API Gateway AWS integration with a request mapping template
• Step Functions task with waitForTaskToken and the consumer calling SendTaskSuccess

Mandatory "why?" answers:

• Why does S3 send a reference rather than the object?
• Why can an S3 upload fail to produce a message?

← Previous: 20 — SNS + SQS Fanout · Index: Course home · Next: 22 — Self-Managed Consumers on ECS, EKS and EC2 →