Self-Managed Consumers on ECS, EKS and EC2

When you run the poll loop yourself. More work than Lambda and more control: long-running jobs, high sustained throughput, and connection reuse — plus one responsibility Lambda hid from you entirely, graceful shutdown, which is where most self-managed consumers go wrong.

1. What you will learn

• Write a production-grade consumer loop with a bounded worker pool
• Implement graceful shutdown so a deploy does not produce a burst of redeliveries
• Autoscale containerised consumers on queue depth with KEDA or ECS target tracking
• Choose correctly between ECS, EKS, EC2 and Lambda
• Manage connections, threads and health checks in a long-lived consumer

2. Why this concept exists

• Lambda's 15-minute ceiling, per-invocation connection cost and concurrency model do not fit every workload
• A long-lived process can hold a warm connection pool, batch downstream writes and run for hours — at a lower cost per message at sustained volume
• The price is that shutdown, health and scaling become yours

3. Beginner explanation

• Your own program runs in a loop: ask for messages, do the work, delete them
• It runs in a container that stays up
• When the container is told to stop, it must finish what it started first

4. How it actually works

• The loop: long-poll for up to 10 → submit each to a bounded executor → process → delete (batched) → repeat
• Bounded queue + CallerRunsPolicy (or a blocking submit) is what gives you backpressure; an unbounded queue gives you an OOM instead
• Graceful shutdown: trap SIGTERM → stop polling → let in-flight work finish → batch-delete → exit. ECS sends SIGTERM then SIGKILL after stopTimeout (default 30 s, raise it); Kubernetes uses terminationGracePeriodSeconds
• ⚠️ What happens without it: the process dies holding N in-flight messages; they wait out the full visibility timeout, then get reprocessed. Every deploy becomes a duplicate-processing event
• A shutdown longer than the visibility timeout is pointless — the message is already redelivered. Size the two together
• Scaling: ECS service autoscaling on a CloudWatch metric; EKS HPA on custom metrics, or KEDA with its native aws-sqs-queue scaler (including scale-to-zero)
• Health checks: liveness should reflect the poll loop actually running, not just the process being alive. A consumer stuck in a non-terminating handler passes a naive health check forever
• Connection management: one SqsClient per process (it is thread-safe), sized HTTP connection pool ≥ concurrent pollers
• IAM: ECS task roles, EKS IRSA / Pod Identity, EC2 instance profiles — never static keys
• Spot instances are viable precisely because SQS redelivers — but only if shutdown handling is correct

5. Diagram

• Consumer fleet: N containers, each with a poller and a worker pool
• Graceful shutdown timeline: SIGTERM → drain → delete → exit, against the visibility timeout
• KEDA scaling loop from queue depth to replica count, including scale to zero

6. Step-by-step flow

• Container starts, builds one SqsClient, registers a SIGTERM handler
• Poller thread long-polls with MaxNumberOfMessages = 10
• Each message is submitted to a bounded executor; a full queue blocks the poller (backpressure)
• Workers process and record successful receipt handles
• A batching deleter flushes DeleteMessageBatch calls
• SIGTERM: stop polling immediately, wait for workers, flush deletes, exit 0
• The orchestrator replaces the task; the queue absorbed the gap

7. Configuration

• ECS stopTimeout / Kubernetes terminationGracePeriodSeconds — longer than the p99 processing time
• Worker pool size, bounded queue capacity, HTTP connection pool size
• KEDA ScaledObject with queueLength and activationQueueLength
• Visibility timeout coordinated with the shutdown budget (Module 08)

8. Production considerations

• Deployment is the most common source of duplicate processing in self-managed consumers. Graceful shutdown is not optional
• Scale-in has the same hazard as deployment — the autoscaler terminating a busy task is identical to a deploy
• Poison-message safety: a handler that can hang forever will hold a worker permanently. Enforce a per-message timeout inside the worker
• Thread-per-message does not scale past a few hundred; for I/O-bound work use async or virtual threads (Java 21)
• Emit your own metrics — SQS's CloudWatch metrics say nothing about your processing time or your pool saturation
• Spot + SQS is a genuinely good pairing, and a genuinely bad one if shutdown is broken

9. Common mistakes

• No SIGTERM handling. Why → a burst of redeliveries on every deploy. Instead → drain
• Grace period shorter than p99 processing. Why → SIGKILL mid-message. Instead → size it from the same p99 as the visibility timeout
• Unbounded internal work queue. Why → OOM instead of backpressure. Instead → bounded, blocking
• A new SqsClient per message. Why → connection churn and latency. Instead → one per process
• Liveness probe that only checks the process. Why → a wedged poll loop looks healthy forever. Instead → probe the loop's last-success timestamp
• No per-message timeout. Why → one hung handler permanently removes a worker. Instead → enforce a timeout

10. Real-world example

• A payments worker on ECS Fargate: 20 tasks, 50 workers each, KEDA scaling 5–100. Show the deploy that generated 4,000 duplicate charges before graceful shutdown was added

11. Interview questions

• 🟢 How does a consumer running on ECS get messages from SQS?
• 🟡 What is graceful shutdown and why does an SQS consumer need it?
• 🟡 How would you autoscale consumers running on Kubernetes?
• 🔴 Your deploys cause duplicate processing. Explain the mechanism and the fix.
• 🔴 How do you apply backpressure inside a consumer?
• 🔴 When is ECS a better choice than Lambda? Answer with numbers.
• ⚫ Design a consumer fleet for 50,000 msg/s with a 99.9% availability target and zero-downtime deploys.

12. Summary

• Long-poll → bounded pool → process → batch delete
• SIGTERM must stop polling and drain in-flight work before exit
• Grace period and visibility timeout are sized from the same p99
• KEDA scales on queue depth, including to zero
• Health checks must prove the loop is running, not that the process exists

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• graceful shutdown
• SIGTERM
• terminationGracePeriodSeconds
• stopTimeout
• KEDA
• IRSA
• bounded work queue

Diagrams to build:

• Consumer fleet topology
• Graceful shutdown timeline
• KEDA scaling loop

Code samples:

• A complete consumer: poller thread, bounded executor, batching deleter
• SIGTERM shutdown hook that drains and flushes
• Per-message timeout enforcement
• KEDA ScaledObject YAML and an ECS target-tracking policy

Mandatory "why?" answers:

• Why does a deploy cause duplicate processing?
• Why is a grace period longer than the visibility timeout useless?
• Why does an unbounded internal queue remove your backpressure?

← Previous: 21 — Event Sources: S3, EventBridge, API Gateway · Index: Course home · Next: 23 — SQS with Java and Spring Boot →