Monitoring and Observability

Every CloudWatch metric SQS emits, what it actually means, and which combinations diagnose which failure. The central lesson: queue depth alone tells you almost nothing — and building your alerting on it is why teams get paged for healthy systems and miss real ones.

1. What you will learn

• Interpret every SQS CloudWatch metric, including the misleading ones
• Diagnose eight common conditions from metric combinations alone
• Build an alarm set that catches real problems without false pages
• Propagate correlation IDs through messages and trace an async flow end to end
• Write a log line that makes a 3 a.m. incident tractable

2. Why this concept exists

• An async system has no stack trace spanning producer and consumer — the request boundary is the queue
• Without correlation IDs you cannot answer "what happened to order 12345?"
• And without the right metrics you cannot tell a healthy busy queue from a stalled one; both look like 'depth is high'

3. Beginner explanation

• CloudWatch shows how many messages are waiting, how many are being worked on, and how old the oldest one is
• Those three numbers, together, tell you almost everything
• Your own logs have to carry an id that links the producer's request to the consumer's work

4. How it actually works

• ApproximateNumberOfMessagesVisible — waiting to be picked up. Backlog
• ApproximateNumberOfMessagesNotVisible — in flight. Work in progress or stuck work
• ApproximateNumberOfMessagesDelayed — waiting out DelaySeconds
• ApproximateAgeOfOldestMessage — ⭐ the best single health metric. It is a latency measure, which is what users feel
• NumberOfMessagesSent / Received / Deleted — ⚠️ Received ≠ Deleted over time means messages are being redelivered. Their ratio is your redelivery rate
• NumberOfEmptyReceives — high value means short polling or over-provisioned consumers
• SentMessageSize — watch for payload growth that will hit the size limit or inflate billing
• Diagnostic combinations:
• Visible↑ + NotVisible flat → consumers are not polling (crashed, scaled to zero, throttled)
• Visible↑ + NotVisible↑ → consumers are polling but too slow, or stuck
• Visible flat + Age↑ → a poison message circulating, or a stuck group
• Received ≫ Deleted → processing is failing; expect the DLQ to grow next
• EmptyReceives high + Visible ~0 → wasted money, no correctness problem
• DLQ depth ↑ → poison message or a downstream outage; the rate distinguishes them
• Metric granularity is 1 minute, and the values are approximate — alarm on sustained conditions, not single datapoints
• Tracing: put the trace/correlation id in message attributes, not the body, so it survives body schema changes; X-Ray and OpenTelemetry both support SQS propagation
• Structured logging: one JSON line per message with messageId, correlationId, queue, receiveCount, processingTimeMs, status

5. Diagram

• Diagram 37 — monitoring architecture: SQS → CloudWatch metrics → alarms → SNS → on-call, plus app metrics and logs
• A decision tree mapping metric combinations to root causes
• Correlation-ID propagation across API → queue → worker → downstream

6. Step-by-step flow

• Emit a correlation id at the API edge
• Attach it as a message attribute on send
• Read it on receive and put it in the logging MDC / context
• Log receive with messageId, correlationId, receiveCount
• Log completion with processingTimeMs and status
• Emit a custom metric for processing duration and outcome
• Alarm on oldest-message age, DLQ depth, and redelivery rate
• Dashboard: depth, in-flight, age, throughput, error rate, DLQ — on one screen

7. Configuration

• CloudWatch alarms on: ApproximateAgeOfOldestMessage (primary), DLQ ApproximateNumberOfMessagesVisible > 0, NumberOfEmptyReceives (cost), redelivery ratio (custom)
• Suggested thresholds tied to the SLO from Module 15 rather than to round numbers
• Composite alarms to suppress the flood when one root cause trips several alarms

8. Production considerations

• Alarm on age, not depth. Depth without processing time is meaningless; age is directly the latency your users experience
• NotVisible that never falls means consumers are receiving and neither deleting nor crashing — often a deadlock or an infinite retry inside the handler
• A DLQ alarm is non-negotiable. Depth > 0 should page or ticket depending on the queue's criticality
• Log ApproximateReceiveCount on every receive — a rising distribution is the earliest warning of a downstream problem
• Dashboards should be per pipeline, not per queue: producer rate, queue, consumer rate, downstream latency, DLQ, all adjacent
• Retain the correlation id into the DLQ — a DLQ message you cannot trace back is nearly useless

9. Common mistakes

• Alarming only on queue depth. Why → false pages on healthy bursts, silence on slow stalls. Instead → oldest-message age
• No DLQ alarm. Why → failures accumulate invisibly. Instead → alarm on depth > 0
• Correlation id in the body. Why → lost on schema change, invisible to filter policies. Instead → message attribute
• Logging the whole message body. Why → PII in CloudWatch Logs, and cost. Instead → ids and metadata
• Treating approximate metrics as exact. Why → flapping alarms. Instead → sustained-period alarms
• No metric for processing time. Why → you cannot compute required concurrency. Instead → emit a duration histogram

10. Real-world example

• "The queue is growing but the consumers look fine." Work the metric combinations to the actual cause — a downstream connection pool exhausted, visible only as rising NotVisible and flat Deleted

11. Interview questions

• 🟢 Which CloudWatch metrics does SQS publish?
• 🟡 What is the difference between Visible and NotVisible?
• 🟡 Which single metric would you alarm on first, and why?
• 🔴 The queue is growing but consumer CPU is low and no errors are logged. Diagnose from metrics.
• 🔴 NumberOfMessagesReceived is 3× NumberOfMessagesDeleted. What does that tell you?
• 🔴 How do you trace a single business transaction across an async SQS boundary?
• ⚫ Design the complete observability stack for a queue-based order pipeline with a 30-second SLO.

12. Summary

• ApproximateAgeOfOldestMessage is the best single health signal — it is latency
• Visible vs NotVisible vs Age, read together, identify the failure class
• Received ≫ Deleted = redelivery = trouble coming
• Correlation ids belong in message attributes
• A DLQ without an alarm is a silent outage

Authoring notes

Terms defined in this module (defined once here, linked from everywhere else):

• ApproximateAgeOfOldestMessage
• NumberOfEmptyReceives
• correlation ID
• structured logging
• composite alarm
• redelivery rate

Diagrams to build:

• Diagram 37 — monitoring architecture
• Metric-combination decision tree
• Correlation-ID propagation

Code samples:

• Correlation-ID propagation via message attributes (producer and consumer)
• A structured JSON log line for receive and completion
• Publishing a custom CloudWatch metric for processing duration
• Alarm definitions as IaC (age, DLQ depth, redelivery ratio)

Mandatory "why?" answers:

• Why is queue depth not enough to determine system health?
• Why can a queue grow while consumers look perfectly healthy?
• Why does Received ≠ Deleted matter?

← Previous: 16 — Security · Index: Course home · Next: 18 — Cost →