AWS Lambda: The Recovery Shelf
A spilled custom-boba order needs a visible recovery path—not a silent disappearance and not an automatic second attempt with no guardrails.
The Business Goal
The topping-pricing runner receives an order during a spill. It partly records loyalty points, then times out before sending the pickup notification. The team says, “Put it in the DLQ so another runner fixes it.” That is only half a plan.
The Story
The first runner cleans up and tries again according to the dispatch policy. If the ticket still cannot be completed, it goes on the Recovery Shelf with a reason it needs attention. A recovery runner may be assigned later, but only after the shop decides what was already done. Replaying the ticket blindly could award points twice.
Meet the AWS Service
Lambda retry behavior depends on invocation type. For asynchronous invocation, Lambda manages an internal event queue and, by default, retries function errors twice; maximum event age and retry attempts are configurable. A Lambda dead-letter queue retains discarded events in SQS or SNS. An on-failure destination can send a richer invocation record to supported targets. With SQS event source mappings, the source queue’s visibility timeout and redrive policy determine when messages return or move to its own DLQ.
Core idea: A DLQ preserves a failed ticket for investigation or deliberate reprocessing; it does not automatically heal the order.
How It Works
The recovery shelf
Async DLQ and destinations
For an exhausted or expired asynchronous event, configure either an SQS/SNS DLQ or an on-failure destination. A destination contains invocation context and can target more services; a DLQ is the original failed event retained for later action. The destination itself needs appropriate permissions.
The ticket rail
SQS batch failure and redrive
An SQS-triggered Lambda receives a batch. By default, an error can make all messages in that batch visible again after the visibility timeout. Partial batch response lets the function report only failed messages for retry, protecting already-completed work from needless repetition. Configure a source-queue redrive policy for messages that exceed the allowed receives.
The duplicate stamp
Idempotency
Functions should safely handle duplicate events. Store or derive an idempotency key so a retry can recognize that the loyalty update or notification was already completed.
Architectural Mapping
| In Byte Burger | In AWS | What it means |
|---|---|---|
| Spill and retry | retry policy | Another attempt under source rules |
| Recovery Shelf | DLQ/destination | Retained failed event or record |
| Recovery runner | separate consumer/runbook | Deliberate reprocessing path |
| “Already made” stamp | idempotency | Duplicate-safe processing |
When to Use It
- The event has business value and must be investigated after failure.
- A batch can contain independent tickets that should not all be replayed.
- Side effects must survive duplicate delivery safely.
When Not to Use It
- Do not configure a DLQ and assume someone or something will inspect it without an owner and alarm.
Painkiller
Problem: Failed work can be retried, duplicated, or discarded.
Pain: A hidden failure loses orders; a blind replay creates duplicate side effects.
AWS solution: Combine source-appropriate retries, retained failures, alarms, and idempotent code.
Knife Cut
Lambda’s async DLQ/destination and an SQS queue’s redrive DLQ are different recovery mechanisms.
The Masthead
What Actually Just Happened
| Story element | AWS | Precise meaning |
|---|---|---|
| Recovery Shelf | DLQ | Failed event held for later action |
| Detailed incident card | on-failure destination | Invocation record with context |
| Ticket returns to rail | SQS visibility timeout | Message available for retry |
A Note From the Author
The shelf is not a second kitchen. A team must monitor it, decide whether to repair the event, and arrange safe replay. Async configuration differs from SQS and stream mappings. See Lambda retry behavior, async error handling, and SQS batch handling.
The Last Bite
Keep the failed ticket, understand its side effects, then choose its recovery deliberately.
Next chapter: AWS Lambda: The Runner’s Kit and Clearance
The recovered ticket reveals another question: what may a runner carry, and which rooms may that runner enter?