This is the project behind the error-handling design I mentioned in my Claude post - a proof-of-concept, built to work through how to detect, capture and recover from failures in a high-volume stream-processing pipeline without ever losing a record or scrambling the order they arrive in.
The premise: payloads from IoT sensors get written to a DynamoDB table, and a Change Data Capture chain - DynamoDB Stream, a mapper Lambda, a Kinesis stream, a writer Lambda - moves each one through to a final storage table. Ordinary operation is the easy part. The interesting problem is what happens when something downstream goes away for a while: a network blip, an AWS service disruption. Failures aren’t expected to be code bugs, but on a high-volume system they’re expected to arrive in bursts, and every one of them has to come back in the order it went in.
Two failures, two strategies
There are exactly two places this pipeline can lose a record: the mapper writing to Kinesis, and the writer writing to DynamoDB storage. Naively you’d handle both the same way - catch the error, dead-letter it, replay later. It turns out that’s wrong, and working out why was most of the value of the exercise.
Mapper failures need to be captured in-Lambda. DynamoDB Streams only retain data for 24 hours, and Lambda’s built-in destination-on-failure mechanism only forwards metadata (shard ID, sequence range) to SQS - not the record itself. So if Kinesis is unavailable, the mapper catches the error itself and writes the full payload straight to an SQS FIFO queue, before the DynamoDB Stream can expire it. A single message group keeps everything in strict order; the sequence number as the deduplication ID makes it safe to retry. Recovery is a manually-invoked Lambda that drains the FIFO queue in order and replays each record to Kinesis.
Writer failures are handled completely differently: block and retry, forever. The writer Lambda just lets the exception propagate, and the Kinesis Event Source Mapping is configured with unlimited retries (maximum_retry_attempts = -1). That freezes the shard iterator at the exact point of failure - nothing advances past it - so every record produced during the outage queues up in Kinesis behind it, in order, for free. When DynamoDB recovers, the ESM just carries on. No recovery Lambda needed for the common case; the safety-net DLQ only catches a genuine poison-pill record.
The asymmetry is the whole point: a dead-letter-and-replay-later approach - which is the instinctive first answer - would have silently broken the ordering guarantee for the writer, because replayed records would land after the ones that succeeded during the outage. Recognising that DynamoDB Streams’ 24-hour retention and Kinesis’s much longer retention aren’t interchangeable is what forces the two designs apart.
Recovery, on purpose, manually
Both recovery paths are invoked by a human, deliberately. Automatic recovery risks a retry storm into a system that might still be unhealthy - so instead, four CloudWatch alarms watch DLQ depth and Lambda error rates, page an operator via SNS on the first failure, and clear themselves automatically once things are healthy again. The operator decides when it’s actually safe to replay.
Working through it with Claude
This is the project I built to design and present the failure-handling strategy at work - reproduced here as a personal PoC so I could actually build and test it rather than just diagram it. I used Claude to work through the design options (an ESM-destination-only approach was the obvious first idea, and Claude helped me see why it silently fails to preserve ordering), to write the ADRs recording why each option was rejected, to generate the Terraform, and to build a Marp slide deck plus a prepared Q&A for presenting it to an architecture review of principal engineers and tech leads. Having something to argue the trade-offs with, out loud, before I had to defend them in a room, made the eventual presentation a lot less stressful.
What’s deliberately left unfinished
It’s a PoC, and the README is upfront about the edges left unresolved rather than quietly ignoring them: a mixed-success batch in the mapper can still replay a handful of records out of order (the fix is documented but not built); writer-recovery has no way to quarantine a genuinely unprocessable record, only retry it forever; and a couple of design questions are explicitly deferred to an ADR-003 that doesn’t exist yet. Writing those down felt more useful than pretending the PoC was production-ready.