Snow White

Work has me building and running a meter data platform, and this is my own version, built from scratch to explore the same problem: take payloads from electricity meters, process them, store them cheaply, and get them back out again fast. It’s serverless, it’s on AWS, and every subsystem is named after one of the seven dwarfs.

The shape of it

A payload lands, gets ingested, processed, enriched, stored hot, then eventually migrated to cold storage - and can be pulled back out at any point along the way.

A payload arrives at Sneezy, either as a REST call or dropped onto an SFTP site, and gets pushed straight onto a Kinesis stream with as little processing as possible - the goal is to keep ingestion thin and fast.

Grumpy picks it up from Kinesis, looks up the meter’s configuration in Bashful, and splits the payload into its individual data streams, each one enriched with “how to process it” instructions from that config. Processing is done with a chain of small, composable cartridges:

More than one cartridge can be chained onto a single data stream, which turned out to be a nice way of keeping processing logic simple and testable while still supporting fairly involved transformations.

Hot, then cold

Processed streams land in Happy - DynamoDB, tuned for speed, with a 7-day TTL. After that window, a stream migrates to Sleepy, cheap S3 storage meant to hold data for up to seven years.

Getting data back out is Dopey’s job. It checks Happy first; on a miss, it falls back to Sleepy, and - this was the fiddly bit to get right - re-warms the data by writing it back into Happy for a couple of days, in case it’s needed again soon.

A payload

{
  "meter": { "serialNumber": "MTR-001", "manufacturer": "Acme", "type": "X100" },
  "unitOfWork": { "unitOfWorkTimeStamp": "2026-06-19T00:00:00Z" },
  "dataStreams": [
    {
      "name": "active-energy-import",
      "period": 900,
      "details": {},
      "measurements": [
        { "timestamp": "2026-06-19T00:00:00Z", "value": 42.3, "details": {} }
      ]
    }
  ]
}

Scale

The design target is 10,000,000 payloads an hour - about 2,778 a second, sustained. With an average of 10 data streams per payload, that’s roughly 27,780 individual stream records a second flowing through Grumpy. Everything is Lambda and Kinesis rather than anything long-running, which makes that kind of throughput a matter of concurrency limits and shard counts rather than needing to provision servers for peak load.

Doc helps test that at scale too - it’s a nightly job that generates synthetic meter data and fires it at Sneezy the same way a real meter would, so the whole pipeline gets exercised even when there’s no real traffic.

Keeping it honest

There’s a UI - Snow White herself - a React/Vite app behind Cognito for browsing meters, payloads and retrieved data, which made debugging the rest of the system much less painful than reading DynamoDB items by hand.

I also leaned on Claude for a lot of this: working through the DLQ replay strategy in the runbook, generating the Terraform for each subsystem, and running a security review over the whole thing before I called it done. It’s the kind of project where getting the design right up front - what goes hot, what goes cold, how retrieval re-warms data - mattered more than any individual line of code, and having something to think it through with out loud was genuinely useful.

83 pytest tests cover the Python side; Terraform is split one module per subsystem plus shared modules for the Kinesis bus, SNS and the UI, with separate dev and prod environments.