What I Learned Running Real-Time Pipelines at Scale
For the past few years I've spent a significant portion of my time building and operating real-time event pipelines — systems that ingest millions of messages daily, process them through multiple services, and trigger personalised actions for end users at scale. What follows are the lessons that weren't in any tutorial.
1. The Schema Is the Contract — Treat It Like One
Early on, we treated event schemas as internal implementation details. If the producer changed a field, we updated the consumer. Simple.
At scale, this breaks. Multiple consumers. Asynchronous deployments. A producer ships a field rename on Tuesday. One consumer gets deployed Thursday. Another consumer's deployment is delayed a week. For five days, two consumers running in production have different expectations of the same event stream.
The fix isn't technical — it's cultural. Schema changes need to be backward-compatible for a defined window, communicated ahead of time, and validated at the boundary. We moved to Pydantic models with explicit versioning, and stopped treating "it's just an internal event" as a reason to skip the contract discipline.
The lesson: event schemas between services deserve the same rigour as external API contracts. The fact that you own both sides doesn't make a breaking change cheaper to fix.
2. At-Least-Once Delivery Means Your Consumers Must Be Idempotent — Without Exception
Every managed message broker — Pub/Sub, SQS, Kafka — guarantees at-least-once delivery, not exactly-once. In practice, duplicates are rare. But "rare" at millions of messages per day is still hundreds or thousands of duplicate deliveries per week.
We learned this the hard way when a consumer restart during a deployment window caused a batch of messages to be redelivered. The consumer wasn't idempotent. Orders were duplicated. Emails were sent twice. The fix was a Redis deduplication layer keyed on event_id — twelve lines of code that should have been there from day one.
The lesson: idempotency isn't an optimisation you add when you see problems. It's a correctness requirement you build in before you ship.
3. Dead-Letter Queues Are Not a Safety Net — They're a Mirror
We set up dead-letter topics because it was a best practice. We didn't think much about them until a consumer bug started silently dropping messages into the dead-letter queue at a rate of a few hundred per hour. Nobody noticed for three days because nobody was watching the dead-letter queue.
Dead-letter queues tell you about the health of your consumers. A growing dead-letter queue means something is wrong — not something you can investigate next sprint, something that needs attention now. We added alerting on dead-letter queue depth (alert at depth > 0, not > some threshold) and a reconciliation job that ran every six hours to triage and reprocess eligible messages.
The lesson: a dead-letter queue without monitoring and reconciliation is a place where data goes to disappear.
4. Backpressure Is Real and You Need to Plan for It
A consumer that processes messages slower than they arrive will eventually fall behind. The subscription backlog grows. If the backlog grows faster than the consumer can catch up, it never catches up — it just falls further behind until messages expire or the team intervenes.
We saw this during a traffic spike that was 4x our usual volume. The consumer was CPU-bound on some transformation logic we hadn't optimised. The backlog grew faster than the consumer could drain it. We scaled the consumer horizontally but hadn't configured the right partition assignment for the workload, so additional instances didn't help as much as expected.
Three things help: design consumers to be horizontally scalable from the start, test with synthetic load that exceeds your expected peak before you need it, and set alerts on oldest_unacked_message_age rather than raw backlog size — an old unprocessed message is more urgent than a large but actively draining backlog.
5. Ordering Guarantees Have a Cost
Strict message ordering — Pub/Sub ordering keys, Kafka partition assignment — solves a real problem. It also introduces a real constraint: ordered delivery within a key serialises processing for that key.
We used user ID as the ordering key, which made sense. User events should be processed in order. But a handful of users generated extremely high event volumes — internal test accounts, automated integrations. Those users' events backed up, and because they shared infrastructure with real users in the same subscription, the backlog from high-volume keys added latency for everyone.
The lesson: ordering keys create implicit sequencing that affects throughput. Know your key cardinality and volume distribution before enabling ordering. For most event types, the ordering requirement is softer than it seems — the consumer can often handle mild out-of-order delivery gracefully with a timestamp check.
6. The Monitoring You Set Up at Launch Is Never Enough
Every time the system encountered a new failure mode, we discovered a gap in our monitoring. A metric we thought was sufficient turned out to miss the thing that actually went wrong.
The monitoring setup that's worked best for us has three layers:
Infrastructure metrics — Pub/Sub backlog, subscription ack latency, consumer error rate. These tell you the pipeline is degrading.
Business metrics — messages processed per minute by event type, end-to-end latency from event publish to downstream action. These tell you whether the degradation matters.
Data quality checks — periodic jobs that sample processed records and verify business rules held. These tell you whether events that appeared to process correctly actually produced the right outcomes.
The third layer is the one most teams skip. An event can be successfully consumed and acknowledged but still produce a wrong outcome — the consumer had a bug in the transformation logic, a downstream write failed silently, a schema mismatch produced null values. Infrastructure metrics won't catch this. Only checking the outputs will.
7. Simplicity at the Cost of Some Performance Is Usually the Right Tradeoff
The most reliable version of a pipeline we've run is the simplest one: events published to a managed broker, consumed by stateless services, results written to a database. No stream processing framework, no complex topology, no state machines.
When we've added complexity — stateful windowed aggregations in the stream, chained multi-step consumers, fan-in patterns — we've added operational burden that paid off in some cases and didn't in others. The cases where it didn't were the ones where the complexity was added to solve a performance problem that turned out to be solvable with a simpler fix (better indexing, more consumer instances, a caching layer).
The lesson: before adding architectural complexity to a pipeline, be sure the simpler version genuinely can't work. The operational cost of a complex streaming topology is real and ongoing.
These lessons took years to accumulate. Most of them are learnable earlier if you know to look for them. The patterns they point to — idempotency, schema contracts, dead-letter monitoring, backpressure planning — show up in every real-time system regardless of the technology stack. Getting them right is what separates a pipeline that works in a demo from one that runs reliably at 3am without anyone looking at it.
About the author
- Senior full-stack engineer and engineering lead with 5+ years building cloud-native systems at scale. I write about the things the docs don't cover — GCP, FastAPI, Terraform, event-driven architecture, and AI-powered backends. No hello-world demos.
Comments
Join the discussionThe dead-letter queue point hits home. I've seen how easy it is to assume things are fine just because the main service looks healthy. Checking what actually made it through matters just as much.
The schema-as-contract point connects directly to what I hit on the application side: validating data at the edges only works if the contract is treated as public API, not as an implementation detail someone can rename on a Tuesday. The idempotency section is the one I'd underline twice. The consumers that break are never the ones you expect - it's the quiet retry path somebody adds six months later that violates idempotency without anyone noticing. Did you end up enforcing it with a dedupe store keyed on event ID, or did you push it down into the consumer logic itself?
The dead-letter-as-a-mirror point is spot on. Something that helped us: alert on the age of the oldest message in the DLQ, not just the count, because a slow trickle looks harmless by count but turns into data loss once it passes the retention window. And for the Redis dedup layer, make sure the key TTL is longer than your longest possible redelivery gap (including replaying from the DLQ), otherwise a replay days later skips dedup and you get the duplicate orders again.
The distinction between acknowledgement and correct output is useful. I would make the processed-per-minute metric count each event_id once and chart delivery attempts separately. During a restart storm, higher throughput can otherwise mean duplicate work. Do your data-quality checks also reconcile expected distinct events with completed downstream actions?
When a consumer processes an event successfully but fails before acknowledging it, the message may be delivered again. How do you design idempotency and reconciliation together to prevent duplicate downstream updates while ensuring that no events are lost during retries or dead-letter queue reprocessing?
How did you handle the gap between recording an event_id in Redis and completing the downstream action? If a worker dies between those steps, marking it first can skip unfinished work; marking it afterward can repeat an email. Did you use a processing/completed state or downstream idempotency keys, and did the dedup retention cover dead-letter replays? (AI-assisted comment.)
The Redis deduplication layer keyed on event_id is a good example of a small fix with a big payoff, especially since the duplicated orders and double emails came from an ordinary deployment restart. Alerting on dead-letter depth above zero rather than a threshold is also a sharp rule, since three days of silent drops is exactly what a threshold hides. And watching oldest_unacked_message_age instead of raw backlog size is a useful way to tell a draining backlog from a consumer that is truly falling behind.
