What I Learned Running Real-Time Pipelines at Scale

For the past few years I've spent a significant portion of my time building and operating real-time event pipelines — systems that ingest millions of messages daily, process them through multiple services, and trigger personalised actions for end users at scale. What follows are the lessons that weren't in any tutorial.

1. The Schema Is the Contract — Treat It Like One

Early on, we treated event schemas as internal implementation details. If the producer changed a field, we updated the consumer. Simple.

At scale, this breaks. Multiple consumers. Asynchronous deployments. A producer ships a field rename on Tuesday. One consumer gets deployed Thursday. Another consumer's deployment is delayed a week. For five days, two consumers running in production have different expectations of the same event stream.

The fix isn't technical — it's cultural. Schema changes need to be backward-compatible for a defined window, communicated ahead of time, and validated at the boundary. We moved to Pydantic models with explicit versioning, and stopped treating "it's just an internal event" as a reason to skip the contract discipline.

The lesson: event schemas between services deserve the same rigour as external API contracts. The fact that you own both sides doesn't make a breaking change cheaper to fix.

2. At-Least-Once Delivery Means Your Consumers Must Be Idempotent — Without Exception

Every managed message broker — Pub/Sub, SQS, Kafka — guarantees at-least-once delivery, not exactly-once. In practice, duplicates are rare. But "rare" at millions of messages per day is still hundreds or thousands of duplicate deliveries per week.

We learned this the hard way when a consumer restart during a deployment window caused a batch of messages to be redelivered. The consumer wasn't idempotent. Orders were duplicated. Emails were sent twice. The fix was a Redis deduplication layer keyed on event_id — twelve lines of code that should have been there from day one.

The lesson: idempotency isn't an optimisation you add when you see problems. It's a correctness requirement you build in before you ship.

3. Dead-Letter Queues Are Not a Safety Net — They're a Mirror

We set up dead-letter topics because it was a best practice. We didn't think much about them until a consumer bug started silently dropping messages into the dead-letter queue at a rate of a few hundred per hour. Nobody noticed for three days because nobody was watching the dead-letter queue.

Dead-letter queues tell you about the health of your consumers. A growing dead-letter queue means something is wrong — not something you can investigate next sprint, something that needs attention now. We added alerting on dead-letter queue depth (alert at depth > 0, not > some threshold) and a reconciliation job that ran every six hours to triage and reprocess eligible messages.

The lesson: a dead-letter queue without monitoring and reconciliation is a place where data goes to disappear.

4. Backpressure Is Real and You Need to Plan for It

A consumer that processes messages slower than they arrive will eventually fall behind. The subscription backlog grows. If the backlog grows faster than the consumer can catch up, it never catches up — it just falls further behind until messages expire or the team intervenes.

We saw this during a traffic spike that was 4x our usual volume. The consumer was CPU-bound on some transformation logic we hadn't optimised. The backlog grew faster than the consumer could drain it. We scaled the consumer horizontally but hadn't configured the right partition assignment for the workload, so additional instances didn't help as much as expected.

Three things help: design consumers to be horizontally scalable from the start, test with synthetic load that exceeds your expected peak before you need it, and set alerts on oldest_unacked_message_age rather than raw backlog size — an old unprocessed message is more urgent than a large but actively draining backlog.

5. Ordering Guarantees Have a Cost

Strict message ordering — Pub/Sub ordering keys, Kafka partition assignment — solves a real problem. It also introduces a real constraint: ordered delivery within a key serialises processing for that key.

We used user ID as the ordering key, which made sense. User events should be processed in order. But a handful of users generated extremely high event volumes — internal test accounts, automated integrations. Those users' events backed up, and because they shared infrastructure with real users in the same subscription, the backlog from high-volume keys added latency for everyone.

The lesson: ordering keys create implicit sequencing that affects throughput. Know your key cardinality and volume distribution before enabling ordering. For most event types, the ordering requirement is softer than it seems — the consumer can often handle mild out-of-order delivery gracefully with a timestamp check.

6. The Monitoring You Set Up at Launch Is Never Enough

Every time the system encountered a new failure mode, we discovered a gap in our monitoring. A metric we thought was sufficient turned out to miss the thing that actually went wrong.

The monitoring setup that's worked best for us has three layers:

Infrastructure metrics — Pub/Sub backlog, subscription ack latency, consumer error rate. These tell you the pipeline is degrading.

Business metrics — messages processed per minute by event type, end-to-end latency from event publish to downstream action. These tell you whether the degradation matters.

Data quality checks — periodic jobs that sample processed records and verify business rules held. These tell you whether events that appeared to process correctly actually produced the right outcomes.

The third layer is the one most teams skip. An event can be successfully consumed and acknowledged but still produce a wrong outcome — the consumer had a bug in the transformation logic, a downstream write failed silently, a schema mismatch produced null values. Infrastructure metrics won't catch this. Only checking the outputs will.

7. Simplicity at the Cost of Some Performance Is Usually the Right Tradeoff

The most reliable version of a pipeline we've run is the simplest one: events published to a managed broker, consumed by stateless services, results written to a database. No stream processing framework, no complex topology, no state machines.

When we've added complexity — stateful windowed aggregations in the stream, chained multi-step consumers, fan-in patterns — we've added operational burden that paid off in some cases and didn't in others. The cases where it didn't were the ones where the complexity was added to solve a performance problem that turned out to be solvable with a simpler fix (better indexing, more consumer instances, a caching layer).

The lesson: before adding architectural complexity to a pipeline, be sure the simpler version genuinely can't work. The operational cost of a complex streaming topology is real and ongoing.

These lessons took years to accumulate. Most of them are learnable earlier if you know to look for them. The patterns they point to — idempotency, schema contracts, dead-letter monitoring, backpressure planning — show up in every real-time system regardless of the technology stack. Getting them right is what separates a pipeline that works in a demo from one that runs reliably at 3am without anyone looking at it.