Right now, somewhere in a planning meeting, someone’s asking “can we make this real-time” and everyone’s nodding. Sounds much better, obviously. New data. Fast decisions. Dashboards on the move. Who would advocate for slower?
I’ve made this jump a few times in large analytics pipelines, and here’s what I wish someone had said in that meeting: real-time is not an upgrade to batch. It’s a different machine, with a different set of issues. You’re not speeding up your batch job, you’re just changing one set of tradeoffs for another, and most of the new tradeoffs don’t show up until you’re already in production.
Here’s what actually breaks, when the trade is worth paying for, and why the line between batch and streaming is starting to disappear.
What actually breaks
- Time gets complicated. Batch makes time easy. You process yesterday’s data, the day is closed, everything that was going to arrive has arrived. In streaming, none of that holds. Events come in late, out of order, or not at all. A click from a phone that lost signal shows up forty seconds after the impression it belongs to. Now you have a question batch never made you answer: how long do you wait before you call a window closed? Wait longer and the numbers are more accurate but less fresh. Wait less and you are fast, but you start dropping late events. In Spark Structured Streaming this is one line, withWatermark(“event_time”, “10 minutes”). Writing the line is trivial. Choosing the number is the hard part, and you will change it more than once.
- Micro-batch or true streaming is a real fork, not a config flag. People treat Spark and Flink as interchangeable. They are not, and this is the distinction most write-ups skip. Spark Structured Streaming comes from the batch world and processes a stream as a sequence of small batches, so it has a latency floor: you cannot get fresher than your batch interval, which usually puts you in the low seconds. Flink was built the other way around, as a native stream processor that handles one event at a time, which is how it reaches sub-second and millisecond latency. The tradeoff runs both ways. Micro-batch buys you simpler exactly-once, simpler failure recovery, and the ability to reuse your batch code, which is why it is the pragmatic default for most analytics. True streaming buys you latency but costs you complexity and a steeper operational curve. The point for any real decision: know which side of that floor your use case needs before you pick an engine, because crossing it later is a rewrite, not a tuning pass.
- “Just re-run it” stops working. This one catches people off guard. In batch, fixing a bug is easy, almost relaxing. Find the problem, rerun the job for the affected dates, move on. You get backfill for free. Streaming gives you none of that. Replaying history is a project on its own. You have to check whether your source even kept the old events, whether replaying creates duplicates, whether everything downstream can take the same record twice. This is why a lot ofteams keep a batch pipeline running purely to backfill and correct the streaming one, and now they are maintaining two systems that are supposed to produce the same answer. [Your example: a time you had to reprocess history and how you reconciled it, described generically.]
- Exactly-once is mainly marketing until you do the extra work. Every streaming tool advertises exactly-once. That is true, but only in one narrow sense: within a single engine, with its own state and sinks. Your actual pipeline cuts across a broker, state store, database, cache, and some external API, with failures occurring right there at those crossings. What you really get at a baseline is at-least-once plus idempotency: every critical operation must be written so that re-processing an event does not change the outcome. Actual end-to-end exactly-once across a boundary is a deliberate build: a transactional or two-phase-commit (2PC) sink that only commits output when the rest of the pipeline commits, or an outbox pattern that couples the write and event emission into a single atomic step. None of this is free, and none comes from the label on the box. Ignore it, trust the marketing, and learn the hard way at your first failover.
- State is what wakes you up at 3am. Batch jobs have almost no memory. They read, write and forget. Streaming jobs remember everything: running totals, open windows, join buffers, sessions. It all has to be checkpointed and restored. And it grows. Key an aggregation on something high-cardinality? Memory pressure. Hold a window open too long for late data? Slow checkpoints. A job that takes forever to restart because it has to reload all that state before it can even handle a single new event? This is not throughput, this is real pain in my experience.
- The number won’t sit still. This is more about people than code and is underrated. As soon as a number goes live, people treat it like the end of the road. But live numbers are supposed to move. They change as new data comes in. The dashboard flicks. Somebody’s going to see the revenue drop for thirty seconds and freak out, even though the final number is right. Batch reports are proven and reliable. Real-time numbers are volatile and current. Without setting expectations with the people reading them, you’ll spend your week explaining why the number moved instead of building anything.
- Always-on means every failure is an incident. A batch job that dies at 2am is a 9am problem, and nobody knows. A streaming job lags behind, dies live and someone watches the lag grow. This means real monitoring of lag, backpressure, and checkpoint duration. This means on-call. And it costs more. You are paying for computing that is always on, as opposed to starting up, doing its work and shutting down, and that cost is easily missed because it is spread across every hour instead of being in a single job you can point to.
The architecture has actually moved
Most write-ups on this topic stop at Lambda vs Kappa. It’s worth knowing about the debate. Lambda runs a batch layer and a streaming layer in parallel and merges them, whereas Kappa collapses to a single streaming layer over a replayable log, replaying history through the same code when it needs to recompute. But that’s not where the field is anymore. And the interesting part is that the storage layer is solving the “two systems that must agree” problem rather than the compute layer. The direction now is convergence.
What has changed is the open table formats. Apache Iceberg, Delta Lake and Apache Hudi were born as a way to bring warehouse-style reliability (think ACID commits, schema evolution, and time travel) to cheap storage objects. They now support native streaming ingestion and incremental upserts, meaning you can use the same table for real-time read and a full historical scan. You stream into it, and you batch over it, and it’s one source of truth instead of two. Each format takes a different path, Iceberg towards openness and running many engines across the same tables, Delta towards Spark-native simplicity, Hudi towards fast streaming updates.This is why “streaming-first lakehouse” is the phrase of the year, and it is the framing a current architecture reviewer will expect you to have.
It’s not magic and the honest version says so. Streaming to these tables converts “daily batch” into “continuous micro-batch”, which introduces a new class of problems: small-file pile-ups that need compaction, commit latency because the table metadata was designed for batch and is strained under high frequency writes, schema changes that have to stay forward and backward compatible, and data-quality checks now have to run in flight, instead of in a nightly job. The ingestion path is increasingly change data capture, usually a tool like Debezium streaming database changes via Kafka into these tables, which has all the same caveats. The reason this belongs in the decision, not just the background: it changes the recommendation. The old answer to “batch or streaming” was “run both and reconcile.” The emerging answer is “converge on one storage layer and choose your read latency per consumer.” That is a better place to be, and it is worth saying out loud that it is where we should be heading.
Batch vs streaming vs converged, at a glance
| Dimension | Batch | Streaming | Converged (stream intoa table format) |
| Latency | Hours, on a schedule | Seconds for micro-batch, sub-second for true streaming | Continuous write, read latency you choose per consumer |
| Cost model | Pay only when the job runs | Pay continuously, always on | Continuous ingest, but one storage layer instead of two pipelines |
| Late or out-of-order data | Wait for the window to close | Pick a watermark, tradeaccuracy for freshness | Same watermark tradeoff on ingest, plus time travel to correct |
| Fixing a bug or backfilling | Re-run over the affected dates | A project: replay, dedup, idempotency | Reprocess into the table with snapshot isolation while you do |
| Correctness | Fully recomputable | At-least-once plus idempotency, 2PC sink for true exactly-once | ACID commits on the table, still need idempotent ingest |
| Operational burden | A failed job is a morning fix | A lagging job is a live incident | Live ingest plus table maintenance (compaction, small files) |
| Where it fits | Reporting, heavy historical recompute | Decisions that happen in seconds | Fresh reads plus historical recompute on one source of truth |
A simple way to decide
Strip away the hype and it comes down to one question: does anything act on this data within minutes, and does being wrong for a while actually cost something? Run it as a quick test. ● If a human or a system acts on the data within seconds to minutes, and stale data causes real harm (fraud, alerting, bidding, live personalization), stream it.
- If the freshest anyone genuinely needs is hourly or daily, batch it, and do not apologize for it.
- If you need fresh reads but also heavy historical recompute and correction, converge: stream into a table format and serve both off the same source of truth.
- Always start from the actual freshness the consumer needs, not the word “real-time.” Half the time “real-time” means “fresher than once a day,” and a faster batch cadence solves it for a fraction of the cost and operational load.
When streaming is worth it
When freshness matters most. Before a transaction clears, fraud checks have to fire. Alerts where a delay of five minutes amounts to no alert. Live personalization, ops dashboards and bidding where every minute counts. Streaming pays for itself when a person or system acts on data the moment it arrives. The operational cost is merely the cost of doing the job.
When batch is still the correct answer
More often than anyone likes to admit. Most reporting. Most analytics. Anything where a few hours of delay are tolerable, anything requiring heavy recomputation over history, anything that matters for cost. “We might want it real-time someday” is no reason to pay for streaming today.
The honest answer is usually convergence
For most systems, this is no longer even a choice between two pipelines. For latency-critical parts, you stream into one storage layer and read the same source of truth at whatever freshness each consumer needs, including historical scans. It is not as clean as a slogan, but that is what holds up in production. The old hybrid meant maintaining two codebases you had to keep in sync. The new one means one place where the data lives and latency as a read-time choice.
If you’re about to make the jump
Start from the latency your users actually need, not the word “real-time.” A faster batch schedule handles more cases than you would guess.
Pick your side of the latency floor up front. Micro-batch or true streaming is an architecture decision, not a setting you flip later.
Settle your correctness story first. Decide exactly what happens to late data and duplicate data, and how far your exactly-once guarantee actually reaches, before you write the job. Plan for backfill and, if you are streaming into a lakehouse, for compaction and small files on day one. You will need both, and you do not want to design them under pressure. Set up monitoring for lag, checkpoint duration, and backpressure from the start. In batch you can dig through yesterday’s run after the fact. In streaming, if you are not watching, you hear about it from someone else.
Bottom line
Real-time is great when you actually need it. Just go in knowing you are trading, not upgrading, and knowing the architecture has moved past “pick a side.” Decide the freshness your users really need, choose your engine and your exactly-once scope deliberately, and lean toward converging on one source of truth you can read at any speed. That is the version that survives contact with production.
Author – Vishnuvardhan Reddy Kaithapuram, Software Development Engineer, Amazon Advertising”