Your Startup Does Not Have a Big Data Problem

A friend running a twelve-person startup once showed me their architecture diagram with something close to pride: Kafka ingesting events, Spark jobs aggregating them nightly, the results landing in a data lake on object storage, and a BI tool querying the lake through a metadata catalogue. I asked how many rows a day the whole pipeline was processing. The answer was around 40,000. A single Postgres server on a laptop could absorb that rate for a decade before the disk filled up, and would answer any query against it in milliseconds rather than the twenty-minute batch window they had built around the nightly Spark job.

This is close to the default mistake, because the architecture diagrams that get shared publicly are almost always drawn by companies operating at a scale nobody in the room actually has, and the tools built for that scale get copied downward without anyone checking whether the problem they solve is the problem in front of them.

I asked a second, more useful question: what happened the last time the Spark job failed. The answer was a Tuesday lost to debugging a partition skew nobody on the team had encountered before, followed by two engineers spending the rest of the week reading documentation for a distributed compute engine neither of them had chosen to specialise in. Meanwhile the actual business question the pipeline existed to answer — how many signups converted to paid in the last thirty days — sat unanswered for that entire week, because the one query that mattered lived behind four layers of infrastructure instead of a single indexed table.

What “big data” actually meant when the term was coined

The term earned its meaning at companies processing volumes that genuinely could not fit on one machine, could not be queried by one CPU in a reasonable time, and could not be held in the RAM of anything you could buy. Google’s original MapReduce paper was solving indexing problems across the entire crawled web. Hadoop existed because a single disk in 2006 could sustain perhaps 80MB/s of sequential read, so scanning petabytes meant spreading the scan across thousands of disks in parallel or waiting months.

None of the constraints that made that architecture necessary still hold for a typical startup’s dataset, and most of them stopped holding well over a decade ago for hardware you can rent by the hour. A single NVMe drive today sustains multiple gigabytes per second of sequential read and hundreds of thousands of random IOPS. A single server can be specified with a terabyte of RAM. A single Postgres instance, tuned reasonably, will comfortably handle tables with hundreds of millions of rows and sustain tens of thousands of transactions per second on unremarkable hardware. The gap between “what one machine can do” and “what the big data tools were built to work around” has closed to the point where most startups never cross it at all.

The arithmetic nobody runs before adopting the platform

Before reaching for a distributed system, it is worth actually doing the sum, because the answer is almost always smaller than intuition suggests.

1
2
3
4
5

Rows per day: 100,000
Row size: ~500 bytes
Daily volume: 50MB
Annual volume: ~18GB
Five-year volume: ~90GB

Ninety gigabytes fits on a phone. It fits in RAM on a mid-range server, meaning the entire five-year history of the business could sit fully cached, with disk touched only for writes. Even at ten times that rate — a million rows a day, which is a genuinely busy consumer application — five years of data is under a terabyte, comfortably inside what an unremarkable single-server Postgres deployment handles with room to spare.

The instinct to reach for Kafka and Spark tends to come from a different place than the actual data volume: it comes from having read that “real” companies use these tools, and wanting the architecture to look serious. But a distributed streaming platform earns its complexity when a single broker cannot sustain the write throughput, or when consumers genuinely need independent, replayable, ordered logs across services that do not yet exist at a twelve-person company. Below that point it is pure overhead: more services to keep running, more failure modes to understand at 2am, and a queryable answer that used to take a millisecond now taking a scheduled batch window.

What a single Postgres box can actually sustain

The honest numbers, from unremarkable production hardware rather than a marketing benchmark, look roughly like this for straightforward OLTP workloads:

Analytical queries — the kind Spark is built for — are the case most often used to justify jumping to a distributed platform, and it is the case single-node tools have closed the gap on hardest. DuckDB and single-node ClickHouse both run genuinely large analytical aggregations, in-process or on one box, at speeds that make a distributed cluster look like the slower option once you account for the coordination overhead a cluster pays on every query. A columnar scan across tens of millions of rows on a single modern CPU, using vectorised execution, finishes in the time it takes a distributed job to schedule its first task.

The headroom nobody accounts for

What makes the single-node case stronger than it looks on paper is that “single node” does not mean “no scaling levers.” A plain Postgres instance already has a substantial ladder to climb before a distributed system becomes the only option, and most teams reach for a completely different architecture without ever touching the rungs closer to hand.

Read replicas split read traffic across additional machines with no application rewrite beyond pointing analytical queries at a follower, a change that takes hours to land. Table partitioning by date or tenant keeps individual indexes small and cache-friendly long after a naive table would have started to degrade, and Postgres has supported declarative partitioning natively for years now, so it needs no bolt-on extension. Connection pooling through something like PgBouncer addresses the ceiling that actually bites first in practice: the number of concurrent connections a server can hold open, well before query throughput becomes the limiting factor, and it ships as a single config file. A dedicated read-only reporting replica, refreshed continuously via logical replication, gives the BI team a sandbox to run expensive aggregate queries against without touching production latency at all.

None of these require adopting a new category of infrastructure. They are configuration and topology changes to the database you already run, and in combination they push the point at which a single logical Postgres deployment stops being sufficient much further out than the twelve-person startup’s dashboard traffic will ever reach. The honest sequence is: index properly, then partition, then add a read replica, then consider a pooler, and only after all four are in place and still insufficient does the conversation about a genuinely distributed system start to make sense. Most teams currently running a distributed pipeline never exhausted the first rung.

Where the real ceiling actually is

None of this is an argument that distributed data systems never earn their keep. There is a real ceiling, and it is worth naming precisely so the decision is about your numbers rather than a vague fear of “running out.”

You are approaching genuine big-data territory when a working set stops fitting in RAM on the largest single machine you can reasonably provision, when write throughput sustained over hours exceeds what a single primary can commit to disk even with good batching, or when the actual requirement is fault-tolerant, replayable, ordered event streams consumed independently by several already-existing services rather than a request-response query. Those are specific, measurable conditions. “We might get big one day” is not one of them, and neither is “the last company I worked at used Kafka.”

The pattern in most machine learning is a spreadsheet you won’t admit to is the same shape: tooling built for one scale gets adopted at another because it carries the reputation of serious engineering, regardless of whether the numbers actually call for it. A database index is just a sorted list you paid for covers the mechanism that lets a single well-indexed Postgres table answer in milliseconds what a naive distributed scan would take seconds to coordinate.

What the distributed platform actually costs you

The case against premature adoption is not aesthetic. It is the specific, recurring cost of running infrastructure a problem does not need.

A Kafka cluster needs a minimum of three brokers to be genuinely fault-tolerant, each one a machine to patch, monitor and pay for, plus a schema registry if you want any discipline around message formats, plus consumer group offset management to reason about when something goes wrong. Spark needs a cluster manager, executors sized correctly for the job, and someone on the team who understands partition skew, because a single skewed key silently turns a ten-minute job into a four-hour one and the failure mode looks nothing like an error message. A data lake needs a table format like Iceberg or Delta to avoid becoming an unsearchable pile of files, a catalogue service to track schema, and a compaction job running on a schedule so small files do not eventually make every query slow.

Every one of those pieces is a real, documented, well-understood technology. None of them is free. Each is a service that can fail independently, a version to upgrade, a configuration surface that can be misconfigured, and a skill the team needs to have on hand at the moment it breaks — usually not during business hours. A twelve-person startup adopting all of this to process 40,000 rows a day has taken on the operational burden of a data platform team without the data volume, or the headcount, that would make it worth the trade.

There is a second, quieter cost that rarely makes it into the decision at all: hiring and onboarding against the architecture you actually built. A new engineer joining a company running Postgres needs to learn one system, well documented, with decades of accumulated troubleshooting knowledge searchable in seconds. A new engineer joining the twelve-person startup with the full Kafka-Spark-lake stack needs to learn four systems, each with its own operational quirks, before they can safely touch the pipeline that processes 40,000 rows a day — a volume that, on the simpler stack, they could have safely modified in their first week.

Troubleshooting: telling the difference before you commit

A short set of honest checks catches most premature adoption before infrastructure gets built around it. Run EXPLAIN ANALYZE against the actual query pattern on a single well-specified Postgres instance, loaded with realistic production-scale data volume:

1
2
3
4
5
6
7

EXPLAIN ANALYZE
SELECT customer_id, SUM(amount)
FROM orders
WHERE created_at > now() - interval '30 days'
GROUP BY customer_id
ORDER BY SUM(amount) DESC
LIMIT 100;

If that returns in milliseconds against your real row counts with a sensible index on created_at, the analytical case for a distributed engine has not been made yet. Check whether the actual write rate, measured rather than guessed, is anywhere near what a single Postgres primary can sustain with reasonable batching — for most startups it is a rounding error against that ceiling. Ask whether the “streaming” requirement is genuinely about multiple independent consumers replaying an ordered log, or whether it is actually a request for near-real-time updates, which LISTEN/NOTIFY or a lightweight change-data-capture tool handles without a broker cluster. And ask, honestly, whether the architecture is being chosen because the numbers require it, or because it is the diagram that gets shared at a conference.

Is a single database ever the wrong call

Yes, plainly, once the measured numbers cross the thresholds above, and choosing a distributed platform at that point is the correct engineering call rather than overkill. The genuine mistake is adopting one before the workload demands it, carried by momentum and reputation rather than a number anyone actually checked. Run the arithmetic first. For the overwhelming majority of startups, the honest answer is that a single well-specified Postgres server, properly indexed and given reasonable hardware, serves as the real data platform for years longer than most teams assume, and the fastest path to slow queries is skipping the arithmetic that already fits comfortably on one box.