At some point in every infrastructure company’s life, there’s a graph nobody wants to look at anymore. Ours was query latency. Specifically the horrifying upward slope that appeared every time a customer crossed a few hundred million events.
Fast at small scale is easy
The first version of our query engine was built during a week when everyone was drinking too much cold brew and saying things like “we’ll optimize later.”
To be fair, it worked.
In 2023 most customers were querying datasets measured in megabytes, not terabytes. Response times felt instant. Everyone was happy.
Then enterprise customers arrived.
One client uploaded three years of event history on a Friday afternoon and accidentally taught us what pain feels like.
Queries that used to take 400 milliseconds started taking twelve seconds.
Nobody enjoys staring at a loading spinner while pretending analytics are “real-time.”
What was actually broken
The bottleneck wasn’t compute.
It was architecture.
The original engine recalculated too much data too often because it assumed workloads would stay relatively predictable. That assumption lasted about eleven minutes into enterprise adoption.
We also discovered an uncomfortable truth:
Most dashboards repeatedly query the exact same slices of data.
We were regenerating answers users had already asked for thousands of times.
The rebuild
The new engine introduced three major changes:
Intelligent caching for repeated query paths
Columnar storage for high-frequency metrics
Incremental aggregation layers
Which sounds very impressive until you remember it mostly translates to:
“Stop recalculating things unnecessarily.”
We also rewrote the planner itself so queries could prioritize speed or accuracy depending on the use case.
Nobody needs millisecond-perfect data for a weekly marketing report.
They do need it for anomaly detection.
Six months later
The last one matters most.
Nobody wants an email titled:
“Exciting infrastructure improvements! Please rebuild everything.”
What we learned
The hardest part wasn’t engineering.
It was timing.
Infrastructure rewrites always compete against visible roadmap work. Nobody tweets excitedly about internal storage optimizations.
But eventually the invisible systems become the product.
Especially when they stop working.
Harvey Brooks
Chief Technical Officer, Metrify
Writes about async teams, internal communication and modern operating rhythms. Previously built remote ops systems for distributed SaaS teams.
Read More
