Velox at Uber: Rebuilding Presto's® Execution Engine for Speed and Scale

Introduction

Every day, Uber’s Presto® fleet answers hundreds of thousands of queries from engineers, data scientists, and analysts across the company, powering everything from ad hoc investigations to scheduled production pipelines. At our scale, that means dozens of clusters, thousands of hosts, and hundreds of petabytes of data being processed daily. It also means that even small inefficiencies in query execution add up to real infrastructure cost.

Over the past two years, Uber’s Presto team has been rebuilding the engine at the heart of that fleet. Instead of running queries through Java's JVM-based engine, we've been moving execution onto Velox, an open-source, vectorized C++ execution engine originally built and open-sourced by Meta® . The result,  which the open-source community calls Prestissimo, is Presto running on Velox: the same SQL, on the same clusters, running dramatically faster and cheaper.

Today, Velox powers roughly 90% of eligible Presto traffic at Uber, processing on the order of 100 petabytes of data a day. Query success rates have gone up, average execution time has dropped by more than half, and CPU consumption per query has fallen by roughly two-thirds compared to the legacy Java engine, all while running the exact same queries our users already wrote.

This blog walks through that journey: why we made the bet on Velox, the unglamorous infrastructure work that had to happen before we saw any of the payoff, the performance numbers and how we actually validated them, and what’s still left to close the gap on the traffic Velox doesn't yet handle..

Why Velox

Presto at Uber spends a lot of CPU on query execution. When we profiled our clusters a couple of years ago, we found that more than 90% of CPU cycles were being spent inside a single stage of query execution, the operator responsible for scanning data off disk, applying filters, and projecting columns.  The JVM's overhead boxed objects, garbage collection, and bytecode that can't reliably auto-vectorize into SIMD instructions  was never going to be great at grinding through that much data efficiently, no matter how much we tuned it.

Figure 1: Presto CPU time by execution stage.

We began exploring Velox for Uber’s Presto deployment in December 2023. The early proof of concept was enough to get everyone’s attention: on scan-heavy queries, exactly the kind that dominate our CPU budget, we saw up to 7x savings in CPU time. That was a big enough number to justify a much deeper investment than a side project.

Early Challenges

The promise of Velox was obvious pretty quickly. Getting there was not. Uber’s data infrastructure runs mostly on Go, Python, and Java, so C++ had a comparatively small footprint in our stack, and adopting a C++-native execution engine meant running into a bunch of problems that had nothing to do with query performance.

Our internal tooling for C++ was immature. Build systems, debugging workflows, CI integration, all the things we took for granted with Java and Go simply hadn’t been built yet for a large, fast-moving C++ codebase like Velox. Dependency management became its own headache, since Velox and Presto’s C++ worker pull in a nontrivial set of native system libraries that needed to be vendored, versioned, and kept consistent across our fleet. Observability was thin, too. We’d spent years building tooling around the JVM, things like heap dumps, GC logs, and JMX metrics, and none of that translates to a native process. We needed equivalent visibility from day one if we wanted to run this safely in production. And in the early days, worker crashes and out-of-memory kills were common, which, without proper observability, made root-causing anything painfully slow.

None of these problems were really about Velox’s execution logic. They were about the operational reality of running a new kind of process at Uber’s scale. Solving them turned out to be the real first phase of the project, even though it’s the part that never shows up on a performance chart.

Building the Foundation

Before Velox could carry meaningful production traffic, we had to give it the same kind of operational maturity we already had for Java Presto.

For observability, we integrated Prometheus® into the Prestissimo process, working off a proposal we developed together with IBM^®. The design runs a small subprocess alongside the main Prestissimo process, buffering metrics with the prometheus-cpp library and exposing them over HTTP. Uber’s M3 metrics platform, our large-scale, open-source, Prometheus-compatible back end, scrapes that endpoint, which gave us the same dashboards and alerting for Velox that we already relied on for Java Presto.

Figure 2: Prestissimo-to-M3 metrics pipeline.

Memory management turned out to be its own project. Getting it right in a native process is unforgiving in a way the JVM's garbage collector normally hides from you. We adopted Velox's memory arbitrator running in shared mode, paired it with the mmap allocator, and spent a lot of time tuning memory limits, pushback thresholds, and per-query caps until clusters stayed stable under real production load.

Rather than flipping traffic over all at once, we rolled Velox out in two distinct routing phases. Phase 1 got us from 0% to roughly 30% migration: every query was shadow-tested first, run on both Java and Velox with results compared side by side, and only fingerprints that came back clean were allowlisted onto Velox — everything else defaulted to Java. Phase 2 took over from there. Shadow-testing every fingerprint at scale stopped being practical, so we flipped the model: instead of an allowlist of known-safe queries, we now run a blocklist of fingerprints known to crash Velox, and let everything else through by default. That opt-out model is what's carried us from 30% to the roughly 90% of eligible traffic Velox handles today.

Figure 3: Query routing decision flow.

Closing the Format Gaps: Parquet

Most of Uber's data lake lives in Apache Parquet®, so Parquet compatibility was never optional, and it turned out to be an early source of some pretty subtle bugs.

On the read side, we found and fixed problems with how Velox's reader handled null values inside complex types like structs, arrays, and maps, which had been producing incorrect query results in some cases. We treat that category of bug as about as serious as it gets, since a silently wrong answer is worse than an outright failure. We also worked through a series of backward-compatibility issues affecting older Parquet files written by earlier versions of our stack, and built a test framework that reuses Presto's existing Java-side unit tests to generate Parquet files, then checks Velox's reader against them. That turned years of accumulated Java test coverage into a useful safety net for the new engine.

The writer told a different story. When we evaluated Velox's Parquet writer, we found it wasn't yet a drop-in replacement everywhere. It didn't support end-to-end schema evolution, meaning you couldn't cleanly update a table's schema over time. Some time-related types, including TIME and TIMESTAMP, weren't fully supported. Compression ratios for numeric types and MAP columns were noticeably worse than what non-native Presto produced, which we traced back to a different encoding policy in Velox's writer. And tables written by Velox's writer couldn't always be read back by Spark SQL, which is obviously a problem in an environment where the same tables get consumed by multiple engines.

We've been upstreaming fixes for the encoding differences, and CLAC support (Uber's columnar access and compression layer) has become one of the bigger contributors to closing that gap. More recently, our focus at the reader layer has shifted toward production stability, fixing heap fragmentation issues that showed up under sustained load and auditing our open-source configuration defaults to catch places where what we assumed didn't match what was actually running.

The Payoff: Performance Results

All of that groundwork existed for one reason: making Presto queries faster and cheaper to run. We’ve measured that payoff a few times now, each time with more rigor than the last.

Our first head-to-head comparisons ran the same set of more than 40,000 queries across over 1,000 unique query shapes. At matched capacity, Velox delivered a 3.73x improvement in CPU time and a 2.5x improvement in elapsed time. Even more interesting, we found we could run the same workload with 30% less capacity on Velox and still come out ahead of Java at full capacity, with a 3.55x CPU improvement and 1.47x faster elapsed time.

To go beyond early benchmarks, we later ran a formal Performance Improvement Factor study to certify a new 48-core machine configuration, colocated with Alluxio, for production Velox deployment. This is the same methodology we'd previously used to evaluate newer-generation compute options, and instead of relying on synthetic tests, it's built entirely around shadow testing live production traffic.

The setup mattered as much as the result. We mirrored real query traffic from a production Velox cluster into two identically sized test clusters, one running Java and one running Velox, using an enhancement to our internal traffic-shadowing tool, Modulator, that let us replay queries with matching submission timing instead of waiting for the original query to finish first. Because Velox couldn't yet run all of Java's traffic (at the time, only 50 to 60% of overall traffic was Velox-eligible), we deliberately used Velox-eligible traffic as our shadow source, so both clusters were being judged against the same fair workload. We also ordered the shadow sessions specifically so the Java cluster got pushed to its actual limits rather than being shielded by a slower feeder upstream.

Figure 4: PIF shadow-test traffic flow.

Figure 5: CPU utilization during the PIF test, Java vs. Velox.

Those results have shown up clearly in our fleet-wide metrics, too. Over the past year or so, the number of Velox-capable Presto clusters has grown roughly twenty-fold, from a single cluster to more than two dozen, with the sharpest acceleration coming in the last several months as configuration tuning matured and workload eligibility expanded. Velox's share of both data read and total query volume followed the same curve, overtaking Java around the end of last year and now accounting for the large majority of traffic on Velox-enabled clusters. Today, Velox handles roughly 100 petabytes of data a day at Uber, double what it was handling when we first started sharing progress publicly.

Figure 6: Velox fleet growth over time.

Figure 7: Velox data read growth over time.

Beyond the Core: Expanding Velox's Reach

Once the core engine was stable and fast, we started pointing Velox at problems that had previously been out of reach or too expensive to solve well.

Geospatial turned into a bigger deal than we expected going in. Geometry functions account for roughly 15% of all Presto CPU time at Uber, which isn't shocking for a company whose core product is built entirely around geospatial data, but it was still a lot of compute concentrated in one place. We've contributed geometry support upstream to the open-source geospatial roadmap, including a merged dependency for geometry manipulation and a geometry type implementation, with serialization support in progress. Internally, we've gone further and implemented functions like ST_Contains and cast operators for geometry types, along with an initial geospatial join using a nested-loop strategy.

We've also kept iterating on how traffic actually gets routed to Velox, rolling out enhanced routing in phases: first shadow-mode validation, then retry logic, then crash-loop protection, so routing decisions stay resilient even when individual queries or clusters misbehave.

Staying Close to Open Source

None of this would have been sustainable if we'd gone off and built a heavily forked, Uber-specific version of Prestissimo and Velox. Early on, we made a deliberate call to keep our fork as close to upstream as possible and to pull changes from open source on a regular cadence, roughly weekly at this point.

That decision has paid off in ways that are easy to underestimate. Staying close to the master branch means we're rarely chasing bugs that have already been fixed upstream, and a single version bump can bring in hundreds of upstream commits at once, which is the opposite of wasted effort. It means we're not solving problems alone: issues we find and fix become community issues too, and the fixes land for everyone, not just us. It means we know exactly what we're pulling in, so we can make informed calls about which upstream features to turn on and when. And it saves us from the trap of dual maintenance, since features like CLAC support and schema evolution have historically conflicted with upstream master, and staying close cuts down how often we have to reconcile our own changes with a codebase that's moving fast.

Just as importantly, it's pushed us to actually engage with the open-source community rather than working in isolation, contributing PRs, joining design discussions, and picking up practices and debugging tricks we probably wouldn't have found on our own.

Closing the Last Mile

Velox now handles the majority of eligible Presto traffic at Uber, but “eligible” is doing some real work in that sentence. A meaningful slice of workloads still can't run on Velox today, and closing that gap is where most of our current effort goes. We're working on retiring retries on Java clusters, which today act as a safety net for queries that don't yet succeed reliably on Velox. We're filling in legacy UDFs that existing pipelines depend on but that don't have Velox equivalents yet. We're chasing down the remaining Parquet tail issues and residual crash scenarios that only show up at the long tail of production traffic. And we're closing out missing feature support, including data anonymization, CLAC writer support, and connectors for systems like Kafka and MySQL that some workloads still depend on.

Looking further out, we're exploring a mixed-worker cluster setup that would let a single cluster route queries needing unsupported catalogs, like Kafka, Pinot, or MySQL, without falling back to an entirely separate Java cluster. We're continuing to invest in geometry functions and spatial joins, in growing the percentage of query shapes with full Velox coverage, and in migrating whatever batch workload is left. Longer term, we're aiming for a Phase 3: shrinking the crash-fingerprint blocklist down to nothing, so routing decisions stop being fingerprint-based at all.

Conclusion

Adopting Velox has been one of the more consequential infrastructure investments Uber's data platform team has made in the past couple of years. Getting here meant building C++ tooling and observability from scratch, patiently closing Parquet compatibility gaps, and validating every performance claim with rigorous, production-grade testing instead of taking synthetic benchmarks at face value.

The payoff has been substantial: roughly 2.4x faster queries, 3x lower CPU usage, based on our most recent and rigorous production benchmark and meaningfully better reliability, running across tens of clusters and around 100 petabytes of data a day, with Velox's footprint still growing. We're grateful to the broader Velox and Presto open-source communities, including Meta's Velox team and our collaborators at IBM, for the shared foundation that made this possible, and we're committed to keep contributing back as we push further.

Acknowledgments 

Cover Photo Attribution: "Stocksy_txp98f576efvPH300_OriginalDelivery_3754794" by YiChao Ma is licensed by Uber. 

Presto® and Prometheus® are registered trademarks of LF Projects, LLC.

Apache^®, Apache Kafka^®, Apache Spark™, Apache Parquet^®, and Apache Pinot^™ are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

PyTorch, the PyTorch logo and any related marks are trademarks of The Linux Foundation.

Alluxio® is a registered trademark of Alluxio, Inc.

The name "IBM", the IBM logo, and ibm.com are registered trademarks of International Business Machines Corporation in the United States and other countries worldwide. [1]

Meta® is a registered trademark of Meta Inc.

Oracle, Java, MySQL, and NetSuite are registered trademarks of Oracle and/or its affiliates.