Making Every Core Count: In Pursuit Of Faster Apache Spark Jobs at Uber

Why We Went Looking

Uber runs one of the largest Apache Spark™ fleets in the industry. Data lake ingestion alone consumes more than 200,000 CPU cores, and the ETL built on top of it consumes more. At that scale even a few percent of efficiency is real impact, so making Spark cost efficient is a never-ending program at Uber rather than a one-time project. Most of that work runs through Drogon, Uber's Spark-as-a-Service platform, which controls how every Spark application is submitted, configured, and rolled out, making fleet-wide engine changes tractable. To optimize query execution for some of the critical production workloads, Uber adopted Apache Gluten™ and Velox , offloading Spark SQL operators to a native vectorized engine: roughly 3.5× faster queries and 3.4× lower CPU cost across migrated workloads and up to 79% runtime improvements, with no user code changes. Uber also contributed more than 30 patches upstream, and one engineer was invited to become an Apache Gluten committer.

Building on our gains with Gluten/Velox, we looked for additional improvements in ingestion and table maintenance; the largest share of Spark compute at Uber. These workflows involve more than SQL operators: they read existing data, apply updates, write files, and maintain table metadata. Most tables in Uber’s data lake are plain Parquet datasets. About 17200 tables are managed with Apache Hudi™ and Hudi workloads account for a substantial portion of ingestion compute, making their read and write paths an important optimization target.

To evaluate this opportunity, we benchmarked Onehouse Quanton® as a candidate storage-aware Spark engine. We used synthetic data modeled on production workload patterns in a controlled test environment. On a 60-billion-record insert-heavy fact table using 1,080 vCPUs, we observed a 5.5× median write-round speedup. On a 10-billion-record update-heavy dimension table using 3,300 vCPUs, the speedup was 4.7×. Each comparison used the same executor allocation for Quanton and the OSS Spark/Hudi baseline; these results exclude metadata-compaction rounds.

This blog outlines our Spark footprint, internal Gluten/Velox production outcomes, Quanton benchmark evaluation results, and future directions for lakehouse execution.

1. Uber's Spark Footprint and Native Vectorization with Gluten/Velox

Apache Spark has long been a cornerstone of Uber’s data infrastructure, enabling everything from data analytics to machine learning.
Every day, Uber's Spark infrastructure handles over 100 PB of data read and written. The platform manages over 80 PB of shuffle data daily while executing workloads across more than 59,000 unique fingerprints. Uber's Spark infrastructure supports over 500,000 applications daily spanning over a million compute cores. 

What Is Gluten/Velox

Apache Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines like Velox. Offloading JVM execution to native engines unlocks powerful capabilities such as running SIMD (single instruction multiple data) which allows faster and efficient execution for compute engines like Spark.

Benchmarking and Results

TPCDS Runtime Comparison:

Figure 1: Runtime Comparison for TPCDS Benchmark with various conditions

Based on the TPC-DS benchmarks, we see a best case improvement of 4x with 64 executors. Further, the worst case improvement was of 2.7x when running Gluten on 32 executors. An average overall improvement of ~3.4x is visible across all scenarios. 

Query by Query Perf Comparison - Native Shuffle:

Next, we compare individual query/df performance and their durations, the following graph shows the duration per query for the 3 iterations of the TPCDS benchmark with 64 executors in Vanilla Spark vs Gluten.

Figure 2: Query by query performance comparison of Spark vs Gluten

This is seen in native shuffle where an additional overhead of columnar to row transformation is incurred. In this specific query Gluten took 1min compared to spark 10 seconds for the additional row shuffle overhead.

The major overhead in the native shuffle is due to the row column transformation; Gluten Native suffers for shuffle intensive queries in the TPC-DS suite. For this reason, next we explore using celeborn columnar shuffle (sort) where we aim to eliminate this overhead. Further the Native shuffle of Gluten is not optimal and often spills to disk.

Query by Query Perf Comparison - Celeborn Columnar Shuffle:

When using Celeborn's sort shuffle with Gluten, we see an average ~3.65x improvement with Gluten, which is better than the previous results of using Native shuffle. Majorly this improvement is seen due to the elimination of the row transformations during shuffle. 

Figure 3: Query by query performance comparison of Spark vs Gluten (Celeborn)

Results on Production

We enabled Gluten/Velox on select production spark workloads and observed significant gains.

• Runtime Improvement: Upto 79% avg runtime improvement (faster query execution) 
• Resource Efficiency: Upto 98% reduction in cost for critical pipelines.

Production Readiness

To achieve production readiness at scale, Uber contributed over 30 patches (and 50+ commits) upstream to open source. Key work done so far includes:

• Fallback Implementation: Implemented fallback mechanisms to mitigate cost increases from row-to-column conversions and tuned memory configurations (splitting off-heap memory for Velox and on-heap memory for other operations).
• Security Features: Added support for encryption and decryption directly in native C++ Parquet readers and writers to support critical pipelines with encrypted Parquet files.
• Operator & Function Support: Implemented missing operators (e.g., Left Semi Join, Existence Join, RangeExec, Collect Limit) and functions (e.g., unix_timestamp, cast functions, date functions, precision round, ANSI support, and custom UDFs).

2. Evaluating Quanton for Storage-Aware Optimization

What Is Quanton

Quanton is Onehouse’s commercial Spark engine. Like the Gluten/Velox approach described earlier, it uses native vectorized execution to accelerate SQL processing. It also includes optimizations to query planning and how data is read, written, and managed in Apache Hudi and Apache Iceberg tables:

• Extract: Native readers handle table-format features, such as Hudi merge-on-read logs and Iceberg deletions. Pipelined I/O overlaps data access with computation.
• Transform: Native vectorized operators accelerate SQL execution, while Spark planner optimizations reshape queries to reduce work such as duplicate scans.
• Load: Optimizations to write and merge operations, file creation, indexing, and table maintenance target the work required to apply changes to existing tables.

Quanton is designed to run existing Scala, PySpark, and SQL applications without changes to application code or SQL.  This evaluation focused on the write path, where most of Uber’s ingestion compute is spent; Quanton’s SQL and read-side acceleration was not benchmarked here.

Benchmarking and Results

We benchmarked Quanton in a test environment using synthetic data designed to reflect the scale, schema shape, and workload patterns of Uber’s production tables.

The Test Tables

We picked two very large tables, one >90% insert and one >90% update, to cover both scenarios. Within each benchmark, table properties are identical between the OSS Spark and Quanton runs; the two tables differ by design.

Here’s the cluster and engine configuration, close to production:

• Spark 3.5.9 and Hudi 1.1.1 on both sides.
• Dedicated, isolated HDFS cluster and on-prem Kubernetes on test tables, standard practice for benchmarking and unrelated to production data protections; this rules out public cloud network latency variance.

Benchmarking Results

Per-round latency was measured from the start of one round's write job to the start of the next, reconstructed from Hudi commit instants and cross-checked against Spark History Server. Both systems hit a metadata-table compaction roughly every 6 to 7 commits; those rounds are excluded from the steady-state figures below so core engine performance is not conflated with table maintenance. On the fact table, the compaction-affected round ran 60.42 minutes on OSS Spark against 21.27 on Quanton, a smaller 2.8× gap than steady-state write performance. 

Figure 4: Steady-state commit latency on the insert-heavy fact table (60 billion records, 1,080 vCPU)—average, median, and p95 across 12 commits, OSS Spark against Quanton.

Figure 5: Steady-state commit latency on the update-heavy dimension table (10.16 billion records, 3,300 vCPU)—average, median, and p95/max, OSS Spark against Quanton.

Measuring Scaling: OSS Spark Versus Quanton

At Uber’s scale, a key question for any Spark engine is how its gains scale with compute. We replayed the fact-table workload on clusters of 141, 564, and 1,128 vCPU (3, 12, and 24 executors of 47 cores and 300 GB each), holding table, index, and engine configuration fixed, with 2 commits per configuration measured the same way.

Figure 6: Commit latency on the fact-table workload as the cluster is cut from 1,128 vCPU to 141 vCPU, where the OSS Spark bar at 564 vCPU is extrapolated from the measured 141 and 1,128 vCPU runs.

Task skew may help explain this difference. At larger allocations, a few long-running tasks can determine stage completion time while other cores sit idle. At smaller allocations, queued tasks can use capacity alongside those longer-running tasks. Median task durations were approximately 10× shorter with Quanton at both cluster sizes, although confirming the contribution of scheduling requires stage-level utilization analysis.

The practical result was comparable completion time with substantially fewer resources: Quanton completed a write round in 69.8 minutes using 141 vCPUs, versus 78.9 minutes for OSS Spark using 1,128 vCPUs. Under fixed executor allocation, this corresponds to 89% fewer provisioned executor vCPU-hours per round. These preliminary measurements cover two commits per measured configuration; sustained efficiency will be evaluated through production integration and comprehensive load testing.

Data Analysis: Where Quanton Actually Improved

The largest measured improvement was in the data-table write phase, which decreased from 475.15 to 60.77 minutes—a 7.8× speedup. Metadata-table writes also improved by 2.8×. These results show that the gains extended to both data-table and metadata-table writes.

Other phases showed little improvement. Metadata-table compaction and small-file partitioning remained approximately unchanged, while deduplication and indexing took about 9% longer. These measurements help identify the remaining optimization opportunities: accelerating data-table writes does not eliminate the costs of indexing, partitioning, and table maintenance.

3. Summary and Future Directions

In this article, we presented our experience of improving Spark  performance for selective production workloads at Uber through native vectorized execution with Gluten/Velox. Our experience showed that choice of the shuffle strategy - native vs Celeborn can have a significant impact on query performance.  We also presented our benchmarking of Quanton - a spark distribution from OneHouse that adds optimizations to query planning and how data is read, written, and managed in Apache Hudi and Apache Iceberg tables. Uber runs a large number of Ingestion workloads written in Spark with Hudi as the table format. 

Our Quanton evaluation focused on write-heavy Hudi ingestion pipelines, with the largest measured gains in data-table writes. Onehouse describes Quanton as accelerating ETL workloads more broadly, including SQL transformations. We will expand testing across additional ingestion and ETL pipelines through production integration and comprehensive load testing, evaluating correctness, compatibility, reliability, and sustained performance, including table maintenance and failure recovery.

Acknowledgments

Thanks to the Onehouse engineering team for the Quanton builds and for walking us through the engine internals, and thank you to Uber's Batch Data Platform teams and Core Infra team for the test environment:  Silvio Simunic, Guru Chaitanya Ganta, Swastik Udupa, Nagi Parimi, and Kayode Yusuf for provisioning and sizing the isolated Kubernetes cluster; Fengnan Li for the dedicated HDFS setup; Mike Kaufman, Manjula Piyumal De Silva, and Gediminas Šimaitis for unblocking the container runtime and image registry; Kalyan Sivakumar for Spark platform guidance; Sonal Desai and KK Sriramadhesikan for the security review; and Mohammad Islam, Ankit Toshniwal, Suprit Acharya, Vijay Mann, and Viv Keswani for sponsoring the evaluation. Thanks also to Rajesh Mahindra of Onehouse.

Cover Photo Attribution: "Olympics Go Get It" is licensed by Uber.

Apache, Apache Hudi, Hudi, Apache Iceberg, Iceberg, Apache Spark, Spark, Apache Parquet, Parquet, Apache DataFusion, DataFusion, Apache Gluten (Incubating), Gluten, and the star logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks. 

Kubernetes® and its logo are registered trademarks of The Linux Foundation® in the United States and other countries. No endorsement by The Linux Foundation is implied by the use of these marks.

Quanton and Onehouse are trademarks of Onehouse, Inc.