With over 2 million Apache Spark^™ applications running daily, Uber operates one of the largest Spark deployments in the industry. Migrating Spark versions is no simple feat, given the scale and complexity of our operations. This blog delves into how Uber successfully navigated this migration, and the innovative automation and tooling we developed to make it possible.

Apache Spark has long been a cornerstone of Uber’s data infrastructure, enabling everything from data analytics and machine learning to real-time processing. Today, users launch over 2 million Spark applications through more than 20,000 scheduled workflows and thousands of interactive sessions each day. Until recently, all of these workloads ran on Spark 2.4. This blog explores the transition from Spark 2.4 to Spark 3.3 and the challenges faced along the way.

Due to its out of the box support for Kubernetes as a resource manager, we decided to move to Spark 3.3 to:

At Uber, Spark applications are written in Java^®, Scala, or Python. Either Apache Hadoop^® YARN or Kubernetes^® handles resource management, with both options supported in production.

Jobs can execute in two primary modes:

The source code for these jobs lives in either Uber’s monorepo or micro-repos, depending on the team’s architecture and ownership model.

Typically Spark jobs at Uber are submitted through Drogon (our proxy to Apache Livy^™) which is integrated throughout the ecosystem powering Jupyter^® notebooks, PySpark, Java applications, Spark shells, and more. The app is submitted through Livy to YARN or Kubernetes based on user preferences. There are several observability hooks powering observability, chargeback, and analytics to debug failures. We also have an in-house external shuffle manager called Zeus which is used for most applications with fallbacks to ESS.

The migration process involves 4 steps:

The first step is to rebase Uber’s Spark fork with the OSS release. Our internal Spark fork has evolved over the years. For example, we have task-level event listeners that publish memory/CPU consumption. This is used for profiling and optimizing applications. To improve security and compliance, we’ve also  coded out column-level lineage and access control support. To reduce local-SSD wear-out, we employ a custom remote shuffle manager called Zeus.

Getting the Spark ecosystem ready for a new migration was all the more challenging. Below is the list of all the dependencies that we had to sort out before we could attempt any migration.

Our Spark apps are written in a Python monorepo as PySpark jobs or they live in a Java monorepo. We have over 2,100 applications of Spark source code. To handle code changes at scale, we had to employ bulk parser/rewriter utilities. We chose Uber’s open-source tool called Polyglot Piranha​, originally built to clean up stale feature flag code. By extending its capabilities, we used it to handle the code changes required when migrating from Spark 2 to Spark 3.

Here’s how Polyglot Piranha works: 

To leverage Piranha, we defined structured rules needed to identify code patterns and changes required for Spark 3.

Let’s first take an example of a basic code change required for a basic Spark application.

// Before upgrade

val conf = new SparkConf()

  .setMaster(master)

  .setAppName(appName)

  .set(“spark.driver.allowMultipleContexts”, “true”)

sc = new SparkContext(conf)

sqlContext = new TestHiveContext(sc).sparkSession

Our migration tool identifies when SparkConf is initialized and adds relevant set configs accordingly.

// After upgrade

val conf = new SparkConf()

  .setMaster(master)

  .setAppName(appName)

  .set(“spark.driver.allowMultipleContexts”, “true”)

  .set(“spark.sql.legacy.allowUntypedScalaUDF”, “true”)

sc = new SparkContext(conf)

sqlContext = new TestHiveContext(sc).sparkSession

In this case, the tool adds the legacy configuration spark.sql.legacy.allowUntypedScalaUDF needed for Spark 3 seamlessly.

During the evaluation and proof of concept, we noticed multiple behavior changes with the newer Spark version. For example:  

Given the above, it was imperative to do shadow testing and data validation. But doing data validation brought challenges. We had over 40,000 Spark apps, so we couldn’t decentralize the data validation. We also had no notion of a staging environment and couldn’t dry-run in production as it may affect production data. Further, no test cases existed. Even if we could solve the staging environment issue, we didn’t know the baseline of the data to validate against. 

There were no OSS utilities that solved the above, so we built a new framework called Iron Dome that could safely test a Spark job with these features: 

We saw major improvements in efficiency, duration, and cost across jobs, including: 

As part of this work, several patches were open sourced, marking this a major infrastructure upgrade for Uber.

The migration also opened up avenues such as Kubernetes and JDK17. Beyond this migration, the agnostic automation build is being used for other upgrades.

We successfully upgraded 100% of all Spark applications to 3.3. This migration resulted in a 50% reduction in runtime and resource usage overall. With the Spark upgrade, several frameworks such as Iron Dome were built, which provided general-purpose sandboxing of Spark applications and paved the way for future Spark upgrades. This accelerates bringing newer Spark features and version upgrades without requiring user intervention. Going forward, we plan to increase coverage for our tooling and support the upcoming Spark 4 upgrade, along with using the tooling for other key infrastructure upgrades.

Stay up to date with the latest from Uber Engineering—follow us on LinkedIn for our newest blog posts and insights.