During our inaugural Uber Technology Day, software engineer Aimee Lucido delivered a presentation on the history of Uber Engineering’s Android codebase. In this article, she expands on the reasons behind Uber’s decision to build a monorepo to support the growth of our Android development.
Today is the day you are going to build a brand new Android app—and good for you, getting started is always the hardest part. What is the first thing you do?
If you are like me, you will create a new project in Android Studio. You make a main activity, hook up Gradle, and maybe even create a git repository so that your friends can collaborate with you on the app. Congratulations! Your code organization now resembles the first version of the Uber Android rider app.
When we launched our Android rider app in 2010, we were a small company. Uber Engineering was under a dozen people. We had one contractor working on the Android platform, and—if you can believe it—we did not even have an Android driver app. So, our single Android engineer wrote the first version of the rider app in a single repository: one big box of code.
Creating a new app using a single repository had some benefits early on:
- Out of the Box Android: A single repo is the free code structure offered in Eclipse, the IDE used for the first version of the Android rider app. Today in 2017, building an early codebase in a single repository is even easier than it was in 2010 because we have Android Studio, which partners closely with Android libraries. But no matter which IDE you use, an IDE will provide a single repository structure out of the box and basic tooling, like build scripts and git integration
- Fast Development for Small Teams: Since all dependencies are in one place, using a single repository allows things to move fast for a small team of engineers by simplifying code-sharing and refactoring.
For several years, the state of Android at Uber was small scale. By 2013, we had hired our first full-time Android engineer, and over this period our engineering team had more than doubled. Only then did we start building an Android driver app.
Building the driver app gave us the opportunity to improve our codebase organization. The rider app still existed in a single codebase, but since we now had the resources and tooling to extract reusable components, we did just that. The core partner code existed in its own separate repository, but we also built out a library full of reusable components for the two apps.
This general structure served us well for awhile, but in 2014, our subsequent growth called for a different solution. Uber had over a hundred engineers. The Android engineering team had grown in size from one to eight engineers. As our number of engineers was growing, so too did our codebase.
We took a look at the direction that we were heading and we realized that if we did not change, we were going to encounter the following issues:
- Long Build Times: Our initial Eclipse project used Ant as its default build tool, which has a tendency to slow down when dealing with large codebases.
- Feature Coupling: A downside of sharing code easily is that it sometimes can be overshared. As the number of Android app features increased, we worried that features would start unnecessarily coupling together.
- Broken Master: How often have you rebased your change onto the latest master, hit a build failure… and then spent an hour debugging it. Then you realized that your build failure had nothing to do with your own code, and everything to do with the person rebasing their code without re-running tests right before you! If you are like me, you have been on both sides of that equation. With multiple engineers contributing to the same codebase and without investing heavily in continuous integration tooling (see Submit Queue below), you risk people landing diffs at the same time that lie in conflict with one another. This means a broken master and wasted hours.
So during the 2013-2014 time period, we made a series of changes to transition to a multirepo codebase in order to preempt these problems. In 2013 we moved our rider app from Eclipse with Ant to IntelliJ with Maven, allowing us to pull artifacts from a server and breaking our library codebase into 20+ smaller decoupled repos. (For example, networking was moved to its own repository, later to be pulled into the consumer apps via a Maven repository at compile time.) Simultaneously, we transitioned our build scripts to Gradle, marking our first foray into a multirepo world.
Our Uber multirepo codebase consisted of several little codebases each representing a single, discrete idea stored as artifacts that are pulled in at compile time by the rider and driver apps. Each repository is like a smaller box of code, with its own IDE project, git repository, and build script. By moving to multirepo, we ensured a solid, future-proof architecture, circumventing problems like long build times, feature coupling, and a broken master.
So the question now becomes: why did we not start with a multirepo from the beginning? In one word: overhead. Breaking features out in their own repositories requires a significant amount of time and expertise to set up. It requires in-depth knowledge touching multiple areas such as Maven, Gradle, VPNs, and artifact management. This knowledge down payment only becomes worthwhile as the scale of the company increases.
For nearly three years we operated, scaled, and thrived with a multirepo organization. But by 2016, our multirepo setup began to reach its limits, and our developers butted up against the following new issues:
- Architecture Silos: With strong decoupling of features, architecture silos began to form. We built a uniform lint system early on that prevented stylistic silos, but that did nothing to prevent different teams from using a wide variety of patterns from Activities and Fragments, to MVC architecture and our own homegrown architecture. On some level, architectural silos are expected and even good: engineers should be able to choose an architecture that fits their use case. But as we scaled, our team began to work between libraries more and more. This in turn meant learning new architectures on a regular basis, which meant a steep and consistent learning curve. As a corollary, a specific library’s architecture implemented incorrectly could make integration with the consumer apps or other features difficult or even impossible without significant code refactoring.
- Dependency Hell: Over time, our dependency graph grew in complexity, and eventually we built out a tool to ensure that new diffs were not causing breaking changes. This solution definitely reduced the effects of dependency hell, but depending on how many libraries your change affected, even running the tool over a codebase could take frustratingly long. Moreover, fixing whatever problems we found could require days of engineering work: identifying the problem dependency, fixing the offending code, and cutting new versions of the affected repositories.
- Long Build Times: Our codebase size started to reach the limits of what Gradle could quickly build. A fresh app build could take 15+ minutes; a series of small XML tweaks could cumulatively waste hours of build time over a day of development.
So what did we do?
The answer to our problems: investing in a monorepo, a single repository containing multiple, independent projects. A monorepo exists in one codebase, just like our initial rider app. But unlike our initial rider app, this new instantiation of all code in one box contained multiple logical components that operate independently. Thus, we can now invest the time and resources into tooling and architecture necessary to fix many drawbacks of a large, single repo:
- IDE Support: Android Studio is now considered the default, lightweight Android developer platform, but it is hardly the only one. With a large monorepo, we have found that IntelliJ works very well for our current size—with a few adjustments made possible by IntelliJ being open source.
- Long Build Times: Uber recently switched from Gradle to Buck, which is a modular build system. Buck was easy to integrate with because our code was already broken out into discrete components. Our home-grown Gradle plugin OkBuck gave us a smooth transition which resulted in build times of fifteen plus minutes an app reduced to below five minutes for a fresh build, and under one minute for an incremental build.
- Broken Master: We recently introduced a system called Submit Queue which rebases changes on master and runs a customizable set of tests before merging them. This prevents engineers from pushing code that breaks the build, keeping master squeaky clean for everyone else using it.
This may seem like a lot of overhead, and it is. There is no out of the box way to create a monorepo, because there are only a handful of companies right now that have the scale requiring it. But for every pain point that we predicted back in 2013, we now have the time, expertise, and resources to build tooling to prevent them from hampering our productivity.
As you can see, it took Uber many years to get to this latest stage of development. When we were a small, scrappy team of just a few engineers, we did not have the time or resources to create Submit Queue or set up Buck. But our early foresight encouraged architectural decisions that have allowed us to scale as quickly as we have. Now that we have scaled even further, we can invest in a development down payment to ensure that the future service growth is seamless and efficient.
Developer productivity is hard but important work, and with every incremental improvement, we are not just bettering Uber but the Android community at large. If growing Uber’s Android monorepo sounds interesting to you, considering joining our mobile engineering team—we are hiring!
Software engineers JJ Ford and Gautam Korlam also contributed to this article.
Photo Header Credit: “Resting reedbucks in late afternoon” by Conor Myhrvold, South Luangwa National Park, Zambia.