Introduction

For years, scaling stateless workloads at Uber relied on a single orchestrator model. This changed when the new failover orchestration architecture was introduced in 2023. The new architecture sought to utilize expensive, idle capacity. This economic challenge evolved into a complex control-plane problem, designing a system where multiple orchestrators could safely control the scale of the same Kubernetes® workload. In this blog, we detail our solution, the introduction of a new controller, and the critical production lessons learned about Kubernetes status, stale caches, and multi-writer control planes.

About the Container Platform Team

The Container Platform team manages over 100 compute clusters across data centers and cloud providers like Oracle® and Google®. With roughly 4,000 services running on 3 million cores, these clusters handle 1.5 million daily pod launches. They power Up, Uber’s service federation layer for microservice management.

Figure 1: The Kubernetes platform at Uber.

One Horizontal Scaling Path for Stateless Workloads 

The starting point for a stateless workload at Uber is Up. Up is our internal platform that works as a federation layer for a large Kubernetes fleet spanning multiple zones, regions, and hardware environments. Service owners use Up to deploy builds, define failure-domain requirements, and set scaling expectations for their services. 

On the Kubernetes side, we have our in-house CRD, which we call UberDeployment. Our team published a blog about the history of that CRD. In short, that object has a few more fields than just an image and a replica count. It has many platform-specific features materialized, such as business requirements, rollout expectations, and resiliency constraints. A dedicated controller, the UDC (Uber Deployment Controller), reconciles that intent into Kubernetes primitives and reports status back to Up.

That status is an important member of the CRD, because Up is more than a front door for deployments. It also manages life cycle workflows, including continuous deployment. Once UDC reports that a deployment is healthy, Up can move on to the next state in its workflow. For years, this single-controller model kept scaling behavior simple: one controller managed the replica count (Up) and another controller applied the replica count and reported the status (UDC).

The Failover Changes Ways

The initiative to evolve regional failovers changed the state of the world for the Compute team.

In normal operational mode, Uber operates on top of ‌active-active data centers across different regions. But when an outage occurs (if a region becomes partially or fully unavailable, unstable, or degraded), traffic may be rerouted to another region (this is called a failover). To make that possible, the surviving region needs enough idle compute capacity to handle the increased load.

Historically, the simplest way to guarantee safety was to keep some reserved idle capacity in all data centers. It worked, but economically it meant carrying around a large set of underutilized resources.

We wanted a better solution, so we decided to change the schema for how failover works. The decision was to reuse capacity from low-tier workloads to allocate high-tier workloads. The mental process was simple: let’s scale down the low-tier workloads and scale up the high-tier workloads.

But the moment we did that, we got a new source of the scaling intent. Up and UDC still needed to own the normal desired state of services. A failover orchestrator now needed the ability to influence scaling decisions as well. And because we expected additional orchestrators over time, we needed a solution that could generalize beyond this single use case.

Why Didn’t We Extend the UberDeploymentController?

Once we introduced a new source of scaling intent, the first question was where this logic should live. The most straightforward option was to extend the existing UberDeployment Controller. UDC was already responsible for applying the replica count and reporting deployment status back to Up, so adding failover-related scaling logic there seemed natural from an implementation standpoint.

However, that option came with important trade-offs. UDC had already been sitting on the hot path for the service’s life cycle at Uber. The controller handled deploys, scaling changes, and other day-to-day operations. Failover, on the other hand, is a rare event by design. Adding failover-specific behavior to UDC would have increased the complexity of the controller that already powered the most common and most critical workflows. A regression in failover handling wouldn’t stay isolated to failover. It could affect normal deployments across the fleet.

The concern wasn’t only about the failover use case. We also expected more scaling orchestrators to appear over time. For example, future integrations with horizontal pod autoscaling or hybrid autoscaling could introduce another actor that needs to influence replica count. If every new scaling use case were added directly to UDC, the controller would quickly become even more complex, which would increase the solution’s blast radius.

The other option was to introduce a separate scaling controller. That approach also had a cost: it meant adding a new controller, a new CRD, and a multi-writer interaction with the underlying Kubernetes workload. But it gave us a cleaner ownership model. UDC could continue to manage the normal service life cycle path, while additional orchestrators could express their scaling intent through a dedicated abstraction.

We chose the second option. The goal was to implement failover-related scaling without adding more complexity to the already complex UDC path, while also creating a model that could support future scaling controllers. That decision shaped the rest of the project and led to the introduction of ServiceScale CRD and the Service Scale Controller.

Introducing the ServiceScale Controller

To support multiple orchestrators cleanly, we introduced a new CRD called ServiceScale and a new SSC (Service Scale Controller).

The goal was straightforward: give each orchestrator a way to express its own scaling desire, then reconcile that intent into the underlying workload using a new controller without overloading the original deployment controller. 

Conceptually, ServiceScale sits on the level of UberDeployment. The main task of the controller is to make scale decisions explicitly. One orchestrator can write its steady-state desired scale. Another can write failover-related scale adjustments. SSC then reconciles the combined intent into Kubernetes objects.

Figure 2: ServiceScale CRD specification.

We deliberately kept the model simple. We didn’t want an additional external database, a separate coordination service, or a control plane that’d become harder to debug under incident pressure. By materializing scale intent directly in Kubernetes, we got two important benefits.

Due to the materialization of the properties, the system became much easier to inspect. When something looked wrong, we could check ServiceScale to see which orchestrator wanted what. Second, failback became easier. Because UDC info and temporary failover info were both saved in the CRD spec, recovery didn’t require reconstructing state from logs or replaying a long sequence of API calls. Reconciliation could simply converge back to the intended steady state.

Figure 3: ServiceScale Controller architecture.

Lesson 1: Terminal Status Fields Don’t Play Well with Kubernetes

Like most Kubernetes controllers, ours consume resources through informer caches. Those caches can lag reality by a few seconds. Normally, that’s acceptable because controllers are built around eventual consistency and continuous reconciliation. But Up treated status.workflow_status as a terminal input, where a success signal from UDC could trigger the next irreversible step in a workflow. This made the status field a critical gate rather than a best-effort observation, rendering stale reads from our informer caches expensive.

To address this without removing the existing contract, We implemented a read-your-own-write consistency guardrail. When a controller updates downstream resources, it attaches its current generation as an annotation. Before reporting status, the controller verifies that its cached data reflects at least that generation, thus achieving local consistency and preventing stale status emission.

Figure 4: Read-your-own-write communication between UDC and SSC.

Lesson 2: The Challenge of Multi-Writer Systems

When we introduced the SSC (ServiceScale Controller), our UDC (Uber Deployment Controller) and the SSC began simultaneously updating the same Kubernetes resource. Kubernetes allows this multi-writer interaction and we used standard protections (like optimistic concurrency control and server-side apply patches). However, real world production timing exposed failure modes that were difficult to anticipate.

The Problem: Inconsistent Workload State

The core issue was a combination of near-simultaneous updates (writes) and delayed information (stale reads from the informer cache). Under specific timing conditions, this combination caused the ReplicaSet, the underlying object managing replicas, to become inconsistent. Specifically, its metadata (annotations) drifted from its configuration (spec). This inconsistency, which was later identified as a bug in the proportional scaling logic of the upstream Kubernetes deployment controller, had serious operational consequences:

• It broke the proportional scaling behavior we needed for performing zero downtime service upgrades (rolling updates)
• It sometimes caused workloads to become completely stuck until manually corrected

Our Layered Response

We immediately added fleet-wide observability to reliably detect the metadata-spec drift. We then built an automated “healer”, a background periodic routine in UDC that scans for metadata-spec drift and patches affected replicasets.

In parallel, we actively pursued and implemented the underlying, long-term fix in the scaling path itself.

This sequence, detect, contain, and heal immediately, while pursuing the root cause fix long-term was necessary for operating a control plane of this size.

Lesson 3:  Rollout and Testing

The rollout was a year-long process designed to be invisible to service owners. We invested heavily in integration testing using the kind (Kubernetes IN Docker®) framework to simulate real controller interactions and catch race conditions early.

Our new scaling path had to support diverse implementations, including native Kubernetes Deployments and OpenKruise™ CloneSets. Testing across these varied service life cycles was critical for stability.

By using staging environments and canary deployments, we validated the system incrementally. This conservative approach ensured the fundamental scaling control plane was updated without any customer-impacting outages.

Conclusion

On paper, this looks like a pretty simple change. We went from one scaling controller to two. In reality, it was anything but simple.

By introducing ServiceScale as a dedicated controller, we separated ownership of scaling intent from execution, allowing multiple orchestrators to coexist without overloading the primary deployment path. That decision helped us keep reliability in the hot path while unlocking more efficient use of failover capacity.

At the same time, the real work wasn’t about the CRD design. It was in going through the realities of distributed systems: eventual consistency, stale caches, and concurrent writers. Addressing those challenges required new guardrails, observability, and a strong investment in testing and rollout discipline.

The takeaway is clear: multi-orchestrator systems aren’t hard because of the APIs. They’re hard because of everything that happens between writes. In practice, real-world production conditions tend to introduce problems that differ from those anticipated during the design phase.

Acknowledgments

Cover Photo Attribution: The “containership (1)” by NOAA’s National Ocean Service is a public domain work.

CloneSet and kind are names of open source projects, features, or tools and may be trademarks of their respective owners.

Docker and the Docker logo are registered trademarks of Docker, Inc. in the United States and other countries.

Google Cloud™ and GCP™ are trademarks of Google LLC.

Kubernetes®, K8s®, and OpenKruise™ are trademarks or registered trademarks of The Linux Foundation in the United States and/or other countries. 

Oracle® and Oracle Cloud are trademarks or registered trademarks of Oracle and/or its affiliates. 

No endorsement by The Linux Foundation, Oracle, Google LLC, Docker, Cloud Native Computing Foundation, OpenKruise, or any other third party is implied by the use of these marks.

Amazon Web Services, AWS, and the Powered by AWS logo are trademarks of Amazon.com, Inc. or its affiliates.