Introduction

Engineering teams use Uber’s stateful orchestration platform, Odin, to run their workloads across on-premises and public clouds. There are around 5 million cores on Odin, but not all cores are created equal. Some CPU cores are old, some are new. We even have both ARM and x86 hosts. This means that single-core performance may vary by up to a factor of almost 2. Therefore, the CPU usage of a workload is dependent on which specific hosts it’s running on, not just the load patterns of the workload. As we have a highly heterogenous fleet, the result is that we often overallocate workloads if they’re running on fast CPUs and in some cases under allocate them on slow hardware. The first is an efficiency concern, and the latter is a reliability concern.

This blog covers how we modeled this difference and formalized it into NCUs (normalized compute units). This allows us to rightsize workloads on a heterogeneous fleet of servers at a global scale and give similar compute power to each workload independent of differences in host performance. This work is built on top of the Odin CPU model, which has been described in a previous blog.

Server Performance

Odin runs with a mix of on-premises data centers and in the public cloud. This means that we cover many generations of different hardware. On top of this, we use x86 and ARM-based hardware. ARM is a recent introduction to the fleet. 

‌This means that single-threaded performance varies by up to almost 100% between the fastest new x86 core and the slowest old core. Even among new servers, there’s a significant difference between the performance of an ARM core and an x86 core. 

Figure 1 shows a use case running on two types of x86 servers in the same data center. The upper blue line is for the slower old-generation hardware compared to ‌the newer generation hardware (green line). The CPU utilization of the old hardware is roughly 23% above the new hardware.

Figure 1: CPU utilization for 2 server types running the same workloads.

Another effect of ‌uneven CPU load is that there might be reliability issues when workloads are moved from fast to slower hardware. Consider a use case with a clear leader-follower load pattern. This is shown in Figure 2. The CPU usage on the leader workload is more than 10 times the usage of the follower workloads. 

Figure 2: CPU usage in a cluster with 3 workloads where the leader workload is running with ‌more than 10 times the load of the follower workloads.

Ways to Adjust for Server Performance

One of the common ways to adjust for server performance is load balancing, which routes more traffic to ‌ servers with a small load and less to servers with a high load. The routing strategy can be based on static weights like performance factors or be based on dynamic data like latencies or CPU utilization. This strategy is used for stateless workloads where a request can go to any one instance of a particular service. The same isn’t the case for stateful workloads. Here, a request has to go to one of the servers that actually holds the queried data. On our stateless fleet, we have a global traffic routing layer. Such a similar global setup doesn’t exist in our stateful fleet to obtain an even CPU utilization.

On Odin, we’ve been running vertical CPU scaling for years. We use CPU sets to guarantee resources for workloads  and shared resources to perform some degree of over allocation. We call it dedicated and shared resources, respectively. The dedicated resources are integer cores and the latter is a decimal shared allocation, which translates into a shared pool on the hosts that all workloads have access to. We use CPU shares to split resources between workloads in case of contention. 

The CPU target for a cluster (a group of workloads on Odin like a data shard) is based on the CPU usage of all the nodes in the cluster. For a leader-follower case, we use the max load across the nodes in the cluster. Historically, we’ve been using base CPU usage from the nodes. However, as shown in Figure 2, this is like comparing apples to oranges. 2 cores of CPU usage on one server type might be equal to 3 cores of usage on a different server from a performance perspective.

Figure 3: Schematic drawing for the setup for calculating and applying CPU resource targets based on normalized compute units.

For workload resource updates and during workload creation, we then take this normalized target and convert it to physical cores for the specific host it is scheduled on. This results in different core allocations across workloads in the cluster but makes ‌CPU utilization more even. Now the low-utilized workloads due to fast hardware are scaled down and start to run hotter. 

Table 1: Performance factors for 4 server types running for Apache Kafka®.

Figure 4: Average CPU allocation per workload for a Kafka use case running on 4 different server types, where the blue line is for the fastest hardware.

Performance-Aware Scheduling in Action

We’ve incrementally been rolling out the system to all of the automatically scaled workloads on Odin. This is roughly 1.4 million CPU cores. Early in the rollout, it became clear that we couldn’t fully rely on ‌performance factors from synthetic benchmarks. We needed to adjust the factors to match the actual performance seen in production. We also quickly saw that the different technologies had significantly different performance numbers per server type. We have therefore set up a system to continuously monitor ‌CPU usage across server types and estimate the performance factors on a technology level. This system is running daily, and we’re seeing significantly different performance factors for Apache Cassandra® and Kafka.

As mentioned initially, the goal was two-fold. We wanted to improve ‌reliability and reduce allocations by equalizing ‌CPU utilization across server types. As shown above, for Kafka, the system changes the workload allocations significantly. This results in a change, as in the CPU utilization as shown in Figure 5. The lower dark blue line is the fastest hardware and the upper blue line is the slower old hardware.

Figure 5: CPU utilization across 5 different server types for Kafka workloads during the NCU rollout.

Figure 5 also shows that we aren’t fully there yet. However, it’ll be difficult to collapse the utilization lines fully. Part of this can be due to load variations across workloads, hot shards, and different server types mixes for the technologies. We’ll continue to monitor and tune the performance factors of the servers to obtain the most equal utilization across server types. 

Whereas the above only focuses on the efficiency improvements of this project, we also have cases where ‌ reliability has improved. There was, in particular, a case where a leader-follower-based use case saw degradations whenever the workloads weren’t placed on the fastest hardware. With the new setup, we can express minimum allocations in terms of both a normalized value as well as correcting the forecast based on the hardware. After the rollout, we haven’t had any reports with this issue.

What’s Next

Until now, Uber’s CPU allocation model has been centered around CPU cores. However, with the above changes, Odin moved up an abstraction layer and now focuses on compute power or performance instead. This is a large conceptual change from the focus on cores. Uber runs its own cloud with a large variety of server types, and where workloads are scheduled across the whole fleet to improve resource utilization and reliability (like a geographical spread of workloads in different data centers). So, we don’t want our internal customers to hand-pick particular server types. This should be abstracted away from the customer, who just needs sufficient resources to run their workload. We consider it a platform’s responsibility to ensure that workloads are given this and at the same time not more than is needed. Performance-aware scheduling makes this more efficient and reliable. 

Conclusion

The performance of different server types is very important when running stateful workloads across a diverse fleet. With the described system, we’re adjusting CPU allocations based on ‌server performance and thereby obtaining a more evenly distributed CPU utilization across different server types. This ensures a more efficient and reliable platform. 

Acknowledgments

This work was carried out in collaboration with multiple teams. In particular, we want to thank the Odin Scheduler team and the Fleet Performance team. Most of the work was carried out by former Senior Software Engineer Ulfat Abjabborov. 

Cover Photo Attribution: “Intel Celeron CPU” by Uwe Hermann Is covered by CC BY-SA 2.0.  

Apache®, Apache Kafka®, Apache Cassandra®, and the star logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.