At Uber, scale and reliability define our infrastructure. Every new server type, kernel upgrade, and configuration change must be rigorously vetted before it touches production. Historically, this qualification process was manual and time-consuming, forcing engineers to stitch together ad hoc benchmarks with limited ability to measure efficiency or ROI. This created delays, added risk, and slowed the adoption of new technologies.

To close this gap, we built Ceilometer—an adaptive benchmarking framework that delivers fast, production-like signals on system performance. Ceilometer empowers us to qualify new cloud SKUs, validate infrastructure changes, and evaluate efficiency initiatives with confidence. It’s also central to testing next-generation skus and new machine learning frameworks before large-scale rollout. ​​Beyond raw performance, Ceilometer ensures reliability by testing system stability under prolonged usage and stress. From catching Redis^® regressions to validating database applications, and benchmarking stateless microservices, Ceilometer has become an essential part of Uber’s engineering life cycle. It ensures that our infrastructure remains performant, reliable, and cost-efficient as we scale into the future.

Ceilometer is our platform for performance and reliability testing, built to run synthetic benchmarks as well as simulate and benchmark production workloads across stateful and stateless services. It automates benchmarking, collects detailed performance metrics, and provides actionable insights on resource utilization, cost efficiency, and service reliability. By integrating with existing internal tools to automate the bring-up of benchmark clusters and deploy Uber services on target hosts, Ceilometer enables engineers to launch tests, visualize results, and quickly identify performance differences or reliability risks. This approach empowers teams to make faster, more informed decisions, optimize resource allocation, and reduce operational overhead—all while lowering the risk of production incidents.

Benchmarking is a critical aspect of understanding and improving technology performance, yet it often comes with a unique set of challenges. Before the introduction of Ceilometer, performance engineers faced significant hurdles that made effective benchmarking difficult, like:   

Ceilometer solved these problems by providing:

Ceilometer is engineered for robustness and scalability, designed to handle the rigorous demands of benchmarking at Uber’s scale. At its core, the framework is built on a distributed system architecture, allowing for parallel execution of diverse benchmarking tasks across various environments.

Key architectural components and data flow include:

Ceilometer’s flexibility extends to its ability to operate across a variety of testing frameworks, providing tailored solutions for different workload types and addressing the unique challenges each presents.

Ceilometer supports a wide array of synthetic benchmarks, which are designed to stress and measure the maximum performance levels of system components and operation types in a reproducible manner. These benchmarks encompass industry-standard tools, such as SpecCPU2017, SPECjbb2015, NetPerf and FIO among others, and are customized to be representative of Uber’s workload requirements. They provide crucial early signals regarding potential performance bottlenecks, allowing for proactive optimization and informed procurement decisions.

Ceilometer provides specialized and representative benchmarking for Uber’s stateful database technologies. These benchmarks are orchestrated by leveraging Odin (Uber’s Stateful Platform), a robust and scalable platform specifically designed for managing and optimizing stateful applications. The benchmarks are not merely for performance measurement but are intricately designed to rigorously evaluate a multitude of critical aspects, including read/write performance under varying loads, transaction throughput to assess the system’s capacity, and data consistency across replicated systems to ensure reliability. Furthermore, these evaluations are conducted under a wide array of configurable load conditions, allowing for the simulation of real-world scenarios and peak demand events. This ensures the stability and efficiency of Uber’s data backbone, a critical component of its infrastructure.

For stateless services, which form a significant part of Uber’s microservices architecture, Ceilometer provides robust testing environments to assess their performance, latency, and scalability. It leverages Ballast’s adaptive load test framework to mimic production traffic and access patterns, robustly assessing and benchmarking service performance. This helps ensure that Uber’s vast array of services can handle fluctuating demands efficiently and reliably. The ability to rigorously test these services is crucial for supporting rapid development cycles and ensuring the seamless operation of our global platform.

Ceilometer drives critical decisions and upholds high performance and reliability standards at Uber. Here’s a deeper dive into two of its most common use cases at Uber.

The process for qualifying new server types involves a multi-stage approach, especially when dealing with server types still under development by vendors.

Vendors execute Ceilometer’s synthetic test suite in their own environments. The results guide their design and optimization efforts, and the benchmark results are also reported to Uber. Uber’s performance engineers then analyze reported results to project performance for Uber workloads, aiding in early decisions regarding server selection and targeted optimization path.

Once the server type becomes available for testing in Uber’s environment, a more thorough suite of Ceilometer’s synthetic and application benchmarks (tailored for Uber’s stateless or stateful use cases) is executed. These results are then used to guide the selection of the most cost-effective next-generation servers by projecting and comparing results against existing production servers. They’re also used to improve the accuracy of Ceilometer’s benchmarks by updating or fine-tuning configurations where necessary.

Evaluating any infrastructure change to existing hardware is a process driven by executing targeted Ceilometer benchmark suites on the new configuration hardware. This typically involves collaboration with service owners and issue identification and resolution. 

Effective partnerships with Uber service owners is essential, as each service is unique. This ensures that the new configuration’s results accurately reflect the service’s performance and are approved by the respective owners.

In cases where performance degradations or unexpected behaviors are observed, Uber’s performance engineers leverage Ceilometer as a powerful diagnostic tool. They work collaboratively with service owners to:

This capability has been crucial for debugging and preventing regressions in Uber’s production systems.

Ceilometer has recently expanded its use case to validating external “touchless” upgrades from cloud vendors in Uber’s canary environments, ensuring high reliability of Uber’s infrastructure.

As Uber’s infrastructure continues its rapid evolution, so too does Ceilometer. Our commitment to ensuring optimal system performance and reliability drives continuous innovation within the framework. We have several exciting areas of focus for future development:

Designing an extensible and evolving benchmarking platform requires careful consideration of several key factors. First, a flexible and adaptable architecture is crucial to seamlessly integrate with new technologies and testing environments as infrastructure evolves. This modularity prevents significant rework and ensures the platform remains relevant. Second, robust data processing and analytics capabilities are essential to handle the sheer volume of benchmark data generated at scale. This includes efficient pipelines for ingestion, normalization, and analysis, along with accessible tools for engineers to derive meaningful insights. Finally, integrating AI/ML techniques wherever possible to automate analysis, predict performance, and identify performance anomalies ensures that the platform remains at the forefront of performance and reliability engineering while being scalable to evolving needs.

Ceilometer has successfully met Uber’s adaptive benchmarking needs, showcasing its capacity to adapt to these critical considerations. We’re excited for the future of Ceilometer as it continues to evolve and serve the cutting-edge benchmarking requirements at Uber.

Stay up to date with the latest from Uber Engineering—follow us on LinkedIn for our newest blog posts and insights.