Shadower is a load testing tool that allows us to provide load testing as a service to any microservice at Uber.

Shadower started as a command line application that allowed us to read a local file to load test a local application. At the time, Maps PEs were heavily investing in Java GC tuning. We needed Shadower to be able to do request mirroring to make sure two different applications get about the same load and different types of loads (test multiple endpoints). In summary, we were starting the same application twice: once with the production configuration and once with the new GC tuning. How’s this different from testing on staging? There are a few differences:

Shadower stayed as a command line application for about a year until we built Ballast. Ballast required a reliable and easy-to-manage load test generator. Ballast is in charge of tracking resources utilization and deciding what is the right load for any load test. Shadower only focuses on executing those load tests.

The changes that were required to make Shadower as a service were:

Shadower uses a coordinator/worker architecture. The coordinator is in charge of querying all the load tests in the scheduled state and finding a suitable worker to run them.

The coordinator is the public-facing component that uses leader election. Any action gets asynchronously replicated to every peer in the cluster. Most of the time it uses differential updates, but whenever it finds a new peer in the cluster it must send a full update to make sure every peer has the same state of the system. There is a tag to keep track of any changes for all the relevant objects.

From any coordinator you can:

Scheduling is done asynchronously. Any coordinator can receive the requests and it initially sets the state to “scheduled,” then the master is querying every second any load test in this state and tries to execute it if there are workers with enough quota.

Any coordinator can receive the request, but it always forwards the request to the master so that the master can check if the current worker running the load test has quota available.

Any coordinator can receive the request. Then the coordinator changes the state to “stopped.” Finally, when the worker sends the next heartbeat, the coordinator finds out that the load test has been stopped and notifies the worker.

The endpoint by default returns all the current load tests, but it provides the following filters:

The worker is in charge of executing the load tests. It has the following features:

It uses heartbeats to transfer the state to the coordinator. It means that the coordinator could have a slightly different state because it is an eventually consistent system. When the coordinator has load tests to be scheduled and decides to use worker X to run it. If by the time it receives the request there are 3 possible scenarios:

We have a blob storage from where we can download previously uploaded files with all the requests.

Additionally, it has the ability to replay payloads a specific number of times, in case you want to ensure that each payload gets executed only once (e.g., stateful services).

If you provide multiple hosts, it can send the same request to every host. As we mentioned in the first section of this document, this is useful to ensure we are sending the same load to every host in the load test.

Each load test accepts a tag that the workers used to emit metrics, which is how we can differentiate between different load tests. If the metric is not provided, metric emission is disabled.

It is a command-line tool that provides encoding/decoding for all the available encodings we use at Uber (JSON, Thrift, Protocol buffer). It allows users to easily modify any raw payload that we sniffer using Ebpf on production hosts.

Example:

maps to →

Imagine that your cache gets invalidated every few days/weeks. The first option that we have is that we can wait for those requests to arrive and cache them, but that means that those initial requests are going to have a latency impact. Instead we can force Shadower to load the popular requests, so when those requests arrive we can quickly dispatch them.

Due to the complexity of Uber’s architecture, a common unified shadow library solution to forwarding traffic to staging for migrations/experiments validation has a few problems:

As reliability engineers we must make sure that each and every service has the proper rate limit. There 2 possible scenarios we want to avoid:

Automating load tests to run on every deployment (in an environment where there is no noise from production traffic) we can easily detect if there has been a significant change in memory, CPU, latency, etc.

Around 40% of our services are Java-based. For those not familiar with it, Java uses JIT compilation, which means that the initial code is slower than the code that runs after a few seconds. In order to improve this, a developer can load test for a few seconds before receiving production traffic.

The journey for Shadower is still not complete, even as it continues to benefit teams across Uber in the current form. We are continuously evolving it and adding features that improve its flexibility, scalability, and ease-of-use. Shadower is meant primarily for stateless services, but we plan to add support for stateful service in the near future. Keep an eye out for future Shadower blogs!