Skip to main content
Engineering, Backend, Uber AI

Cinnamon Auto-Tuner: Adaptive Concurrency in the Wild

7 December 2023 / Global
Featured image for Cinnamon Auto-Tuner: Adaptive Concurrency in the Wild
Image
Figure 1: The relationship between max concurrent requests and throughput. At some point the service can’t handle more and it drops fast.
Image
Figure 2: Architecture diagram of Cinnamon, with the scheduler and auto-tuner part highlighted.
Image
Figure 3:  Prioritized request scheduling
Image
Image
Figure 4: The lower the limit – the more tolerance to latency increase.
Image
Figure 5: The aggregation process from individual request timings to a smoothed value.
Image
Figure 6: The ever drifting targetLatency issue in effect. When resetting targetLatency the new targetLatency is captured at a higher limit, which leads to both of them drifting up. Note that at ~14:10 the overload stopped.
Image
Figure 7A: Positive covariance between the number of inflight requests and throughput.
Image
Figure 7B: Negative covariance between the number of inflight requests and throughput.
Image
Figure 8: The effect of covariance when resetting the inflight limit. Both inflight limit and the latency samples (i.e., request timing) are now stable.
Image
Figure 9: Overloading one node in production using Ballast
Image
Figure 10A: Throughput and latency during overload
Image
Figure 10B: Inflight limit (top); the ratio between the number of inflight requests
 and the inflight limit (middle); CPU usage (bottom)
Vladimir Gavrilenko

Vladimir Gavrilenko

Vladimir Gavrilenko is a Software Engineer at Uber on the Inventory and Catalog team based in Aarhus, where he focuses on reliability and scalability.

Jakob Holdgaard Thomsen

Jakob Holdgaard Thomsen

Jakob Holdgaard Thomsen is a Principal Engineer at Uber, working out of the Aarhus office, helping to make Uber's systems more performant and more reliable.

Jesper Lindstrom Nielsen

Jesper Lindstrom Nielsen

Jesper Lindstrom Nielsen is a Staff Engineer at Uber on the Inventory and Catalog team based in Aarhus, trying to make everything run a bit faster and scale a bit more.

Timothy Smyth

Timothy Smyth

Timothy Smyth is a Staff Engineer at Uber on the Delivery Backend Platform team based in New York City, helping to increase the resilience of Uber Delivery.

Posted by Vladimir Gavrilenko, Jakob Holdgaard Thomsen, Jesper Lindstrom Nielsen, Timothy Smyth