Introducing Shard Manager

Introduction

Distributing shards evenly across hosts doesn’t guarantee an evenly distributed workload. Some shards receive substantially more traffic than others, leaving individual hosts overloaded while capacity remains available elsewhere. For engineers operating sharded services, this imbalance costs hosts, because a cluster is sized for its hottest host, and it complicates capacity planning and troubleshooting.

Shard Manager is a general-purpose, open-source sharding service available in the cadence-workflow GitHub organization. It separates shard ownership decisions from application logic and provides load-aware placement, client libraries, and operational tools. Although Cadence is its first production application at Uber, the integration model supports any sharded service. The first onboarded service shows a 38% reduction in CPU usage on its hottest host. 

This post explains the design, its integration with Cadence, and the production results at Uber. Engineers facing uneven workloads can use these lessons to evaluate whether explicit shard placement would help their own systems.

Background 

Cadence previously relied on the Go implementation of Ringpop, a consistent-hashing library, to distribute shards. Each participating host independently derived ownership from its view of cluster membership.

Figure 1: Hash ring solutions like Ringpop can lead to skewed shard distribution, since they don’t account for shard load.

This approach had three limitations. First, distributing shard counts didn’t account for differences in shard load. Second, inconsistent membership views made ownership difficult to diagnose because there was no authoritative assignment record. Third, operators couldn’t directly steer placement: moving work required changing cluster membership.

In Cadence’s history and matching services, uneven shard placement created hot hosts. Without autoscalers, operators had to continually add and remove hosts to redistribute shards and relieve overloaded hosts. With autoscalers, the system kept adding hosts until all hosts fell below the target utilization, leaving much of the fleet underutilized. For a system processing billions of requests every day, this meant either constant operational toil or substantial wasted capacity.

These limitations motivated a service that makes ownership an explicit, inspectable decision. The resulting implementation uses Go, gRPC™, and etcd®, with application clients that report load and consume assignments.

The central design consideration was how much responsibility to put in the shared service. Applications retain responsibility for processing their shards and defining meaningful load measurements; Shard Manager handles placement and ownership coordination. This division allows different applications to use the same infrastructure without adopting Cadence-specific processing logic. Application requests never pass through Shard Manager, as Shard Manager only orchestrates who owns what.

Alternatives

Our design drew inspiration from Facebook® Shard Manager, Google® Slicer, Azure® Service Fabric, and Apache Helix™. These systems provided useful precedents for separating shard management from application processing. Google Slicer, for example, describes a control plane that makes placement decisions outside the request path, balancing workload while limiting movement.

We considered Apache Helix, but its primary Java interface made it hard to integrate with Go services. Uber developed a partial go-helix client, which was archived in 2020. Helix clients also communicate directly with Apache ZooKeeper™ and depend on its Helix-specific data layout. Adopting this architecture would have introduced operational overhead, including provisioning ZooKeeper clusters. We chose a service interface that keeps storage details separate from application clients, with plans to make etcd optional and remove an infrastructure dependency.

We also considered embedding the placement controller in the sharded services themselves. This would avoid operating a separate service, but would distribute management state across applications and add coordination responsibilities to host services. A standalone service provides a clearer separation of responsibilities and a shared foundation for other applications.

Key Design Considerations

Several requirements guided the architecture:

• Ownership and recovery: Minimize periods with missing or conflicting owners, and retain assignment state across controller failures.
• Discovery: Let callers determine which host owns a shard.
• Different shard lifecycles: Support both predefined shard sets and dynamically created shards.
• Application integration: Keep processing logic in the application while exposing shard management through a service interface.
• Open source adoption: Make the service usable without dependencies on proprietary infrastructure.
• Separation of data and control planes: Keep Shard Manager out of the primary request path for reliability and low latency. 

These considerations led to a standalone service with Go clients and gRPC interfaces. Applications report shard load and process assigned work, while Shard Manager coordinates ownership and placement.

Architecture

The architecture has five main components:

• Shard Manager service: Determines ownership and runs rebalancing, with one elected leader per namespace.
• Spectators: Applications that need to know who owns which shard and how to route to the owners. 
• Executors: Applications that process assigned shards and report shard loads for the shards they own.
• etcd: Persists assignment state and supports leader election.
• smctl: CLI tool that provides inspection and operational commands.

Namespaces separate configuration and balancing decisions for different sets of shards.

Figure 2: Shard Manager assigns shards to Cadence Matching service (executor) hosts.

Figure 3: Cadence Frontend service (spectator) receives the shard map.

Figure 4: Cadence Frontend service (spectator) routes sharded requests to the right Matching service (executor) host.

In Cadence, Matching hosts process task-list shards, while Frontend uses the spectator client to locate their owners. A service can also perform both roles when it needs to process local shards and route requests to peers.

The key architectural distinction from the previous hash-ring approach is explicit ownership combined with application-reported load. Placement can change in response to workload conditions without requiring a membership change.

UI/Design

Shard Manager’s operational interface is the smctl command-line tool. Operators can inspect a shard’s owner, list executors, drain a host, and drain or restore individual shards.

For example:

Figure 5: Example Shard Manager CLI commands showing shard inspections and executor drains.

For application developers, integration happens through Go clients. The application supplies a shard processor with lifecycle methods and a load report. The executor client manages heartbeats and assignment updates, while the spectator client provides ownership lookups.

Development Challenges

A key engineering tradeoff is balancing responsiveness against unnecessary shard movement. Reacting to every short-lived spike would cause repeated reassignments, while reacting too slowly would prolong overload.

The greedy balancing algorithm addresses this through smoothed load measurements, upper and lower thresholds, a movement budget, and a cooldown after ownership changes. It considers moves that improve balance and selects a shard that brings source and destination loads closer together.

For example, suppose one host has load 160, comprising shards with loads 100 and 60, while another host has load 40. Moving the 60-load shard gives both hosts load 100. Moving the largest shard would produce a less balanced result.

These controls deliberately trade some speed of convergence for fewer moves. Their effectiveness also depends on applications reporting load measurements that reflect actual work.

Coordinating the Migration Between Ownership Systems

Migrating an existing service to Shard Manager requires coordinating both shard ownership and request routing. During the transition, all participating services must agree on which ownership system applies to a given shard. If some instances use the existing system while others use Shard Manager, conflicting ownership and routing decisions can stall shard processing.

To reduce propagation delays, we moved Cadence’s migration control into its database-backed configuration store. This gave instances a shared source of truth and shortened the window in which they could observe different migration settings.

Use Cases at Uber

Cadence’s Matching service now uses Shard Manager throughout Uber. Matching routes workflow and activity tasks to workers, with task lists serving as its shards.

In a production cluster presented in the original blog post, CPU utilization on the hottest Matching host fell by 38%. The hottest-to-average host utilization ratio decreased on average from approximately 2.3× to 1.4× on average. Because cluster sizing depends on the hottest host, the improved distribution allows the same traffic to run on about 40% fewer hosts.

Placement quality was also compared with a constraint solver. Across 24 workload snapshots, Shard Manager’s load spread was within 5% of the solver’s optimal result in 23 snapshots and within 1% in 14.

These results demonstrate improved capacity utilization. Operationally, explicit ownership and targeted draining also give engineers more direct control during investigation and maintenance.

Figure 6: Shard Manager rebalances shards to keep host loads within an acceptance band around the average.

Next Steps

The next planned integration is Cadence History, which manages workflow state and execution. Moving its shards is more complex than moving Matching task lists, but the team expects opportunities for further utilization improvements.

Another planned change is making etcd optional, enabling Shard Manager to run entirely in memory. The project also welcomes integrations beyond Cadence.

Together, these directions extend the same engineering objective: use infrastructure capacity more efficiently while making sharded services easier to operate.

The Shard Manager roadmap will be published and tracked in Shard Manager’s Projects Page. We welcome community input: open an issue to ask questions, request features, or share feedback.

Conclusion

Shard Manager makes shard placement an explicit operational capability. It combines application-reported load, centralized ownership, and direct operator controls to address limitations of count-based distribution.

Cadence’s production results at Uber demonstrate the value of this approach. As an open-source, general-purpose service, Shard Manager also gives other engineering teams a foundation to evaluate and adopt for their own workloads.

Explore the source code and service integration guide to get started.

For a deeper look at load balancing and production results, read Introducing Shard Manager on the Cadence blog. 

Acknowledgments

We thank Seva Kaloshin, Eleonora Di Gregorio, Ilya Ojereliev and Taylan Isikdemir for their contributions to Shard Manager’s design and development.

Andreas Holt and Theis Mathiassen collaborated with the team on their master’s thesis, Extending the Load Balancing for Cadence, which analyzes the load-balancing algorithm, evaluates its placement quality against a near-optimal solution, and explores potential extensions.

We also thank our leadership for their guidance and support, and our partner teams across Cadence for their reviews and feedback throughout the project.

Cover Photo Attribution: Owned by Uber. 

Apache Helix, Apache Zookeeper, and the star logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. The use of these marks implies no endorsement by The Apache Software Foundation.

Azure® is a registered trademark of Microsoft, Inc. 

etcd® and gRPC are registered trademarks of The Linux Foundation® in the United States and other countries. No endorsement by The Linux Foundation is implied by the use of these marks.

Facebook® is a registered trademark of Meta Platforms, Inc.

Google® is a trademark of Google LLC and this blog post is not endorsed by or affiliated with Google in any way.