ZFR (zone failure resilience) is a critical aspect of modern distributed systems, especially for real-time analytics platforms like Apache Pinot^™ that power many Tier-0 use cases at Uber. As part of our regional resilience initiative, ensuring Pinot can withstand zone failures without impacting queries or ingestion is paramount. This blog details how we’ve achieved zone failure resilience in Pinot by leveraging its instance assignment capabilities, integrating with Uber’s in-house isolation group concept, and consequently accelerating our release processes.

Initially, our Pinot clusters at Uber relied on two key strategies: tag-based instance assignment, which groups servers by tenant to ensure logical isolation, and balanced segment assignment, which spreads data segments evenly across those servers. While effective for distributing data evenly across servers within a tenant, this approach didn’t inherently guarantee distribution across different physical zones. If all instances assigned to a table, or all replicas of a segment, happened to be in a single zone, a failure in that zone would lead to significant service disruption.

To overcome this, we adopted a more sophisticated approach: pool-based instance assignment in conjunction with replica-group segment assignment.

Pool-based instance assignment allows us to organize servers into distinct pools. This strategy is primarily designed to accelerate no-downtime rolling restarts for large shared clusters by ensuring different replica groups are assigned to different server pools.

When combined with replica-group segment assignment, replicas of data segments are distributed across these defined pools (which correspond to zones). This means that even if one pool of servers experiences a failure, the serving of replicas remains unaffected because other replica groups in different pools are still online. For example, if a table has three replicas and two are down, the remaining replica can continue to handle write and read traffic.

The key to making our Pinot deployments truly zone-resilient lies in integrating with Uber’s in-house infrastructure concept: the virtual resource isolation group. We leverage this concept by using the isolation group ID as the pool number for each Pinot server instance.

An isolation group provides an abstraction that makes zone or rack selection transparent to applications deployed on Odin, Uber’s container orchestration platform. Crucially, a node and its replacement node are guaranteed to always belong to the same isolation group, even if the underlying physical host changes. In an ideal scenario where zone isolation is prioritized for isolation groups, multiple groups wouldn’t share a single zone. Therefore, distributing segment replicas across as many different isolation groups as possible inherently means distributing them across as many different zones as possible.

For Pinot to be zone failure resilient, we ensure that the number of isolation groups configured is greater than or equal to the table’s replication factor (that is, the number of replica groups). For most of our clusters, this means configuring at least two isolation groups to match the default replication factor of 2. This setup ensures that even if one zone goes down, we maintain another available isolation group, allowing the cluster to continue operating.

Each Node on Odin (akin to a Kubernetes^® pod) has a local worker container, periodically running a job so as to ensure the database and other sidecar containers converge to the expected GoalState. Aside from the GoalState, the worker fetches placement information, including the isolation_group, which is determined by the centralized Uber Odin Scheduler.  When a node is added onto the host by the host agent, the worker container attempts to start the database container. If the pinot-server database container is successfully spawned, the associated helix instance ID is already registered in Apache Zookeeper^™. The worker sends a POST HTTP request to the pinot-controller with the query parameter of the instance ID and the payload including the pool number. 

We use two replicas instead of three because we have two identical regional clusters. If a zone fails, a regional cluster has only one copy of the data, which can lead to slower performance. We’ll discuss our failure recovery plan in a future blog post. 

An internal drill proved that the other replica can continue serving the incoming traffic without query degradation when the single zone is completely down. As shown in Figure 3, when Isolation Group 0 is down, traffic routes to the other good replica-group in Isolation Group 1.

At Uber, we manage 400+ Tier 0-1 Pinot clusters and thousands of tables. Migrating them to a zone-failure-resilient setup without causing any degradation was a significant challenge. The migration execution occurred in these steps:

Pinot table configuration backfill is managed by the internal metadata service, which oversees the entire life cycle of Pinot tables. To facilitate migrating Pinot tables to ZFR configurations, we developed a suite of granular migration APIs, enabling migrations at various levels: per table, per server tenant, or per region. These APIs include auditing capabilities and allow for rollbacks if issues arise. After table configurations are migrated, a table rebalance is triggered to redistribute data tenant by tenant. 

Our strategy focuses on rebalancing necessary tables only: upsert tables and those with retention longer than six months. For other tables, new data assignments naturally reflect the distribution as old segments are automatically purged after six months. This approach, combined with our internal regional traffic failover strategy as a mitigation tool, allows us to minimize data movement across servers and avoid causing performance degradation on live traffic.

Beyond just fault resilience, the adoption of isolation groups has a direct and positive impact on our operational efficiency, specifically in accelerating release and restart processes. The new isolation-group-based claim policy and release pipeline allows for more aggressive, yet safe, parallel operations.

Previously, we’d only restart one node at a time to minimize risk. With the new policy, we can work on multiple nodes at the same time, as long as they’re in the same isolation group. This has significantly sped up our rollout process. For example, in a test with a 20 node cluster with medium disk size, increasing the number of parallel operations from 1 to 5 reduced the rollout time from 255 minutes to 88 minutes. When it comes to production environments, the rollout time is reduced from roughly a week to 1 day for the largest fleets that are created for the Uber internal logging use case. Figure 4 demonstrates the rollout parallelism with isolation-group-based claim policy. 

The rate-limiting mechanism can be conceptualized as a distributed semaphore. Each node operation goes through Uber’s centralized Odin Accouter service. We perform the following hierarchical rate-limiting check:

In addition to rate limiting, the Odin Accounter service performs health checks on each technology. To prevent widespread query failures, we reject any node claims if we detect that a node in a neighboring isolation group is unhealthy. This proactive measure prevents a cascading failure that could take down the rest of the online replicas.

The default Odin release pipeline is unaware of the isolation group and randomly picks nodes into the release waiting queue when some nodes are rolling out. Because of this, while the rate-limiting policy speeds things up, the order in which isolation groups were rolled out was unpredictable and could be interleaved. This could cause a bad software update to affect all isolation groups and all data replicas. It could also lead to a starvation problem where some nodes wait longer than necessary, slowing down the overall release.

To fix this, we created a release pipeline based on isolation groups. In this new pipeline, each phase of the release focuses on nodes from a single isolation group, and the isolation groups are processed one after another. We also added a baking time after completing the rollout of the first isolation group to check for any performance issues.

Building zone failure resilience into Pinot has transformed how we operate our real-time analytics platform at Uber. By marrying Pinot’s pool-based instance assignment with Uber’s isolation groups, we’ve created a deployment model that naturally spreads data across zones, keeping queries and ingestion flowing even when a zone goes dark.

Rolling this out to hundreds of clusters was no small feat. Automated configuration management, granular migration APIs, and selective rebalancing let us strengthen resilience without disrupting live workloads—a balance that was essential for Tier-0 systems.

And the benefits don’t stop at reliability. Isolation group—based policies have also sped up our release cycles, enabling safe parallel rollouts and more predictable deployments, all while safeguarding performance.

With this foundation in place, we’re now turning our attention to the next challenge—failure recovery. In our next blog, we’ll dive into the strategies we use to recover from different failure scenarios, and how we keep Pinot running smoothly even in the most adverse conditions.

We’d like to thank Xin Gao, former Uber Staff Engineer, for implementing pool registration. We also acknowledge the Uber Odin Platform, which provided many out-of-the-box features that supported this project.