We previously published a blog about how we scaled adoption of MIT Kerberos^™ at Uber. We built an automation system called KDP (Keytab Distribution Pipeline) that generates and distributes Kerberos keytabs (credentials) to systems that must authenticate with Kerberos. 

With the help of this system, we drove adoption of Kerberos authentication for several critical use cases. Some of the key use cases include:

Growth in use cases drove up the number of keytabs to over 100,000 over 5 years. Due to the volume and security requirements of the use cases it supports, we chose to enable periodic rotation for Kerberos keytabs. 

Rotating over 100,000 Kerberos keytabs presents significant challenges, primarily due to two key factors:

To address these challenges, we designed and built robust, failure-domain-aware automation that safely orchestrates keytab rotation without human intervention. In this blog, we’ll walk through the architecture and solutions that enabled us to achieve safe, scalable keytab rotation. Note that this blog requires an in-depth knowledge of Kerberos and its protocol.

Keytabs are distributed to different systems (or workloads) using Uber’s internal SMP (Secret Management Platform). With 100,000 keytabs distributed to tens of thousands of workloads, it’s impractical to rotate keytabs manually. SMP provides a pluggable framework for enabling automatic rotation of secrets at scale.

SMP keeps track of all the secrets at Uber. For every secret, it keeps track of important metadata such as the last updated time, secret type, and the secret provider that’s responsible for generating and deleting the secret. Additionally, SMP holds information about the rotation policy (N days) for every secret type. A component called the Secret Lifecycle Manager takes care of scheduling Cadence workflows calling the necessary Secret Provider as a secret gets closer to the rotation date. 

In the context of Kerberos Keytabs, there are two main components within KDP that work together to manage the life cycle of keytabs. The existing Kerberos Bridge was modified to implement the Secret Provider interface. With this, Kerberos Bridge supports the ability to generate, rotate, and delete keytabs. Kerberos Bridge receives requests from the Secret Lifecycle Manager to rotate Keytabs.  

An on-host component called Keytab Manager resides alongside Kerberos KDC (Key Distribution Center). This component is responsible for issuing commands to change passwords and generate new keytabs as requested. KDP keeps the internal state of keytabs issued for different principals and how they are consumed by different workloads. The Keytab Manager is responsible for uploading the latest keytab to the Secret Store. Workloads that need to authenticate using Keytabs are configured to fetch Keytabs periodically from the Secret Store. 

With this integration between SMP and KDP, we were able to scale out automatic rotation of keytabs without any human involvement. The next set of challenges we had to solve were around ensuring automation does the work safely without disrupting our systems.

We looked further into the Kerberos protocol to identify and solve the possibility of failures if keytabs on either the client or server side were to undergo rotation while any of the Kerberos requests were in progress. Here are cases where we found kvno mismatch during keytab rotation can cause failures:

We’ll go into details about the problem for the two possible failure cases and describe how we minimized the chances of failure.

To implement Keytab rotation safely with the above architecture, we put together a timeline chart to understand the sequence of events that would happen during rotation.

The timeline steps were: 

Between t2 and t3, the application can’t obtain a new TGT, as keytab 1 is invalidated and keytab 2 isn’t yet available to the application. If t2 < t4 < t3, the application will experience authentication errors and possible downtime. To avoid this situation, we had to:

This logic was implemented into the automation code to minimize the chance of authentication failures. In certain specific highly available applications, where authentication failure was unacceptable at all costs, we worked with the application owner to provide two different keytabs with different principals, and ensured that only one of them is rotated by our automation. This enabled the application with a fallback solution: retry authentication with a second keytab in case authentication fails with the first one.

In Uber’s environment, HDFS NameNode, YARN ResourceManager, and Presto Coordinator are examples where they act as servers in Kerberos context. HDFS clients connect with NameNode to read files, YARN clients connect with ResourceManager to submit jobs, and Presto^® clients connect with Coordinator to submit queries.

Clients obtain service tickets from the KDC before it connects with the server. The service ticket is encrypted with the server’s key. In the case of rotation, the client may have received and cached a service ticket encrypted with server kvno 1, and when the server receives the request, its keytab may contain kvno 2 due to server key rotation. This can lead to failures as the server wouldn’t be able to decrypt a service ticket with kvno 2 if it was generated with kvno 1.

To avoid this failure on the server side, we ensure that the server keytab has both previous and current kvno (illustrated by a yellow bar in Figure 4), so it can try both keys to decrypt service tickets during the transition phase.

The timeline steps were: 

To distinguish between client and server, we identified applications that showcase behavior like client and server (and sometimes both), and captured that in KDPs metadata. This enabled us to automate the necessary logic to cover both cases described above.

We took several measures to ensure the safety of automating keytab rotation. 

There were containerized applications that ran on thousands of nodes using a single Kerberos principal/keytab. If the problem suggested in the section above were to happen for this single keytab, it’s very likely that a significant part of (if not the entire) application would fail.

To prevent this scenario, we worked with the application owner to migrate from a single keytab to node-specific principals/keytabs. This increased the number of keytabs but enabled us to rotate them safely in smaller batches based on the application’s fault domains (described below). This limited amount of any possible disruption to the existing fault domains of the application.

For critical applications, we introduced a fallback keytab derived from a different principal (other than the node-level principal) that the application can fall back to in case of failures. This two keytab approach reduced the possibility of disruptions even further.

The majority of systems that depend on Kerberos are managed by Odin, a Kubernetes^®-like deployment system. Odin standardizes common concepts such as availability zones, clusters, and roles for nodes, and exposes this information and workload placement metadata in a queryable manner. 

We amended KDP to cache information about metadata, particularly around fault domains. This enabled us to build a simple rate-limiting feature. The objective of this feature is to avoid rotating keytabs for not more than N nodes of a particular role within a given application cluster. For critical control plane roles (Figure 5) which are fewer (tens) in number, we applied more restrictive rate limits and time delays between rotations. For less-critical data plane roles, which are larger in number (tens of thousands), we applied less restrictive rate limits to rotate their keytabs at scale as shown in Figure 6.

This simple feature took roughly 2 weeks to implement, but it has been the most effective feature that has been guarding the automation system from causing accidental chaos for critical applications that rely on Kerberos when their keytabs get rotated.

A technology (like HDFS, YARN, Presto) has many clusters of varying importance, characterized by its tier. When rolling out keytab rotation for a technology, we use a cluster-based allowlist to enable gradual, controlled adoption. This approach enabled us to start with the least critical test clusters and progressively expand to more important ones. By doing so, we could learn from errors early on and refine our rollout strategy and rate limits before we enabled keytab rotation for critical tier clusters.

With provisioning and rotation of keytabs automated, the last logical step was to enable automatic deletion of keytabs. 

We relied on some of the key features we built for factoring rotation to enable safe deletion of keytabs, like: 

There’s a pattern where the server uses its keytab to just decrypt the service ticket AP_REQ sent by the client. The server never logs in with Kerberos (that is, no AS_REQ from the server), so we should instead look for TGS_REQ where others request service tickets to this server.

We look for a set of signals, such as the existence of the system that consumed the keytabs, time period since last login and rate limit on the cluster, before proceeding to delete the keytab from the Secret Store and associated principal from KDC. With this, we enabled automatic deletion of keytabs.

Kerberos was originally introduced in Uber back in 2016 as we were scaling our Hadoop infrastructure to support analytics and machine learning use cases. Since then, we’ve scaled adoption of Kerberos to satisfy several critical use cases for Uber. We also built support for rotating all Kerberos keytabs automatically through integrations with the Secret Management Platform. Our metrics suggest that this system has rotated over 30,000 keytabs in a month at its peak. 

However, it’s time to explore the next major step function change for Kerberos at Uber. We’ve been busy evolving our architecture and systems to be cloud-native. We’ve also been investing in hardening and scaling Uber’s Workload Identity/PKI infrastructure. This opens up further opportunities to explore solutions such as PKINIT (introduced in RFC 4556), which can replace long lived keytabs with certificate based authentication. This brings us several benefits:

In a future blog, we’ll describe key learnings from our experience modernizing authentication for data infrastructure.

Stay up to date with the latest from Uber Engineering—follow us on LinkedIn for our newest blog posts and insights.