At Uber, we run over 5,000 microservices, 5,000 databases, and over 500,000 analytical jobs per day to support millions of people worldwide using our apps. Over 150,000 secrets facilitate authentication among these large, distributed ecosystems with multiple stakeholders. This also includes over 400 third-party vendor integrations and 400 SaaS applications.

Cyberattacks continue to rise industry-wide, with exposed credentials being a leading cause. With this in mind, we set out on a mission to build a centralized, automated, and scalable solution for secrets management. To combat secrets sprawl, shadow IT vaults, and insecure secret sharing, and to facilitate the rotation of secrets when necessary, we kicked off a transformative journey building a Secret Management Platform. We assembled a set of thought leaders within the company to define Uber’s Secret Management Standard and a visionary solution (Figure 1). In this article, we’ll share how we built Uber’s Secret Management Platform, solving key challenges and setting a new standard for secrets management.

Secrets scattered across code, configs, and other unencrypted systems can make it easier for malicious actors to find and exploit them. We deployed both preventive and remediation strategies to address potential secret sprawl. 

Developers make code changes as part of their daily work. To prevent secrets from getting into codebases, we introduced a CLI tool that runs as the pre-commit hook in Git^®. This tool blocks commits containing secrets, stopping leaks at the earliest stage of the development process—before they reach the repository. By catching secret exposures upfront, we reduce the need for incident response, minimizing cases where developers must revoke and rotate leaked secrets.  

Our remediation strategy involves both real-time and scheduled scanning methods.

In either of the above cases, if a secret is detected, the automation system identifies the owner and generates a ticket with appropriate priority, notifying them of the issue. The owner is required to revoke the secret and remove it from any exposed locations. Once the remediation actions are completed, the ticket is closed, ensuring the risk has been addressed. We continue to add new targets to scan as we scale the system. With this system in place, we detect and remediate secrets that developers accidentally leak outside the secret vault. We’re currently working on enabling a security copilot within IDEs to notify developers as soon as a possible secret is added to code, in the early stages of the development cycle.

Uber’s Secret Management Standard describes several aspects of managing secrets, covering areas such as secret storage, rotation, and response to leaks. The secret management standard dictates that all secrets must be stored in approved secret vaults managed by a single team—the Secrets team. Two big areas of work resulted from this. The Secrets team has been operating HashiCorp^® Vault on-prem. However, the team didn’t operate secret vaults in the cloud. The team set up a secret vault per cloud vendor and another within the corporate network, with appropriate security controls, and worked on deprecating vaults that other teams operated. 

The other area of work was migrating secrets from custom secret vaults and databases into one of the existing secret vaults managed by the Secrets team.  This enabled one team to govern and ensure security baselines are met uniformly across all secret vaults, and offload the overhead on other teams with managing their own secret vaults. 

As we embarked on this effort, we came across challenges with migration and broad secrets distribution. The scope of the migration involved secrets used by workloads developed by hundreds of engineers across different teams. We implemented various integrations (such as using Kubernetes^® External Secrets Operator) and supported seamless migration of secrets from the existing custom secret vaults to the centrally managed secret vaults. This ensured that no code changes were required by teams outside the central team conducting the migration.  

We also found that secrets that are intended for certain data pipelines owned by one team are distributed to and accessible by all pipelines launched by Piper (a data workflow orchestration system). A similar pattern existed within other parts of the Data Platform and CICD infrastructure (Buildkite). We introduced monitoring of secret access within containers launched by these pipelines, to identify and attribute secrets to specific pipelines that accessed them. This enabled us to scope the secrets appropriately to specific pipelines/workloads, thereby ensuring that secrets are only distributed to pipelines that require it. We conducted some simulations before. After enabling the feature, we noticed up to a 90% reduction in secrets distributed to workloads (Figure 2).

With this initiative, we converged 25 secret vaults across multiple clouds owned by different teams across Uber into 6 managed by a single team. Managing 6 vaults made it possible to be operated by a single team and at the same time limit the blast radius to a specific deployment environment, in case one of the vaults was involved in a reliability or a security incident.

Consolidating secrets vaults into the ownership of one team enabled us to standardize and govern all secrets used within Uber. This laid the foundation for building a central management and governance layer across multiple vaults—the Secret Management Platform. As we consolidated the secret vaults, we realized that there were common patterns of issues we had to solve:

We developed a metadata model (shown in Figure 3) that became the cornerstone of our Secret Management Platform. 

The key properties we identified for each secret include:

We built a component that keeps track of the secret inventory and its metadata. We leveraged a set of heuristics to backfill insufficient metadata centrally (for example, derive impact levels based on the resource the secret protects). For the ones we couldn’t backfill, we crowdsourced missing metadata about secrets from the teams that owned them. 

We built new components over time around the secret inventory/metadata component that evolved to become the Secret Management Platform. This included:

The next two sections describe the key capabilities we have enabled as part of the Secret Management Platform.

Building the capability to automatically rotate secrets at scale can be a challenging endeavor, particularly when dealing with a heterogeneous and complex tech stack like Uber’s.

There are multiple deployment platforms that deploy containerized workloads at Uber: Up and Kubernetes^® for stateless, Odin for stateful/storage, Athena for Apache Flink^® stream processing jobs, YARN for batch analytics workloads, and a few others such as Piper and DSW. We heavily rely on integrations with deployment platforms to distribute secrets to containers when they’re launched. This architecture decouples the Secrets tech stack from any logic within workloads, enabling us to scale systems independently and build features through integrations with deployment platforms. And yes, we have several deployment platforms today. Our Kubernetes migration initiative is in progress at the time of writing ‌this blog.

There are several hundreds of secret types (like cloud credentials, OAuth keys, database username/passwords, SaaS secrets). Certain secret types can be generated and deleted programmatically through APIs, while others can’t. Some of them don’t support seamless rotation of the password since it supports only a single valid password at any given point of time, which means potential downtime for a service consuming the password when it gets rotated. To work around this, we have to consider techniques such as considering the pair of username and password as a single secret and create a new pair as an atomic operation for rotation.    

To support secret rotation and its overall lifecycle management, we designed two sets of APIs: Secret Provider API and Deployment Platform API. The Secret Provider API has functions such as Generate (a new version), Revoke (disable a secret) and Rollback (re-enable a previous version). On the other hand, the deployment platform-facing APIs provide a function to indicate rollout (of secrets for a particular workload), in addition to the API that supports Secrets Fetch (before launching workloads). An orchestrator component called Secret Lifecycle Manager based on Cadence workflows manages states and ties these APIs together in a sequence that supports automatic rotation of a secret (Figure 4).

Based on secret inventory, we prioritized enabling automatic rotation for certain secret types and deployment platforms for enabling these API integrations one by one, based on risk associated with secrets (like resources that they protect) and the volume of secrets (indicating manual work to rotate in absence of automation).

This system orchestrated automatic rotation for 20,000 secrets per month on average, without any human involvement. However, enabling automatic rotation for all secrets is challenging due to the diverse types of secrets and the numerous integrations required across hundreds of secret providers. The effort required to build and maintain these integrations makes full automation impractical in many cases. As a result, we’re pivoting towards secretless authentication and exploring alternative authentication mechanisms to reduce dependency on traditional secrets.

As we introduced advanced monitoring of secrets access through the Secret Management Platform, we identified orphaned secrets with no ownership or usage patterns. We extended the Secret Lifecycle Manager to run a set of checks (ownership, existence of resources it protects, and workloads that consume the secrets) and safely decommission these secrets.

Uber manages integrations with a variety of third-party (3P) vendors, using secrets for authentication or data exchange. Following conversations with teams managing these integrations regarding secret exchanges with third-party vendors, we realized there’s a need to securely exchange secrets between 3P and Uber without exposing them to humans and deploy them into Uber’s production systems safely. 

We designed a SSX (Secure Secret eXchange) system to securely and efficiently share secrets between Uber’s 3P vendors and Uber. An Uber employee can initiate the secret exchange by providing two key pieces of information: 3P vendor contact information—email and Uber internal service name that integrates with the vendor’s APIs (Figure 5).

SSX facilitates this exchange by verifying the vendor with a one-time passcode (OTP) sent to the vendor’s email and allowing the vendor to input the secret into the SSX UI. Once the vendor completes these steps, SSX takes care of securely transferring the secret directly into our internal secret vault without any involvement from Uber employees. 

We took this another step further by extending the system to handle the last-mile delivery of the secret from the secret vault. This was achieved through the integration we built with the deployment platforms described in the previous section. The integration provided two key capabilities to ensure reliability of Uber’s microservice that is integrated with 3P vendor API:

The service owner within Uber was given the flexibility to use either of these capabilities to eliminate manual testing burden and ensure reliability of the service integration with 3P APIs.

So far, SSX has facilitated the exchange of hundreds of secrets between different 3P vendors and Uber.

The team managing Secrets Management Platform consists of 10 engineers. It wouldn’t have been possible to make major improvements described in the above sections, without strong support across the company, accelerating the adoption of new capabilities. The following key changes enabled us to scale out the program company-wide.

We identified key issues in the secret management space early on, backed by data. This was followed by directional objectives and recommendations. We consulted with engineering leadership across the company to gather feedback. Supporting data and surfacing the current state of secrets management led to promoting the project proposal, made it possible to align engineers across several departments, and energized engineers in working towards making the program successful.

We identified key stakeholders within the company who really care about security and secrets as design partners. The department managing Uber’s customer data and payment information is super motivated when it comes to the topic of security. We partnered closely with them to understand their perspective and design solutions. The SSX (Secure Secret eXchange) system was one of the solutions envisioned from this partnership. We also partnered with several peer Platform teams with domain expertise in different areas: Service Deployments, Cloud Infrastructure, and Data/Storage. This led us to build integrations with several internal systems that improved the benefits of building the overall Secret Management Platform. 

Among the identified scope, we used data collected from building the secret inventory to reason about risk and ROI to prioritize features and company-wide adoption strategies. Example: for enabling automatic rotation for secrets, we started by investing in cloud credentials as they guarded some of Uber’s critical resources and cloud adoption was growing over time. So, we prioritized building automatic rotation support for cloud credentials.

Here are our key learnings from building and hardening the Secret Management Platform and building out its integrations:

During the same period that we developed Uber’s Secret Management Platform, vendors such as CyberArk^® and Akeyless^® introduced comparable solutions, grounded in similar principles for managing secrets across multiple cloud vendors. 

Work in the security space is never done. The industry is moving towards secretless architectures and bringing secret management closer to machine/workload identity management (2024 Gartner Report). Looking forward, we’re continuing to improve existing integrations and build new features to improve the security posture of Uber’s infrastructure. 

By building the secrets inventory, we were able to gather valuable insights on the type of secrets and how they are used at Uber. We’re making a major leap towards secretless by eliminating the need for secrets in specific areas, such as our cloud integrations leveraging workload identity federation facilitated by SPIRE. 

Automation failures during secret rotation pose risks to the reliability of the tech stack supporting Uber’s business. We’re currently working towards enabling deployment safety by improving our integration with deployment platforms to treat secret updates similar to code-change deployments. Uber also uses a variety of SaaS apps to support its internal operations. Integrating our platform with Uber’s internal SaaS app management process and supporting rotation of secrets for SaaS is another major area we’re investing in.

At Uber, security is a collective responsibility between the Engineering Security departments and numerous departments across Uber. We’d like to thank Hasibul Haque, Arun Kumar R, Sung Hon Wu, and Gaurav Bansal for helping out on multiple fronts all the way from uniting us on this bold vision to helping navigate and unblock several execution challenges as we built and scaled this system. We also appreciate the efforts of everyone who contributed to the design, implementation, and stabilization of the overall system. Their dedication and expertise have been instrumental in making this initiative a success.