Uber leverages storage technologies to run its real-time operations. This includes storing online data in open-source databases like MySQL^®, Apache Cassandra^®,  etcd^®, and Apache Zookeeper^®, as well as Uber’s homegrown storage solutions like Docstore and Schemaless hosted on top of Uber’s Stateful Platform.

Database backup recovery is vital for Uber’s business continuity and disaster recovery scenarios. This supports use cases like outage mitigation, recovery from data corruption, and forensics and regulatory compliance. It also supports production-like environment simulation for load testing, data integrity, and security.

Uber’s online storage solutions hold 10s of petabytes to serve critical business operations. They also serve millions to billions of requests per second across Uber’s global business verticals. With that scale, close to 100 PB of data is backed up from databases at different intervals. TBs to PBs of data can be restored within a few minutes to a few hours. 

This blog describes recent developments in Uber’s robust backup recovery system for online databases.

Evolving Uber’s backup recovery system at scale presented multiple challenges:
  

This blog describes some of the optimizations we’ve implemented to address such challenges.

The enhanced Backup Recovery system on top of Uber’s Stateful Platform provides a unified experience by abstracting all aspects of managing the backup restore ecosystem and its underlying components. These aspects include centralized adaptive scheduling for backup workloads across the fleet of stateful databases. Each backup workload backs up the current database snapshots with the overall state propagation and monitors backups. On the restore side, it conducts periodic recovery testing to verify completeness and correctness of existing backups with the underlying restore automation. Overall, this establishes the notion of a CBCR framework (Continuous Backup Continuous Recovery).

The Backup Recovery system uses a snapshot-based backup and recovery architecture to manage the Stateful fleet for improved disaster recovery. 

The Backup Recovery system contains these components:

Continuous Backup allows reliable backups for all databases to be taken continuously. It allows backups to be taken at a sufficient rate while improving storage costs and network usage. 

Time Machine is a critical component of the Continuous Backup framework, serving as the centralized orchestrator for backups. It functions as a global, adaptive scheduler, periodically triggering backup workloads. This system uploads at least a few PBs of data daily. Uber’s networking bandwidth is shared across multiple Uber business-critical workloads in online/offline flows. Time Machine addresses the challenge of scheduling backups adaptively without disrupting the serving stack’s network usage. 

Time Machine has an optimal selection engine with inbuilt client-side rate-limiting to decide which databases to back up in a given scheduling cycle. It considers multiple signals like:

This intelligent scheduling allows fleet-wide database backups to be efficiently and evenly distributed at scale. Upon making optimal selection decisions, Time Machine invokes database-specific triggers to initiate runtime backup workloads, providing reliable backups, ‌integrity, and protection.

The backup process follows these steps: 

The Backup framework is a generic backup driver integrated with technology-specific plugins to execute database snapshotting logic and upload efficiently and reliably to Uber’s blob storage. A runtime backup workload runs as an on-demand sidecar container in parallel to mainstream containers like technology node-worker and database containers. A node is a unit of stateful workload running as a containerized cgroup powered by the host fleet, following a goal-state-driven architecture. 

Here’s how the backup flow works: 

Here’s how the database snapshotting logic works for each technology:

The Restore framework is technology-agnostic, similar to the Backup framework, designed to enable automated and consistent restoration. Its modular design allows the framework to be extensible and adaptable to different database architectures. Furthermore, the framework eliminates manual intervention, reduces recovery time, and minimizes the risk of human error in any in-place and out-of-place recovery scenarios..

The framework provides a generic driver with extensible database technology-specific plugins to define key components. The Manifest Provider plugin is customized to fetch a database-specific backup index manifest as described above, where each backup is associated with a detailed backup index. The backup index references all snapshot files from different backup types (full, incremental, or differential) and files needed during a restore. Database Loader defines the database ingestion logic tailored to the database architecture (MySQL, Cassandra, etcd) to bring the database to a usable state.

Here’s how the database loading process works for each technology:

The Restore framework’s integration with the Continuous Restore framework facilitates continuous validation and readiness for real-world situations.

The Continuous Restore framework confirms data completeness and correctness by frequently validating the restored backups. Framework offers intelligent scheduling with configurable restore test cadences that offer regular and ad-hoc validation. The scheduling logic considers hardware resource availability to prevent production impact. 

Restore testing strategies include dedicated database testing and random database testing. With dedicated testing, it uses predefined databases with known data to perform detailed, end-to-end restoration and validation. With randomized testing, production-scale databases are selected with defined characteristics for broader validation of restore workflows under real-world conditions.

Post-restore, the framework performs robust data verification, including file integrity validations and byte-to-byte data comparisons for known dedicated databases. The framework also collects detailed insights, including restore success rates, restoration rates against a backup footprint, data integrity validation results, and multiple performance-oriented signals, reported further for monitoring and analysis.

The restore evaluation process follows four phases:

Benefits of the Continuous Restore framework include: 

By continuously validating restore processes, our framework reinforces disaster recovery readiness, safeguards critical data, and strengthens system resilience at scale.

Backup Recovery for the stateful fleet at Uber is crucial for Uber’s business continuity and disaster recovery plan. We hope to explore additional system developments in future blogs.