Introduction

Managing a global corporate network at Uber’s scale can feel a bit like running an ultra-marathon. There are long stretches of smooth sailing, but you’re always preparing for the unexpected mountain pass or sudden change in weather. For years, our engineering teams have navigated this terrain with a traditional, monolithic monitoring system. Frankly, it felt like running in heavy hiking boots—sturdy, but slow, inflexible, and exhausting to scale up any hill.

We knew we needed to switch to a modern pair of carbon-fiber running shoes. This meant a complete overhaul: a journey to replace our legacy system with a cloud-native observability platform built for speed, flexibility, and endurance on an open-source stack.

Training Plan: Scope and Vision

Before diving deeper, it’s important to clarify where this system operates and what we wanted to achieve.

The Scope: Uber’s Corporate Network

The CorpNet Observability Platform focuses exclusively on Uber’s corporate network—the infrastructure that connects offices, data centers, cloud environments, and internal services.

It’s not a production telemetry platform; instead, it monitors and analyzes:

The mission is simple: make Uber’s internal network as observable, reliable, and automated as the systems it supports.

The Vision: Upgrading Our Gear with Open Source

Our vision was to build a new system on the pillars of data quality, scalability, and actionable data. We chose a foundation of best-in-class open-source tools.

A Cloud-Native Architecture Built for the Long Run

To achieve our vision, we adopted a cloud-native, microservices-based architecture deployed on Kubernetes®. Each component is a modular, containerized service that communicates via APIs, making the system open to new integrations.

The system is deployed globally across regions (USC, EMEA, and APAC). This is like having well-stocked aid stations along the race course, ensuring our monitoring probes provide accurate, low-latency measurements by being geographically close to our devices.

Running on Kubernetes also gives the system an incredible ability to bounce back. If a component stumbles, it’s automatically restarted, ensuring high availability.

Looking at other components of the architecture, we use a scalable, open-source stack. We use Prometheus for core time-series metrics, extended by Thanos for long-term storage and a global query view. Telegraf is our versatile data collection agent, while Grafana and Kibana provide unified dashboards for metrics and alerts.

Finding the Rhythm on the Mountain: The Architecture Behind the Run

Our observability platform is built like a trail race—every component works in sync to keep momentum on unpredictable terrain.

Keeping Up the Pace: Dynamically Tracking Network Changes

In trail running, staying still means falling behind. The terrain changes under your feet—the weather, the altitude, the fatigue—and success depends on how fast you adapt.

Our observability system faces a similar challenge. Without adaptability, it’d be like running the same loop forever—predictable, static, and quickly outdated.

That’s why we introduced the Dynamic Config App, a service that keeps the entire monitoring system moving in rhythm with Uber’s corporate network.

Before this, Telegraf pollers relied on static configuration files—great for a short sprint, but not for a long, evolving route. Every time a new device appeared in the inventory or a router changed region, or a device was set for the maintenance state. It was functional, but it couldn’t scale or keep pace with a live, global environment.

The Dynamic Config App changed that completely.

This service dynamically tracks updates in the inventory stored in Elasticsearch, which is continuously refreshed from ServiceNow® (the source of truth). Whenever a device is added, removed, or reassigned, the Config App immediately reflects that change in Telegraf’s configuration—no manual edits, no redeploys.

Each Telegraf agent is launched with parameters that make it self-adjusting:

Using parameters like /v1/snmp/cisco?region=EMEA, the app filters which devices each Telegraf agent should poll based on region, site, or device type.

With this setup, Telegraf periodically checks the configuration URL to see if anything has changed. If the Last-Modified timestamp differs, it automatically reloads its configuration.

This mechanism allows us to:

The result? A system that never stands still.

Just as a good ultrarunner adapts to every climb and descent, our observability platform adapts to every change in the corporate network—automatically, continuously, and effortlessly.

Finding the Trail Beyond the Map: From Monitoring to Observability

Monitoring answers “Is it up?”
Observability answers “Why does it feel slow uphill?”

We transitioned from static metrics to context-rich observability, powered by automation and smart data flow.

In trail running, checkpoints tell you where you’ve been, but not how you’re doing. They confirm you’re still on the course—but they don’t explain why your pace has dropped, where the next climb begins, or what’s happening behind the next ridge.

Traditional monitoring systems are like those checkpoints. They tell you if something’s up or down, but not why or what’s next. Observability, on the other hand, is about understanding the entire race—the terrain, the conditions, and how every part of the system moves together.

That shift—from monitoring to observability—is where the Alert Ingestion App comes into play.
If Telegraf and the Dynamic Config App represent the legs and lungs of our system, the Alert Ingestion App is its coach and race strategist.

Built on FastAPI™, Celery™, and Redis®, this service listens to every signal along the route—not just from Prometheus alerts, but also from third-party systems like Meraki®, HPE MIST™, and other webhook integrations.

It ingests alerts asynchronously, processes them in parallel, and ensures that critical events reach the right teams without flooding Slack® or PagerDuty® with noise.

Think of it as an intelligent checkpoint system:

By routing every alert—from Prometheus, MQTT (Message Queuing Telemetry Transport) sensors, or network APIs—into a single Elasticsearch index, the Alert Ingestion App creates one continuous trail of operational awareness. Engineers can retrace the full journey of an incident, from its first ping to its full resolution, across all systems and regions.

Together with the Dynamic Config App, it transforms a static monitoring setup into a living, breathing observability ecosystem—one that adapts to the terrain, listens to the data, and keeps perfect pace with the ever-changing rhythm of Uber’s corporate network.

Paving the Way for a Personal Best: AIOps and Faster Resolutions

Centralizing all our alert data—both firing and resolved—creates a rich, historical dataset. This is like a detailed training log of every run we’ve ever done, and it’s the perfect fuel for AIOps (AI for IT Operations). This historical context allows us to bring in an elite performance coach.

An AI engine processes all alert sources to provide advanced insights into network behavior patterns. By analyzing past performance, it helps engineers pinpoint root causes faster, significantly lowering our MTTR (Mean Time To Repair) and helping us set a new personal best for incident resolution.

This architecture also supports interactive AI agents, like a Slack bot, that act as a pacer during an incident. Engineers can ask questions in natural language (“How have alerts for this site code trended over the last 24 hours?“) and get immediate, context-rich answers, helping them push through the toughest parts of an incident.

Conclusion: The Finish Line

By migrating to this modern, cloud-native observability platform, we’ve changed our shoes and our entire approach to running the network marathon. We now have:

This project has been a rewarding endurance event. The result is a platform that empowers our teams with the deep insights needed to manage a network at Uber’s scale and paves the way for a future of even smarter, AI-driven network operations