Introduction

In a previous blog we introduced the high-level architecture of Uber’s cloud-native observability and alerting platform and drew a parallel with trail running. This time, we go one level deeper and focus on a real implementation, while keeping the same analogy.

For this blog about context-rich observability alerting, we’re going to keep the same tradition of the previous article and make a parallel with an ultra marathon: Lavaredo Ultra Trail by UTMB. If you’ve ever stood at the start line in Cortina d’Ampezzo at midnight, surrounded by thousands of runners waiting for the countdown, it’s easy to think that the race begins with the starting gun. In reality, that’s when the easy part starts.

The Real Race Happened Months Earlier

Hundreds of kilometers of training, experimenting with nutrition, choosing the right equipment, studying the route, preparing mentally for hours of solitude, and learning when to push and when to conserve energy. Race day is just the moment where all that preparation is validated.

Building an observability platform is remarkably similar. Collecting metrics, logs, traces, and telemetry is necessary, but it’s only the preparation. During day-to-day operations, all of that should become almost invisible—a silent worker continuously gathering information in the background. What’s important is what engineers can do with that information when something goes wrong.

As discussed in the previous blog, traditional monitoring is far from obsolete. It’s still an essential building block. However, monitoring alone is reactive—it tells us what happened. Observability takes the next step by helping us understand why it happened and, more importantly, what action should be taken next. That shift, from collecting data to delivering context, is where the real value lies for engineering teams. 

Just like the breathtaking landscapes of the Dolomites aren’t a reward for just showing up at the start line, actionable observability is the result of months of preparation. Moments like seeing the early light over Tre Cime di Lavaredo or running through the rugged Dolomites only happen because of the groundwork laid before the race. Similarly, the meaningful notifications an engineer receives during an incident are the visible outcome of all the behind-the-scenes work.

The Right Tools Are Only Part of the Solution

Another important thing to note is the sheer abundance of tools available. Some are open-source, like Grafana®, Prometheus®, and Alertmanager, while others are commercial platforms that have become industry standards, like PagerDuty®, and Atlassian Jira®. We’re not mentioning the cloud-managed tools for WiFi or telemetry, only the ones related to observability and alerting.

These tools are powerful, but they can also become your biggest enemy. Used correctly, they provide valuable information. Used without context, or used excessively, they generate alert fatigue, false positives, and eventually cause engineers to ignore the notifications designed to help them.

In our own environment, Alertmanager can be quite noisy based on our alerting rules. That naturally raises an important question: how many of those alerts represent actionable incidents, how many are transient flaps, and how many are simply false positives? Alert tuning is an ongoing feedback loop, and every engineering team faces the same tradeoff. Tighten the thresholds too much, and you risk missing the early indicators of incidents with the potential for significant business impact. Loosen them too much, and engineers become overwhelmed by noise. Add webhook-based integrations from multiple platforms, where the same underlying incident may be reported several times from different perspectives, and it becomes increasingly difficult to separate signal from noise. Figure 1 shows the firing alerts distribution over a week.

Figure 1: Firing alerts distribution.

Nutrition is one of the most important factors in an ultra-marathon. Carbohydrates are your best friend—but only if consumed at the right moment, in the right quantity, and adapted to your pace. Too little energy, and performance drops. Too much, too early, or the wrong combination, and your race may end before it has truly begun.

Alerting behaves the same way. An alert delivered without context is no different than eating without a nutrition strategy. More isn’t better. Better is better.

Every Piece of Equipment Has a Purpose

Mandatory equipment in an ultra-marathon is designed around resilience. The running vest carries water, food, a waterproof jacket, a headlamp, an emergency blanket, running poles, and spare batteries. Together, these items allow you to safely handle whatever the mountains throw at you. Our corporate network is no different.

Uber’s corporate network environment consists of multiple platforms, including Cisco Meraki®, HPE Mist™, HPE Aruba® for cloud-managed Wi-Fi, Cisco ThousandEyes®, HPE CapeSensors, traditional on-premises infrastructure, and our own telemetry pipelines. Each platform contributes a different perspective on the network, and none of them tells the complete story on its own.

Are SNMP-Based Alerts Still Enough?

This naturally leads to an important question. With modern networking platforms capable of generating streaming telemetry, enriched events, and AI-assisted analytics, how much should we continue relying on alerts built exclusively from SNMP metrics?

While we’ve focused on traditional SNMP metrics, this strategy remains just as effective for metrics ingested/fetched via modern methods like gRPC dial-in.

SNMP metrics still matter, but they remain extremely valuable for visualization, long-term trending, capacity planning, and cross-vendor dashboards. They provide a vendor-neutral view of our infrastructure and remain an important source of operational data.

However, building an alerting strategy around individual SNMP metrics is no longer the most effective approach. Independent alerts are bad because they generate noise and create alert storms. Most importantly, they force engineers to manually correlate information that the system should already understand

Consider these examples:

• An isolated CPU utilization alert rarely tells the complete story. 
• An interface error counter, viewed independently, is simply another data point. 
• 10 independent alerts usually don’t describe 10 independent problems. More often than not, they describe a single incident viewed from 10 different perspectives. 

To illustrate, we define Prometheus interface alerts based on traffic, triggering only if the interface has shown activity in the past 24 hours, excluding specific descriptions. Due to an issue, all access switches in a specific office began alerting on interfaces that should have been excluded, resulting in around 600 individual alerts (see Figure 2). This was certainly not our intent. Furthermore, when factoring in the associated actions—such as alerts, Slack messages, and tickets—the operational situation quickly spiraled from bad to worse.

Figure 2: Firing alerts.

To address this, we structured our alerting model into two complementary layers. This approach mirrors Uber’s See the forest and the trees value: while the second layer helps us analyze individual infrastructure trees—the raw data points—the first layer gives us the forest view, allowing us to grasp the broader operational landscape. 

The first (and most important) layer consists of platform-generated notifications. Modern networking platforms already perform sophisticated internal correlation and AI-assisted analysis. 

To put this into perspective, examining a monthly span of notifications from HPE Mist—a pillar of our modern observability stack—reveals approximately 100 actionable insights, including internal testing. This represents a remarkably controlled volume compared to traditional noise.

Figure 3: HPE Mist alerts distribution for 30 days

We complement this layer with a handful of carefully selected infrastructure alerts, such as WAN-down detection, where our own reachability validation continuously stores results in Prometheus.

The second layer consists of traditional SNMP-based alerts. These alerts are intentionally treated as supporting evidence rather than primary notifications. Why? Because the first layer tells us that users are experiencing a problem; the second layer helps explain why. Instead of competing with each other, the two layers complement one another. One delivers the incident, and the other enriches the context.

To understand why we demoted SNMP alerts to a supporting role, look at the Alertmanager data over that exact same 30-day span. Alertmanager generated ~5.5k SNMP-based alerts. Even acknowledging that alert definitions can always be further optimized, triaging this volume manually is an impossible marathon. It consumes massive amounts of engineering energy just to read the notifications, let alone fix them. By ensuring these two layers complement rather than compete, one delivers the incident, and the other enriches the context.

Figure 4: SNMP based alerts.

Correlation Is Where Observability Begins

So how do we transform hundreds of independent events into something meaningful? This is where our Alerts Ingestion service becomes the entry point for the entire system.

Every event—whether originating from Alertmanager, Cisco Meraki, HPE Mist, ThousandEyes, Kibana®, CapeSensors, or any future platform—is received through webhooks and normalized into a common processing pipeline. And let’s not forget about logging alerts defined in Kibana. We treat log messages exactly the same way we treat any other event.

To recap from the previous article, here is the high-level overview of the system architecture, with a few auditions made since then. More details about the polling part and the design can be found there.

Figure 5: HLD of the observability&alerting system.

To accomplish this, we needed a common correlation key. For example, in our network environment, we use the correlation key of the office/data center ID. Finding that common identifier is only possible because our infrastructure follows consistent automation and naming standards. If you’re interested in that journey, see the previous blog about infrastructure automation.

Without this grouping, a single site outage that triggers alerts across multiple platforms produces a storm of fragmented data: N individual Jira tickets, N Slack channel posts, and N PagerDuty incidents. To solve this, our caching layer trades a short buffer window for consolidation, creating one parent Jira ticket for the group and attaching all related alerts as subtasks.

Figure 6: Caching  layer.

Atomic State Transitions at Scale

Handling concurrent webhook fires at scale introduces a significant race-condition risk. If a major outage triggers 50 alerts in the same second, the system must definitively know which process owns the parent ticket creation.

To prevent collisions, all state-machine operations in the caching layer execute as Lua-atomic scripts within Redis. This guarantees that concurrent workers cannot double-create sessions or trigger a double-flush.

Every time a state transition occurs, the system enforces state consistency and cleanup. For example, every successful Lua mutation automatically re-applies a Time-To-Live (TTL) to the session key. This acts as a self-cleaning mechanism: if the service experiences a hard crash, the session will eventually evaporate and the next alert will start clean, preventing permanently locked states.

Figure 7: Alerts state.

• BUFFERING: The first alert lands and subsequent arrivals dynamically push the flush timer forward using a 60-second debounce window (min_wait_seconds) to capture the full storm. To ensure an incident isn't buffered indefinitely, this state is strictly capped by a 180-second maximum wait (max_wait_seconds), guaranteeing a timely notification.
• FLUSHING: Once the timers expire, the system locks the session and proceeds to synchronously open the parent ticket. If the ticket creation fails (e.g., a Jira API timeout) the state will roll back to BUFFERING so the next attempt can safely retry.
• OPEN: Now that the ticket exists, the session remains open for a 3600-second join window (join_window_seconds), allowing any late-arriving alerts to seamlessly bypass the buffer and join as subtasks. Finally, when downstream platforms report that all alerts have cleared, after waiting out a 600-second flap-protection window (grace_seconds), it safely triggers the auto-close logic.

Much like an ultra-marathoner adjusts their pace based on the terrain, our caching layer relies on configurable timers (min_wait, max_wait, join_window, and grace) to dynamically adapt to network behaviors and avoid alert flapping.

Context Is the Real Product

Up to this point, we’ve discussed architecture and design principles. We have time-series data, streaming telemetry, platform-generated incidents, and correlated events grouped by correlation key. In other words, we have context.

This is precisely where AI becomes valuable, because AI performs best when provided with rich, structured context. Beyond the current state of the network, we can enrich that context with historical incident records, internal knowledge bases, documented resolutions, and operational procedures.

With context, AI becomes an operational assistant rather than just another analysis tool. Meaningful context extends beyond raw telemetry and includes correlated events, affected sites or services, historical incidents, recent configuration changes, platform-generated insights, known resolutions, internal runbooks, and operational knowledge. Equipped with this information, AI can generate recommendations that are relevant to the specific incident instead of generic responses.

To showcase this, the following picture depicts a real situation where an entire office was lost because both ISP links went down. Without the correlation engine, that would produce 20 tickets. There is also the AI agent analysis.

Figure 8: AI + correlation example.

Conclusion

Crossing the finish line in Cortina d’Ampezzo is an emotional moment, but experienced trail runners know that success isn’t defined by a single climb or one perfect descent. It’s the accumulation of hundreds of small decisions made correctly over many hours—training consistently, pacing wisely, eating at the right moments, choosing the proper equipment, and adapting to changing conditions.

Observability follows the same principle. Success isn’t measured by how many metrics we collect or how many alerts we generate. It’s measured by how quickly engineers can understand an incident, make informed decisions, and restore service. 

The goal is to produce the right notification, at the right time, with the right context. Just like finishing an ultra-marathon, the destination matters, and it’s everything that happens before reaching the finish line that makes success possible.

Acknowledgments

Cover Photo Attribution is from the author’s personal gallery.

The Grafana Labs® Marks are trademarks of Grafana Labs, and are used with Grafana Labs’ permission. We are not affiliated with, endorsed or sponsored by Grafana Labs or its affiliates.

Jira® is a registered trademark of Atlassian Pty Ltd 

Kibana® and Elasticsearch® are registered trademarks of Elasticsearch BV, registered in the U.S. and in other countries.. No endorsement by Elasticsearch is implied by the use of these marks.

Meraki® and ThousandEyes® are registered trademarks of Cisco Technology, Inc.

MIST™ and Aruba®  are trademarks of HPE.

PagerDuty® is a registered trademark of PagerDuty, Inc.

Slack® is a registered trademark and service mark of Slack Technologies, Inc.

Thanos™, Prometheus® are trademarks or  registered trademarks of The Linux Foundation® in the United States and other countries. No endorsement by The Linux Foundation is implied by the use of these marks.