Skip to main content
Engineering

JVM Profiler: An Open Source Tool for Tracing Distributed JVM Applications at Scale

June 28, 2018 / Global
Featured image for JVM Profiler: An Open Source Tool for Tracing Distributed JVM Applications at Scale
Figure 1. In a distributed environment, Spark applications run on multiple servers.
Figure 2. Our JVM Profiler is composed of several different profilers that measure specific metrics related to JVM usage and performance.
Figure 3. Our JVM Profiler integrates with Uber’s data infrastructure system.
roleprocessUuidmaxHeapMemoryMBavgProcessCpuLoad
executor6d3aa4ee-4233-4cc7-a628-657e1a87825d2805.2553257.61E-11
executor21f418d1-6d4f-440d-b28a-4eba1a3bb53d3993.9695829.56E-11
executora5da6e8c-149b-4722-8fa8-74a16baabcf83154.4844748.18E-11
executora1994e27-59bd-4cda-9de3-745ead954a272561.8473748.58E-11
Figure 3. Our JVM Profiler identified that 70 percent of applications were using less than 80 percent of their allocated memory.
Bo Yang

Bo Yang

Bo Yang was a Senior Software Engineer II at Uber. Bo worked in the Big Data area for 10+ years in various companies building large-scale systems including a Kafka-based streaming platform and Spark-based batch processing service.

Nan Zhu

Nan Zhu

Nan Zhu is a software engineer in Uber’s Seattle office focused on Spark Observability.

Felix Cheung

Felix Cheung

Felix Cheung is a senior software engineer in Uber's Seattle office focused on Spark.

Xu Ning

Xu Ning

Xu Ning is a Senior Engineering Manager in Uber’s Seattle Engineering office, currently leading multiple development teams in Uber’s Michelangelo Machine Learning Platform. He previously led Uber's Cherami distributed task queue, Hadoop observability, and Data security teams.

Posted by Bo Yang, Nan Zhu, Felix Cheung, Xu Ning

Category: