Performance visualisation tools

Overview

grav

A collection of tools to help visualise process execution.

This blog post has some detail on the rationale and implementation detail.

Scheduler profile

Visual cues to inform whether your application's threads are being pre-empted by the kernel scheduler before they are ready to yield the processor.

Scheduler Profile

Pre-requisites: install iovisor BCC

Usage:

$ ./bin/scheduling-profile.sh $PID
Recording scheduling information for 10 seconds
Wrote scheduler-profile-$PID.svg

CPU tenancy

Determine whether application threads would be better restricted to a certain set of CPUs.

CPU Tenancy

Pre-requisites: install perf_events

Usage:

$ ./bin/perf-cpu-tenancy.sh $PID
Recording samples..
Wrote cpu-tenancy-$PID.svg

JVM heap allocation flamegraphs

Use built-in UDST DTrace probes to capture information about heap allocations in a running JVM.

Allocations

Pre-requisites: the following repositories need to be cloned and available locally:

Enable allocation probes using the following JVM params:

-XX:+DTraceAllocProbes

Usage:

# set up environment variables
$ export PERF_MAP_AGENT_DIR=/path/to/perf-map-agent/
$ export FLAMEGRAPH_DIR=/path/to/flamegraph/
$ ./bin/heap-alloc-flames -p $PID -e "java/lang/String" -d 10
Profiling application to generate stack trace symbols
Recording events for 5 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 3 times to write data ]
[ perf record: Captured and wrote 1.156 MB /tmp/perf-$PID.data (3498 samples) ]
Wrote allocation-flamegraph-$PID.svg

Parameters:

-p PID
-i regex of stacks to include (e.g. -i "mycompany" "java/lang/Double")
-e regex of stacks to exclude (e.g. -e "java/lang/String" "C[]")
-d duration of sample in seconds
-s sampling interval (e.g. -s 1000 -> sample every 1000th allocation)
-j path of libjvm.so (default: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so)

Java flamegraphs with thread name

Annotate JVM flamegraphs with thread names for easier focus.

Named threads

Pre-requisites: the following repositories need to be cloned and available locally:

Usage:

# set up environment variables
$ export PERF_MAP_AGENT_DIR=/path/to/perf-map-agent/
$ export FLAMEGRAPH_DIR=/path/to/flamegraph/
$ ./bin/perf-thread-flames.sh $PID
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.343 MB /tmp/perf-$PID.data (835 samples) ]
Wrote flamegraph-$PID.svg

To filter out particular threads, supply a regex as the second argument:

$ ./bin/perf-thread-flames.sh $PID ".*GC.*"
Capturing stacks for threads matching '.*GC.*'
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 7 times to write data ]
[ perf record: Captured and wrote 1.026 MB /tmp/perf-$PID.data (496 samples) ]
Wrote flamegraph-$PID.svg

Animated flamegraphs

Create several flame graphs in time and one animated SVG FlameGraph that captures all of them.

Animated FlameGraph

Pre-requisites: the following repositories need to be cloned and available locally:

Usage:

# set up environment variables
$ export PERF_MAP_AGENT_DIR=/path/to/perf-map-agent/
$ export FLAMEGRAPH_DIR=/path/to/flamegraph/
# run animate-flames PID NUMBER_OF_RECORDINGS SLEEP_SECONDS
# (in this example  NUMBER_OF_RECORDINGS = 5, SLEEP_SECONDS=10)
$ ./bin/animate-flames $PID 5 10 
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.114 MB /tmp/perf-$PID.data (~4993 samples) ]
1 Wrote /tmp/out-threads-$PID-1.collapsed
Wrote flamegraph-$PID-1.svg
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.120 MB /tmp/perf-$PID.data (~5249 samples) ]
2 Wrote /tmp/out-threads-$PID-2.collapsed
Wrote flamegraph-$PID-2.svg
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.117 MB /tmp/perf-$PID.data (~5127 samples) ]
3 Wrote /tmp/out-threads-$PID-3.collapsed
Wrote flamegraph-$PID-3.svg
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.118 MB /tmp/perf-$PID.data (~5143 samples) ]
4 Wrote /tmp/out-threads-$PID-4.collapsed
Wrote flamegraph-$PID-4.svg
Recording events for 10 seconds (adapt by setting PERF_RECORD_SECONDS)
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.118 MB /tmp/perf-$PID.data (~5158 samples) ]
5 Wrote /tmp/out-threads-$PID-5.collapsed
Wrote flamegraph-$PID-5.svg
Wrote animated-flamegraph-$PID.svg

vagrant-grav

A Vagrant box that can be used as a grav development environment on non-linux machines.

Pre-requisite:

Usage:

$ cd vagrant-grav
$ vagrant up

will run a vagrant box with:

  1. ubuntu zesty64
  2. Java OpenJDK8
  3. perf
  4. iovisor BCC
  5. perf-map-agent
  6. FlameGraph scripts
  7. grav scripts
$ vagrant ssh 
...
vagrant@vagrant-ubuntu-trusty-64:~$ echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64/
vagrant@vagrant-ubuntu-trusty-64:~$ echo $GRAV_DIR
/vagrant/grav
vagrant@vagrant-ubuntu-trusty-64:~$ echo $FLAMEGRAPH_DIR
/vagrant/FlameGraph
vagrant@vagrant-ubuntu-trusty-64:~$ echo $PERF_MAP_AGENT_DIR
/vagrant/perf-map-agent

The VM is now ready for grav development.

vagrant@vagrant-ubuntu-trusty-64:~$ logout
Connection to 127.0.0.1 closed.
$ vagrant port
The forwarded ports for the machine are listed below. Please note that
these values may differ from values configured in the Vagrantfile if the
provider supports automatic port collision detection and resolution.

    22 (guest) => 2200 (host)
  8080 (guest) => 18080 (host)

Forwarded port 8080 to port 18080 to allow access to a server running inside the box.

Maintainers

Mark Price

Amir Langer

Comments
  • scheduling-profile seems not to be working on kernel 4.4

    scheduling-profile seems not to be working on kernel 4.4

    Hi,

    First of all many thanks for sharing this work!

    I'm interested in trying the scheduling-profile tool on Ubuntu 16.04, kernel 4.4. I have installed bcc as per iovisor instructions. However, it seems not to be working:

    $ sudo ./scheduling-profile 2918 Recording scheduling information for 15 seconds /virtual/main.c:41:63: warning: incompatible pointer to integer conversion initializing 'char' with an expression of type 'void *' [-Wint-conversion] struct proc_counter_t new_counter = {.proc_name = NULL, .count = 0}; ^~~~ include/linux/stddef.h:7:14: note: expanded from macro 'NULL' #define NULL ((void *)0) ^~~~~~~~~~~ 1 warning generated. No samples for pid 2918

    Can the tool be made compatible with 4.4 kernels or it requires some new eBPF capability found in newer kernels?

    Thank you!

    opened by stefanodoni 4
  • Permission Denied error while attaching to java process inside the container.

    Permission Denied error while attaching to java process inside the container.

    Hi team,

    We aren't able to attach to the java process. Got the below mentioned error while trying to attach to the java process which resides inside the container from the host. Changed the owner permission to all the scripts and tried attaching to the process using the user who owns the process but still the same issue. Could you please review.

    ./scheduling-profile 90961 ./scheduling-profile: line 25: /tmp/jstack-90961.txt: Permission denied

    Thanks Sattish.

    opened by sattishv 3
  • Syntax error in BPF script

    Syntax error in BPF script

    When running heap-alloc-flames I get the error

    /virtual/main.c:49:37: error: expected ';' after top level declarator
    BPF_STACK_TRACE(stack_traces, 10240)
                                        ^
                                        ;
    1 error generated.
    

    Adding the semicolon to src/heap/heap_profile.py appears to fix this issue.

    opened by jschewebbn 1
  • heap_profile fails to parse some stacks

    heap_profile fails to parse some stacks

    Failed to decode stack: [u'start_thread', u'java_start', u'JavaThread::run', u'JavaThread::thread_main_inner', u'thread_entry', u'JavaCalls::call_ virtual', u'JavaCalls::call_virtual', u'JavaCalls::call_helper', u'call_stub', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'I nterpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'com/codahale/metrics/WeightedSnapshot:::', u'dtrace_object_al loc Runtime1 stub', u'SharedRuntime::dtrace_object_alloc'] Failed to decode stack: [u'start_thread', u'java_start', u'JavaThread::run', u'JavaThread::thread_main_inner', u'thread_entry', u'JavaCalls::call_ virtual', u'JavaCalls::call_virtual', u'JavaCalls::call_helper', u'call_stub', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'I nterpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'com/codahale/metrics/CsvReporter:::reportGauge', u'Interpreter', u' Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Java_java_net_Inet6AddressImpl_loo kupAllHostAddr', u'jni_NewObjectArray', u'ObjArrayKlass::allocate', u'SharedRuntime::dtrace_object_alloc'] Failed to decode stack: [u'start_thread', u'java_start', u'JavaThread::run', u'JavaThread::thread_main_inner', u'thread_entry', u'JavaCalls::call_ virtual', u'JavaCalls::call_virtual', u'JavaCalls::call_helper', u'call_stub', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'I nterpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'java/util/TreeMap:::entrySet', u'dtrace_object_alloc Runtime1 stub' , u'SharedRuntime::dtrace_object_alloc'] Failed to decode stack: [u'start_thread', u'java_start', u'JavaThread::run', u'JavaThread::thread_main_inner', u'thread_entry', u'JavaCalls::call_ virtual', u'JavaCalls::call_virtual', u'JavaCalls::call_helper', u'call_stub', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'I nterpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'Interpreter', u'com/codahale/metrics/MetricRegistry:::getMetrics', u'dtrace_object_ alloc Runtime1 stub', u'SharedRuntime::dtrace_object_alloc']

    opened by epickrram 1
  • collapse stacks for threads with similar prefixes

    collapse stacks for threads with similar prefixes

    If threads in a thread pool have the same prefix, add an option to collapse their stacks together:

    message-parsing-pool-thread-1 message-parsing-pool-thread-2

    should collapse to message-parsing-pool-thread-*, this can be useful when all threads in a pool perform the same function.

    opened by epickrram 0
  • Scheduling profile - breakdown of incoming processes.

    Scheduling profile - breakdown of incoming processes.

    Hi Mark,

    first thank you for the very useful tools you have here - didn't have time to play with them yet but it is on my todo list. Quick question, in you blog post you mentioned the following:

    "To make the scheduling-profile tool more useful, I intend to annotate the runnable state portion of the bar chart with a further breakdown detailing the incoming processes that kicked application threads off-CPU. This will provide enough information to direct system-tuning efforts to ensure an application has the best chance possible to get CPU-time when required."

    Is this feature still planned? I think it would make the scheduling profile even much more useful for fine-tuning application jitter.

    Thank you and kind regards, Petr

    opened by postol 1
  • aggregate stacks based on regex

    aggregate stacks based on regex

    It would be useful to be able to aggregate stacks based on (e.g.) package name.

    Something like:

    ./heap-alloc-flames $PID -g "org.hibernate.*"

    this will reduce the number of flames, making rendering quicker.

    opened by epickrram 1
Owner
Mark Price
Mark Price
An application metrics facade for the most popular monitoring tools. Think SLF4J, but for metrics.

Micrometer Application Metrics An application metrics facade for the most popular monitoring tools. Instrument your code with dimensional metrics with

Micrometer Metrics 3.7k Dec 30, 2022
Small set of tools for JVM troublshooting, monitoring and profiling.

Swiss Java Knife (SJK) SJK is a command line tool for JVM diagnostic, troubleshooting and profiling. SJK exploits standard diagnostic interfaces of JV

Alexey Ragozin 3.2k Jan 3, 2023
Sends stacktrace-level performance data from a JVM process to Riemann.

Riemann JVM Profiler riemann-jvm-profiler is a JVM agent that you can inject into any JVM process--one written in Clojure, Java, Scala, Groovy, etc.--

Riemann 288 Sep 21, 2022
A scientific charting library focused on performance optimised real-time data visualisation at 25 Hz update rates for data sets with a few 10 thousand up to 5 million data points.

ChartFx ChartFx is a scientific charting library developed at GSI for FAIR with focus on performance optimised real-time data visualisation at 25 Hz u

GSI CS-CO/ACO 386 Jan 2, 2023
A scientific charting library focused on performance optimised real-time data visualisation at 25 Hz update rates for data sets with a few 10 thousand up to 5 million data points.

ChartFx ChartFx is a scientific charting library developed at GSI for FAIR with focus on performance optimised real-time data visualisation at 25 Hz u

GSI CS-CO/ACO 385 Dec 30, 2022
GMC-Tools - Plugin with basic tools for Minecraft server administrator

GMC-Tools - Plugin with basic tools for Minecraft server administrator. Currently we do not support configuration files and we do not recommend using this plugin on production servers.

GamesMC Studios 4 Jan 14, 2022
Performance analysis tools based on Linux perf_events (aka perf) and ftrace

perf-tools A miscellaneous collection of in-development and unsupported performance analysis tools for Linux ftrace and perf_events (aka the "perf" co

Brendan Gregg 8.8k Dec 28, 2022
A simple configuration library for Java applications providing a node structure, a variety of formats, and tools for transformation

Configurate Configurate is a simple configuration library for Java applications that provides a node-based representation of data, able to handle a wi

SpongePowered 274 Jan 3, 2023
JCTools - Concurrency tools currently missing from the JDK.

JCTools Java Concurrency Tools for the JVM. This project aims to offer some concurrent data structures currently missing from the JDK: SPSC/MPSC/SPMC/

null 3.1k Dec 28, 2022
Tools for working with generic types

TypeTools A simple, zero-dependency library for working with types. Supports Java 1.6+ and Android. Introduction One of the sore points with Java invo

Jonathan Halterman 585 Jan 9, 2023
Automon combines the power of AOP (AspectJ) with monitoring or logging tools you already use to declaratively monitor your Java code, the JDK, and 3rd party libraries.

Automon Automon combines the power of AOP (AspectJ) with monitoring tools or logging tools that you already use to declaratively monitor the following

Steve Souza 561 Nov 27, 2022
Testing tools for javac and annotation processors

Compile Testing A library for testing javac compilation with or without annotation processors. See the javadoc for usage examples. License Copyright 2

Google 639 Dec 14, 2022
Tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures.

PROJECT STATUS: RETIRED The Simian Army project is no longer actively maintained. Some of the Simian Army functionality has been moved to other Netfli

Netflix, Inc. 7.9k Jan 6, 2023
Pcap editing and replay tools for *NIX and Windows - Users please download source from

Tcpreplay Tcpreplay is a suite of GPLv3 licensed utilities for UNIX (and Win32 under Cygwin) operating systems for editing and replaying network traff

AppNeta, Inc. 956 Dec 30, 2022
BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

BPF Compiler Collection (BCC) BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and

IO Visor Project 16.3k Dec 30, 2022
Tools for tracking down memory / JVM problems & generating predictable-as-possible VM behaviour

Hawkshaw Tools for tracking down memory / JVM problems & generating predictable-as-possible VM behaviour You can Use Hawkshaw to mimic application obj

Martijn Verburg 40 Jan 9, 2021
An application metrics facade for the most popular monitoring tools. Think SLF4J, but for metrics.

Micrometer Application Metrics An application metrics facade for the most popular monitoring tools. Instrument your code with dimensional metrics with

Micrometer Metrics 3.7k Dec 30, 2022
Small set of tools for JVM troublshooting, monitoring and profiling.

Swiss Java Knife (SJK) SJK is a command line tool for JVM diagnostic, troubleshooting and profiling. SJK exploits standard diagnostic interfaces of JV

Alexey Ragozin 3.2k Jan 3, 2023
Full Featured Google Chrome Dev Tools to JavaFX WebView browser debugging.

JavaFX WebView Debugger Via WebSocket connection to Google Chrome Dev Tools JavaFx WebView debugging with Chrome Dev tools is highly dependent on Goog

Vladimir Schneider 56 Dec 19, 2022