# ./bin/perf-java-top 19835
Error: Could not find or load main class net.virtualvoid.perf.AttachOnce

scripts expects attach-main.jar to be found in out dir, while it is in the project root dir.

I think PMA is in a unique position to break new grounds in Java profiling by reporting the Java stack while highlighting the separation between real frames and virtual frames(inlined methods). To do this we will need:

1. Report all frames when reporting inlined data: This can easily lead to VERY long strings. I'm not sure what the maximum 'method' name length supported by perf is.
2. Support from flame graphs in colouring and displaying the frames.

Just a few small improvements, if you want them:

• fflush(): to avoid a 4k buffering issue where the map file was always missing the most recent entries.
• livemap: an option to write to an alternate file. This allows other software to be used to tidy up and write the real map file before running perf. (This is a step I've found to be necessary with complex production code, where -- despite a correct map file and perf profile -- perf's translation step gets messed up, unless the map file is tidied first. I have a program to do this I'll publish.)
• msig: I changed the behavior to not include the method signatures, unless you set this option.

I also have a fix (or rather, a hack) in-hand for the frame pointer issue I need to publish as well...

HI guys!

I have produced a flame graph with:

PERF_COLLAPSE_OPTS="--kernel --tid" PERF_RECORD_FREQ=99 PERF_RECORD_SECONDS=10 PERF_MAP_OPTIONS=unfoldall ~/perf-map-agent/bin/perf-java-flames <pid>

But seems that (some) C stack frames are not correctly shown as children of the related Java calls. Am I missing anything in the configuration?

Thanks, Franz

Hi

I'm trying to use your project on a fedora 24 using openjdk 1.8.0.101 . The problem can be seen using perf-java-top and flame graphs as well . The create-java-perf-map seems to work well but it did saw how could get its output with perf to plot with frame. Jstack works as well (but as you mention it would be a terrible poor performance and quality).

I thought it would be something with preserving the framepointer but the jdk version is compatible and I'm using the following JVM options:

-XX:+PreserveFramePointer -XX:+StartAttachListener -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:-OmitStackTraceInFastThrow -XX:+ShowHiddenFrames

Do you have any suggestions about how to solve this problem?

Thanks in advance Luan

Hi, This is a bit of a mixed bag, so feel free to cherry pick the bits you find most relevant. The most important aspects are:

1. Fix a bug in the inlined address range computation where the range was assigned to the wrong method.
2. Error handling for JVMTI calls (only in method signature, the same pattern needs applying to all JVMTI interactions).

In addition I added some comments, renamed variables for clarity (I think), and added a more compact format for inlined methods printing. Happy to have a discussion to make the PR more palatable. I would like to improve formatting further and settle on good way to configure it, but not sure how important backwards compatibility is. Thanks, Nitsan

I often see "Interpreter" frames. I think I was expecting more symbols.

   100.00%     java  perf-20844.map  [.] Interpreter
               |
               --- Interpreter
                   Interpreter
                   Interpreter
                   Interpreter
                   call_stub
                   JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)
                   jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*)
                   jni_CallStaticVoidMethod
                   JavaMain
                   start_thread

Hi I have ran cmake . but show me as below do you know what it is happened? finished or unfinished?

   -- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at CMakeLists.txt:24 (include):
  include could not find load file:

    UseJava


CMake Error at CMakeLists.txt:30 (add_jar):
  Unknown CMake command "add_jar".


-- Configuring incomplete, errors occurred!

Hi,

I'm seeing some methods being unexpectedly listed high on perf top, which are called very infrequently. I have cross-checked with a JMC flight recording (which I assume to be quite accurate), and these paths were not showing up there at all.

Is there a chance that the map is somehow inaccurate, perhaps over time with hotspot recompiling / deoptimizing / moving stuff around?

Btw, this agent is quite useful, keep up the good work :)

Regards, Viktor

I am trying to build flamegraph, flamegraph is getting created successfully, but it is having a lot of methods without name / Generic symbol.

Out put which i get is below.

[aemauthor@local-aem62-test bin]$ ./perf-java-flames 17715 Recording events for 15 seconds (adapt by setting PERF_RECORD_SECONDS) [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.696 MB /tmp/perf-17715.data (618 samples) ] Failed to open 3E, continuing without symbols Flame graph SVG written to PERF_FLAME_OUTPUT='/home/aemauthor/perf-map-agent/bin/flamegraph-17715.svg'.

Perf data, method map and Flamegraph are attached.

Archive.zip

During our code profiling tests on 16-core systems running 8-16 worker threads, we are observing a significant CPU time (25%) in "vtable chunks" based on the generated symbol map. I think this symbol is listed multiple times as well in the symbol map. This overhead also increases as the number of running threads is gradually raised from 8 to 16.

Is this an overhead of perf-map-agent or something else ? Would like to also confirm if its related to virtual methods in our code.

Another unrelated issue is that perf top always displays "unable to open vsyscalls". Is there a way to resolve it. Does this mean that perf is unable to display system calls and their CPU time ? All processes are being executed by the "root" user.

I am facing this issue with trying to load agent into JVM when system process PID out of integer range:

/usr/lib/jvm/java-8-oracle/bin/java -cp attach-main.jar:/usr/lib/jvm/java-8-oracle/lib/tools.jar net.virtualvoid.perf.AttachOnce 115323
Exception in thread "main" java.io.IOException: Non-numeric value found - int expected
	at sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:299)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:63)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgentPath(HotSpotVirtualMachine.java:88)
	at net.virtualvoid.perf.AttachOnce.loadAgent(AttachOnce.java:51)
	at net.virtualvoid.perf.AttachOnce.main(AttachOnce.java:34)

Is there any other work-around than reboot Linux?

Generally speaking, is this "integer" PID restriction for hotspot VM attachment has been fixed in any recent Java 8 or OpenJDK version?

pi@raspberrypi:~/perf-map-agent $ make
[ 50%] Built target perfmap
[ 66%] Building Java objects for attach-main.jar
error: error reading /home/pi/jdk1.8.0_181/lib/ct.sym; cannot read zip file
Fatal Error: Unable to find package java.lang in classpath or bootclasspath
make[2]: *** [CMakeFiles/attach-main.dir/build.make:70: CMakeFiles/attach-main.dir/java_compiled_attach-main] Error 3
make[1]: *** [CMakeFiles/Makefile2:110: CMakeFiles/attach-main.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

The ct.sym file is there. Edit: The libperfmap.so is correctly compiled, it's the generation of the http://fem.rupy.se/attach-main.jar that fails, luckily I had it from a previous compile all it does is to attach libperfmap.so to the JVM. So I manage to get most things working but when I run sudo perf script > out.perf I get: Failed to open /home/pi/jdk1.8.0_181/jre/lib/rt.jar, continuing without symbols So the Java SE methods do not get replaced properly but you can guess what they are, the dumped ones however do: http://move.rupy.se/file/graph.svg

Ran ./bin/dtrace-java-record-stack but the command does not work. I think the dtrace command inside the ./bin/dtrace-java-record-stack is not used properly.

cat /tmp/dtrace-25840.data Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I]] -s File.d [-o ]

On a dtrace data file of 2342965 lines and a perf map file of 19072 lines, processing of dtrace-perf-map.pl took well over 5 hours.

Investigation demonstrated that this was because of the inner loop over the @map_entries array; reducing the length of that array to 10 would give a 5s runtime, 20 a 10s, 30 a 15s, etc. Perl is good for a lot of things but not for 4 billion array iterations.

It seemed to me that we could improve this by sorting the array and applying binary search (An extent map was also considered but thought too complex to build, though fun). Processing the same file now takes roughly 20s on the same machine (11s on mine).

Or, binary search go brrr.

Implemented Agent_OnLoad so that the agent may be loaded at startup. Previously the agent could only be attached at runtime.

The agent may be loaded using the following cmd line syntax: perf record java -XX:+PreserveFramePointer -agentpath:/path/to/perf-map-agent/out/libperfmap.so app args

This solves #42 by adding minimal support for JDK 9 and later versions. Issues addressed are:

• lack of tools.jar since JDK 9 (replaced by adding jdk.attach module)
• lack of javah since JDK 10 (now handled by javac, so no longer required)