GUI for comparing JMH results

David Karnok

Last update: Sep 24, 2021

Related tags

Profilers jmh-compare-gui

Overview

jmh-compare-gui

GUI for comparing JMH results

Requires Java 8.

Usage

Copy the output of JMH to the clipboard with the header
Click paste
Repeat with other JMH results

The program remembers all of its tabs, settings and contents on exit.

Download

Precompiled, runnable JAR

jmh-compare-gui-1.3.1.jar (SHA1: 2a7c06346ea57a812160a7d9cf76f0fb14d5a1e3)

Previous versions

jmh-compare-gui-1.3.0.jar (SHA1: f8aef4a1e3036566df155ec0f4df29df5379b24d)

jmh-compare-gui-1.2.0.jar (SHA1: 1e3e3983ae1f78cecbea9b956bcacbbbef31ac4c)

jmh-compare-gui-1.1.0.jar (SHA1: 6f184ea6bcd07570dfcda4c73e2f6cfbaf4d0a5a)

jmh-compare-gui-1.0.5.jar (SHA1: ae3a583812b5da14c6082d4905520a05bbf6b9f8)

jmh-compare-gui-1.0.jar (SHA1: 92b8563e6e478605f276b057617500a1c16a40dd)

Comments

Unsupported number format

I am trying to import the contents below into the GUI and I'm getting an Error dialog with "Unsupported number format"

Benchmark                   Mode   Cnt    Score    Error   Units
Test.slickDelete           thrpt     2    0.003           ops/ms
Test.slickInsertBatch      thrpt     2    0.040           ops/ms
Test.slickInsertSeparate   thrpt     2    0.023           ops/ms
Test.slickQuery            thrpt     2    0.005           ops/ms
Test.slickUpdate           thrpt     2    0.006           ops/ms
Test.srDelete              thrpt     2    1.069           ops/ms
Test.srInsertBatch         thrpt     2    0.389           ops/ms
Test.srInsertMapper        thrpt     2    0.012           ops/ms
Test.srInsertSeparate      thrpt     2    0.146           ops/ms
Test.srQueryConvert        thrpt     2    0.167           ops/ms
Test.srQueryMap            thrpt     2    0.156           ops/ms
Test.srQueryTo             thrpt     2    0.123           ops/ms
Test.srUpdate              thrpt     2    0.224           ops/ms
Test.slickDelete            avgt     2  212.773            ms/op
Test.slickInsertBatch       avgt     2   25.639            ms/op
Test.slickInsertSeparate    avgt     2   53.300            ms/op
Test.slickQuery             avgt     2  300.195            ms/op
Test.slickUpdate            avgt     2  262.242            ms/op
Test.srDelete               avgt     2    1.093            ms/op
Test.srInsertBatch          avgt     2    3.000            ms/op
Test.srInsertMapper         avgt     2   82.597            ms/op
Test.srInsertSeparate       avgt     2    7.986            ms/op
Test.srQueryConvert         avgt     2    6.729            ms/op
Test.srQueryMap             avgt     2    7.225            ms/op
Test.srQueryTo              avgt     2    7.901            ms/op
Test.srUpdate               avgt     2    5.162            ms/op
Test.slickDelete          sample     8  280.396 ± 83.529   ms/op
Test.slickInsertBatch     sample    78   26.102 ±  1.098   ms/op
Test.slickInsertSeparate  sample    38   54.234 ±  3.824   ms/op
Test.slickQuery           sample     9  272.834 ± 58.646   ms/op
Test.slickUpdate          sample     8  283.836 ± 66.506   ms/op
Test.srDelete             sample  1749    1.142 ±  0.020   ms/op
Test.srInsertBatch        sample   660    3.031 ±  0.055   ms/op
Test.srInsertMapper       sample    26   82.727 ±  1.391   ms/op
Test.srInsertSeparate     sample   250    8.016 ±  0.077   ms/op
Test.srQueryConvert       sample   295    6.792 ±  0.119   ms/op
Test.srQueryMap           sample   281    7.122 ±  0.077   ms/op
Test.srQueryTo            sample   247    8.096 ±  0.088   ms/op
Test.srUpdate             sample   375    5.350 ±  0.192   ms/op
Test.slickDelete              ss     2  249.530            ms/op
Test.slickInsertBatch         ss     2   25.078            ms/op
Test.slickInsertSeparate      ss     2   78.401            ms/op
Test.slickQuery               ss     2  274.850            ms/op
Test.slickUpdate              ss     2  261.700            ms/op
Test.srDelete                 ss     2    3.052            ms/op
Test.srInsertBatch            ss     2    3.093            ms/op
Test.srInsertMapper           ss     2   85.137            ms/op
Test.srInsertSeparate         ss     2    8.081            ms/op
Test.srQueryConvert           ss     2    6.811            ms/op
Test.srQueryMap               ss     2    7.371            ms/op
Test.srQueryTo                ss     2    8.095            ms/op
Test.srUpdate                 ss     2    5.161            ms/op

question

opened by darkfrog26 4

Pasting JMH clipboard output does not show correctly
Hi,

When I pasted the following JMH output to the GUI, the columns were not showing correctly.

So the text I copied and pasted was:

Benchmark Mode Cnt Score Error Units RxFlatmapVersusDeferBenchmark.helloWorld thrpt 200 2191821684.391 ± 37772457.512 ops/s

The resulting screen was:

Can you please suggest if I'm doing it incorrectly? Why are some columns not shown with a name and why the error column is showing "null". How can see an output similar to illustrated screen on Github project page.

My JMH version is: 1.17.3
opened by NiinS 2
JMH output with extra symbols between score and error
The following example fails to parse because there are extra symbols between the score and error in the rows. My guess is that JMH version > 0.9 has introduced this output format.

Benchmark (crossCL) (size) Mode Cnt Score Error Units Misaligned.read_aligned false 4096 avgt 25 8.755 Â± 0.086 us/op
enhancement
opened by akarnokd 1

Trim leading/trailing spaces or stars when parsing a JMH output

The following two examples fail to parse because of the leading spaces:

        Benchmark                   (crossCL)    (size)  Mode  Cnt      Score      Error  Units
        Misaligned.read_aligned         false      4096  avgt   25     8.755 Â±     0.086  us/op

or because it is part of a javadoc or comment:

    /*
     * Benchmark                   (crossCL)    (size)  Mode  Cnt       Score      Error  Units
     * Misaligned.read_aligned         false      4096  avgt   25      8.755 Â±     0.086  us/op
     */

enhancement

opened by akarnokd 1

One of the parameters as columns

Given the following example:

Benchmark                   (crossCL)    (size)  Mode  Cnt          Score      Error  Units

Misaligned.read_aligned         false      4096  avgt   25         8.755 Â±     0.086  us/op
Misaligned.read_aligned         false     65536  avgt   25       139.500 Â±     1.312  us/op
Misaligned.read_aligned         false   1048576  avgt   25      2349.098 Â±    35.448  us/op
Misaligned.read_aligned         false  16777216  avgt   25     91687.487 Â±   202.527  us/op

Misaligned.read_aligned          true      4096  avgt   25         8.728 Â±     0.056  us/op
Misaligned.read_aligned          true     65536  avgt   25       139.393 Â±     2.506  us/op
Misaligned.read_aligned          true   1048576  avgt   25      2328.423 Â±     7.654  us/op
Misaligned.read_aligned          true  16777216  avgt   25     91699.157 Â±   186.216  us/op

It would be great if (crossCL) value would be threated as a new value column whenever values are pasted in, so crossCL == true and crossCL == false could be compared against side-by-side.

enhancement feature request

opened by akarnokd 1

Incorporate error in gui?

I'm wondering what might be a good way to incorporate jmh error into the view. The score error field corresponds to half the 99.9% confidence interval. You could choose m standard deviations (some number less than or equal to 3.09 standard deviations = 99.9% confidence interval) and when mean1 in (mean2-m*sd2, mean2+m*sd2) or the converse then the view could overlay the red or green with some symbol to indicate that the level of uncertainty is high in the comparison. Any thoughts?
enhancement

opened by davidmoten 1

Releases(v1.3.2)

v1.3.2(Sep 19, 2016)

Added "Reverse colors" option to visually swap the coloring so no need to reconfigure the coloring just because in a benchmark, smaller means better (green):

Source code(tar.gz)
Source code(zip)
jmh-compare-gui-1.3.2.jar(3.75 MB)
v1.3.1(Jan 25, 2016)

Fixes a small crash when the comparison was active and a column was deleted.
Source code(tar.gz)
Source code(zip)
jmh-compare-gui-1.3.1.jar(3.77 MB)
v1.3.0(Oct 29, 2015)
This release has three notable improvements:

Take a screenshot of the table and put it onto the clipboard; no more manual print-screen and cutting the image around manually.

Management buttons now have icons instead of text; reduces the spanning of the toolbar and leaves room for further buttons.

Display the percentage difference relative to the baseline; comes in handy when quantifying improvements or regressions besides the high-level coloring.

Source code(tar.gz)
Source code(zip)
jmh-compare-gui-1.3.0.jar(3.75 MB)
v1.2.0(Aug 7, 2015)
Notable changes:

Column-related operations have been moved into a popup menu:

Tabs and columns can be easily duplicated now.

The column or tab name no longer turns into null if the user cancels the input dialog.

Added the ability to remove selection and focus from the table which makes it easier to take screenshots of it.

Fixed empty parameter values if the pivoted table didn't have data for every column in certain rows.

Added the ability to display the errors besides the measured values. They receive the same coloring as the measurement value when comparing against a baseline. This works with existing data.

Source code(tar.gz)
Source code(zip)
jmh-compare-gui-1.2.0.jar(3.74 MB)
v1.1.0(Apr 17, 2015)
Bugfixes

Fixed color options not saved properly on exit.

Export

The currently selected tab's contents can now be exported into a CSV or "XLS" format. Experience shows that CSV parsing is highly locale dependent, especially with Excel, therefore, the File > Export settings... option let's you specify the CSV column separator (usually comma or semicolon) and whether the floating point numbers should use dot or the current locale-specific decimal separator (i.e., comma).

The "XLS" format is not really a binary Excel file but rather a HTML file containing a single table (with colors if a comparison baseline column was selected). Even though some Excel versions will complain when opening the generated file, it should display properly.

Paste and pivot

This is a new feature that allows using one of the parameter columns of the JMH results as pivot columns, and this column's distinct values become the columns of the table.

For examle, if you benchmarked some code with a parameter that enables or disables certain behavior, you can pivot based on this parameter and compare the runs side-by-side.

Benchmark (times) Mode Samples Score Score error Units r.i.AtomicPerf.atomicIntCASCheckFailure 1 avgt 5 2,668 0,057 ns/op r.i.AtomicPerf.atomicIntCASCheckFailure 1000 avgt 5 1534,512 74,719 ns/op r.i.AtomicPerf.atomicIntCASCheckFailure 1000000 avgt 5 2284937,183 28254,968 ns/op r.i.AtomicPerf.atomicIntCASCheckSuccess 1 avgt 5 16,064 0,576 ns/op r.i.AtomicPerf.atomicIntCASCheckSuccess 1000 avgt 5 11408,954 213,337 ns/op r.i.AtomicPerf.atomicIntCASCheckSuccess 1000000 avgt 5 11417083,239 142717,458 ns/op

Note that this option clears the contents of the current tab.
Source code(tar.gz)
Source code(zip)
jmh-compare-gui-1.1.0.jar(3.60 MB)
v1.0.5(Mar 6, 2015)
This release contains parsing changes to accept more variety of JMH output formats, i.e., surrounded by spaces, embedded in comment sections or containing empty lines between values.

Related issues:

#1: skip empty lines

#4: trim leading/trailing spaces and comment symbols

#5: ignore additional text between score and error among the value lines.

Source code(tar.gz)
Source code(zip)
jmh-compare-gui-1.0.5.jar(3.59 MB)

Owner

David Karnok

GitHub

"Trust no one, bench everything." - sbt plugin for JMH (Java Microbenchmark Harness)

sbt-jmh SBT plugin for running OpenJDK JMH benchmarks. JMH about itself: JMH is a Java harness for building, running, and analysing nano/micro/milli/m

750 Dec 30, 2022

Log analyser / visualiser for Java HotSpot JIT compiler. Inspect inlining decisions, hot methods, bytecode, and assembly. View results in the JavaFX user interface.

JITWatch Log analyser and visualiser for the HotSpot JIT compiler. Video introduction to JITWatch video Slides from my LJC lightning talk on JITWatch

2.8k Jan 3, 2023

Log analyser / visualiser for Java HotSpot JIT compiler. Inspect inlining decisions, hot methods, bytecode, and assembly. View results in the JavaFX user interface.

JITWatch Log analyser and visualiser for the HotSpot JIT compiler. Video introduction to JITWatch video Slides from my LJC lightning talk on JITWatch

2.8k Jan 3, 2023

A simple fast search engine written in java with the help of the Collection API which takes in multiple queries and outputs results accordingly.

6 Oct 24, 2022

Trust-java - Test Results Verification library for Java

TRUST - Test Results Verification library for Java The TRUST's primary goal is to provide the simple way of different test results verification. Gener

2 Nov 19, 2017

Google Search Results JAVA API via SerpApi

Google Search Results JAVA API This Java package enables to scrape and parse Google, Bing and Baidu search results using SerpApi. Feel free to fork th

21 Dec 18, 2022

Java CMS engine. Host and develop multiple websites inside a single instance through the GUI and benefit from features like A/B testing, affiliate tracking tools, and a high performance template engine with CSS stylesheets processing & scripts minification.

Free enterprise Java CMS https://ncms.softmotions.com ηCMS features Multiple websites on a single ηCMS application node. Can handle hundreds of simult