Vitals June 22 update

This is a larger patch collection. Contains several new features, fixes and cleanups which have accumulated upstream in my dev branch.

(Note: a documentation for the Vitals now exists at https://github.com/SAP/SapMachine/wiki/SapMachine-Vitals).

High Memory Reports

SapMachine now has "high memory reports". Briefly, high memory reports are an early warning system for impending memory pressure. Too large to describe it here; please see https://github.com/SAP/SapMachine/wiki/SapMachine-High-Memory-Reports for more detail.

Implementation lives in vitals_linux_himemreport.cpp (its Linux only for now). Note that this feature, albeit running under "Vitals", has little to do with the Vitals proper. It uses an own monitoring thread. The only common code is the proc fs parsing code on Linux.

Container support

Vitals now shows cgroups relevant information if the VM is containerized (more correclty, if it runs inside a memory cgroup with limits, so that may include systemd-based physical machines). It shows memory usage, memory limit, soft limit, kernel memory usage. It works for both v1 and v2 cgroups.

Debated with myself whether to include memory+swap usage and limit, but decided against it, since it rarely is used and its meaning differs between cgroups v1 and v2.

Example output:

...
      cgroup-lim: cgroup memory limit [cgrp]
     cgroup-slim: cgroup memory soft limit [cgrp]
      cgroup-usg: cgroup memory usage [cgrp]
     cgroup-kusg: cgroup kernel memory usage (cgroup v1 only) [cgrp]
...
---------------------------------system---------------------------------
                                      -----cpu------ ------cgroup-------
avail comm  crt swap si so p t  tr tb us sy id st gu lim  slim usg  kusg  
54.3g 19.3g  57   0k  0  0 2 22  1  0  1  0 99  0  0 8.0g      120m   2m
54.3g 19.3g  57   0k  0  0 2 22  1  0  1  0 99  0  0 8.0g      120m   2m 
54.3g 19.3g  57   0k  0  0 2 22  1  0  1  0 99  0  0 8.0g      120m   2m
54.3g 19.3g  57   0k  0  0 2 22  3  0  1  0 99  0  0 8.0g      119m   2m
54.3g 19.3g  57   0k       2 17  2  0                8.0g       65m   2m

                                                      ^     ^    ^     ^

cgroup support was tested and works for both cgroup v1 and v2, under Docker, bare metal, and in artificially created cgroups.

NMT value printout

Vitals now shows more NMT values (arbitrarily chosen among the many NMT categories because they came up as questions in support cases with Andreas):

• "gc" : GC overhead that is not heap
• "oth" : aka "other" (name of the NMT category): Outside memory, typically DirectByteBuffer memory
• "ovh" : Overhead of NMT itself (can be significant)

Example:

...
         nmt-mlc: Memory malloced by hotspot [nmt]
         nmt-map: Memory mapped by hotspot [nmt]
          nmt-gc: NMT "gc" (GC-overhead, malloc and mmap) [nmt]
         nmt-oth: NMT "other" (typically DBB or Unsafe.allocateMemory) [nmt]
         nmt-ovh: NMT overhead [nmt]
...
-----------------------------------------------jvm------------------------------------------------
--heap--- ----------meta----------      ---------nmt--------- -----jthr----- --cldg-- ----cls-----
comm used comm used csc  csu  gctr code mlc map  gc  oth ovh  num nd cr st   num anon num  ld  uld 
130m   7m   3m   3m 320k 229k  21m   8m 43m 193m 75m  2k 992k  12  1  0 952k  34   31 1341   0   0 
130m   7m   3m   3m 320k 229k  21m   8m 43m 193m 75m  2k 992k  12  1  0 952k  34   31 1341   0   0 
130m   7m   3m   3m 320k 229k  21m   8m 43m 193m 75m  2k 992k  12  1  1 952k  34   31 1341   0   0 
130m   7m   3m   3m 320k 229k  21m   8m 43m 193m 75m  2k 992k  12  1  5 852k  34   31 1341 862   0 
130m   2m 128k  54k  64k  <1k  21m   7m 41m 189m 75m  0k 223k   9  1    612k   3    0  479         
                                        
                                                 ^    ^  ^              

Fixed display of empty columns in Vitals table

Many columns depend on context. E.g. NMT values are shown only if NMT is on, many system values only if the kernel supports them etc. It was confusing for users, since these columns were omitted ("where are my NMT values?"). OTOH, displaying all columns and leaving them empty is a waste of horizontal space.

Vitals now omit columns that have no values, but will still show them in the legend. Additionally, legend clearly marks which columns are context dependend.

Example:

          cpu-us: CPU user time [host]
          cpu-sy: CPU system time [host]
          cpu-id: CPU idle time [host]
          cpu-st: CPU time stolen [host]
          cpu-gu: CPU time spent on guest [host]
      cgroup-lim: cgroup memory limit [cgrp]
     cgroup-slim: cgroup memory soft limit [cgrp]
      cgroup-usg: cgroup memory usage [cgrp]
     cgroup-kusg: cgroup kernel memory usage (cgroup v1 only) [cgrp]
-----------process------------
            virt: Virtual size
         rss-all: Resident set size, total
        rss-anon: Resident set size, anonymous memory [krn]
        rss-file: Resident set size, file mappings [krn]
         rss-shm: Resident set size, shared memory [krn]
            swdo: Memory swapped out
       cheap-usd: C-Heap, in-use allocations (may be unavailable if RSS > 4G) [glibc]
      cheap-free: C-Heap, bytes in free blocks (may be unavailable if RSS > 4G) [glibc]
...
   [host]: values are host-global (not containerized).
   [cgrp]: if containerized or running in systemd slice
    [krn]: depends on kernel version
   [glibc]: only shown for glibc-based distros
  [delta]: values refer to the previous measurement.
    [nmt]: only shown if NMT is available and activated
     [cs]: only shown on 64-bit if class space is active
  [linux]: only on Linux

Expanded regression tests

Regression tests have been expanded again, a lot this time. Specifically, we now have VitalsValuesSanityCheck.java, a test that does sanity checks on all values Vitals prints out. There is a lot of heuristics in there.

The high memory report feature comes with its own set of regression tests (TestHiMemReportXXX.java).

Code restructuring

I tried to avoid core refactoring, but some was needed. In particular, on linux, proc fs parsing moved from vitals_linux.cpp to vitals_oswrapper.cpp, into an own buffering helper class. Needed because that layer is now used by the Vitals proper as well as the high memory sampler thread.

Bug Fixes:

• Fixed a bug that causes VM to crash if we shutdown NMT early (#1148)
• Fixed names, legend of "processes running" and "processes blocked" columns. These values are really (kernel) threads, not processes.
• Fixed errors on newer glibc's (>2.24) where mallinfo2 exists and mallinfo is marked as obsolete (we now resolve both functions dynamically).
• Fixed display of C-heap used memory for 32-bit platforms

Other things:

• Vitals are now versionized themselves.
• Vitals now use UL logging, consistently, during initialization (-Xlog:vitals).

[1] https://github.com/SAP/SapMachine/wiki/SapMachine-High-Memory-Reports [2] Vitals Documentation

fixes #1148 fixes #1124

Summary: This is the implementation of JEP 358: Helpful NullPointerExceptions. Reviewed-by: coleenp, clanger, rschmelter, rriggs, forax, mr

(cherry picked from commit dfdf8d16151bd27aa288b7f7c981e70651350ad8)

The description of this pull request goes here.

fixes #528

Hi, may I please have reviews for this change. Details and motivation see https://github.com/SAP/SapMachine/issues/861.

The patch:

• introduces ExitVMOnOutOfMemoryError as an alias to CrashOnOutOfMemoryError
• adds stack dumping to tty for both
• switches off cores if either one is set, unless cores had been explicitly requested by the user
• Wires all xxOnOutOfMemoryError switches up to thread creation too - all these will fire when we cannot start native threads (note that OnOutOfMemoryError=command probably won't work since it requires the ability to fork). Note that I decided to wire this up after resource-exhausted-JVMTI-hook is called; does that make sense or should I do it before? to me its a coin toss, really.)
• Adds a new jtreg test to test SapMachine specific logic
• Also adds stack dumping to the otherwise unchanged ExitOnOutOfMemoryError

Testing:

• ran the jtreg test
• ran hotspot:tier1 locally
• tested manually all these switches both with OOM from Metaspace and OOMs from thread creation; the latter with ulimit -u set. Works. We now print thread stack to tty, then assert, but without core:

thomas@starfish$ ulimit -u 3000
thomas@starfish$ ./images/jdk/bin/java -XX:+CrashOnOutOfMemoryError de.stuefe.repros.MultiThreadTest --num-threads=3000 --wait-time=100
Will start 3000 threads, wait time 100s...
<press key>
 ... continuing.
Created: 300...
Created: 600...
Created: 900...
Created: 1200...
Created: 1500...
Created: 1800...
Created: 2100...
[1,119s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
Aborting due to java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 372  java.lang.Thread.start0()V java.base@17-internal (0 bytes) @ 0x00007fe389087ac4 [0x00007fe389087a60+0x0000000000000064]
J 368 c1 java.lang.Thread.start()V java.base@17-internal (71 bytes) @ 0x00007fe381cd0934 [0x00007fe381cd07e0+0x0000000000000154]
j  de.stuefe.repros.MultiThreadTest.call()Ljava/lang/Integer;+93
j  de.stuefe.repros.MultiThreadTest.call()Ljava/lang/Object;+1
j  picocli.CommandLine.executeUserObject(Lpicocli/CommandLine;Ljava/util/List;)Ljava/util/List;+94
j  picocli.CommandLine.access$900(Lpicocli/CommandLine;Ljava/util/List;)Ljava/util/List;+2
j  picocli.CommandLine$RunLast.handle(Lpicocli/CommandLine$ParseResult;)Ljava/util/List;+29
j  picocli.CommandLine$RunLast.handle(Lpicocli/CommandLine$ParseResult;)Ljava/lang/Object;+2
j  picocli.CommandLine$AbstractParseResultHandler.execute(Lpicocli/CommandLine$ParseResult;)I+16
j  picocli.CommandLine.execute([Ljava/lang/String;)I+31
j  de.stuefe.repros.MultiThreadTest.main([Ljava/lang/String;)V+15
v  ~StubRoutines::call_stub
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/debug.cpp:366
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/shared/projects/openjdk/sapmachine/source/src/hotspot/share/utilities/debug.cpp:366), pid=441918, tid=441919
#  fatal error: OutOfMemory encountered: unable to create native thread: possibly out of memory or process/resource limits reached
#
# JRE version: OpenJDK Runtime Environment (17.0) (fastdebug build 17-internal+0-adhoc.thomas.source)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 17-internal+0-adhoc.thomas.source, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# CreateCoredumpOnCrash turned off, no core file dumped
#
# An error report file with more information is saved as:
# /shared/projects/openjdk/sapmachine/output-fastdebug/hs_err_pid441918.log

fixes #861

Hi team,

sapmachine-11-jre package is no longer in "http://dist.sapmachine.io/debian/amd64 ./" repo. Could you please assist?

it is used within "https://github.com/cloudfoundry/apt-buildpack" for Cloud Foundry db deployment.

Best regards, Egor

This is a collection of changes to the Vitals. Changes have accumulated, and its time to bring them to the sapmachine. The biggest changes are in vitals/vitals.cpp, formerly services/stathist.cpp. Review-wise it may be easier just to look at that file in its current form (specifically, all classes starting with "Sample") instead of parsing the diff, but I leave this to you.

Changes in detail:

• consistently named the Vitals Vitals, removing all traces of the former "stathist" or "StatisticalHistory"
• moved the code into an own sub directory (/share/vitals)
• On Linux, added a new column, named "hp" for "process heap", which contains the sum of rss+swap for the process heap (I included swap in case we are swapping and rss is not the full story). The data are read from proc/pid/smaps.
• "productified" most Vitals options, removing the Diagnostic marker.
• Added a new option to print vitals to stdout at VM exit (we had already "DumpVitalsAtExit", same thing different sink)
• now there are some (basic) gtests for vitals
• and a new jtreg stress test printing vitals in quick succession

vitals.cpp:

• I now use a Mutex to guard table update and table printing.
• Where before the downsample tables were cascaded from the first table (so, samples falling out of the first FIFO were fed into the subsequent mid term table, and falling out of that one into the long term table), now the tables are independent and are fed the same samples, just with different periods. So, the short term table gets every sample, the mid term table every nth sample, etc. The effect is that each table can be looked at individually, without having to know the other tables. And that the time range the downsample tables cover starts now, not at now-x.
• Iterating over the sample tables was streamlined and simplified. Walking the table is done with a closure interface now.

Sorry for the big change. I hope after this change things become reasonably stable.

Testing: I will let the patch cook in our nightlies for a while.

fixes #820

TL;DR: Will there be an ARM version?

I (and I'm sure others) would be interested to know if an ARM version is planned? Apple is currently switching to ARM. So many developers will soon be using ARM. I also see more and more Macbooks among SAP employees.

I would appreciate a short info and maybe more details about the SAP strategy.

Thank you 👍

I also posted the question on the SAP Platform for such questions: https://answers.sap.com/questions/13107725/sap-and-arm-cpu-platform-aarch64-arm64-armv8-apple.html

This is the first implementation of the dt_filesocket transport. It still resides in the jdk.jdwp.agent module and has not yet the best possible behaviour in case of errors (when the pipe/domain socket cannot be created).

fixes #207

Hey there,

we recently updated our SapMachine from 11.0.14 to 11.0.16. We use the latest Elastic APM Java Agent with the profiling feature (it uses the async-profiler project). After the update the following warnings occurr frequently in the logs:

WARNING: Could not find Java_java_lang_ClassLoader_00024NativeLibrary_load

It seems like the issue came with changes in openJDK11. See these post for reference:

• https://stackoverflow.com/questions/72178216/java-error-java-java-lang-classloader-00024nativelibrary-load0
• https://github.com/openjdk/jdk11u/commit/f7346f087a14f1878029a00104005e04468d333f#diff-c106c06bd7c092b3b91cb0487c089c1e96d33305c6bf599e2082a709271e3948R338

Do you plan to fix this issue?

Kind regards.

Hello! I have problem at initialisation on my local. My specs : MacBook Pro (2021) with Apple M1 Max. This problem doesn't occur on Windows. I had similar problem for jdk11 wit build 11.0.16+8-LTS-sapmachine after i switch on build 11.0.15.0.1+10-LTS the problem was fixed. Now I have it again for jdk17. I tried different build versions but the result is the same. The issue is available only with integration pack added. I will attach the error log.

Thank you! Andrei Fere logerrorjdk17.txt .

I was using the

SAP GUI for Java 7.70 rev 3 (Version ID 077000040300) 2021-08-05 08:57:00 +0200 gclm0108, 760_REL, 2082961 Java VM: SAP SE Version 11.0.12+7-LTS-sapmachine OS: Mac OS X(x86_64) Version 12.0.1

The SAP GUI Always exit unexpectedly, I don't know why. And the error log is showing a fatal error has been detected by the Java Runtime Environment:(Here is the error log)

hs_err_pid12819.log hs_err_pid16938.log

Please if anyone can tell me how to fix the bug!

Thank you so much.

settting the TZ variable e.g. to Europe/Berlin or Europe/Oslo has no effect, this might be because of all timezone files in /usr/share/timezone are missing

I have tried using both the DMG and the tar.gz MacOS x64 binaries and they all report as corrupted.

The exact message is:

“sapmachine-jdk-20.jdk” is damaged and can’t be opened. You should move it to the Trash.

Much older 20ea builds were working fine before. I was able to run sapmachine-20+15 and earlier without issue. Starting with sapmachine-20+16 up to sapmachine-20+24, I see the binaries as 'damaged' every time.

Note: I am using MacOS Monterey version 12.6.1

Please provide support for the command update-java-alternatives in SapMachine Docker images.

This would allow for switching between different installed JDKs easily with a single command. This helps especially in cases where the default OpenJDK gets installed though a package dependency and overrides SapMachine.

For this a .jinfo file must be provided, similarly as found for example in openjdk-11-jre-headless.

Very rare. In some environments, the automatic limit heuristic seems to fail and we get

[0.154s][warning][vitals] Vitals HiMemReport: limit could not be established; will disable high memory reports (specify -XX:HiMemReportMax=<size> to establish a manual limit).

Seen two cases: http://test.sapmachine.only.sap:1080/servlet/testresult?testid=245774322 http://test.sapmachine.only.sap:1080/servlet/testresult?testid=245796780

on linux aarch64 and ppcle.

Hi,

can you please implement that the SapMachine installer sets the following registry values?

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\JavaSoft]

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\JavaSoft\Java Runtime Environment]
"CurrentVersion"="11.0"

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\JavaSoft\Java Runtime Environment\11.0]
"JavaHome"="C:\\Program Files\\SapMachine\\JDK\\11"
"MicroVersion"="0"

Further Information: https://github.com/kaikramer/keystore-explorer/issues/292

Thank you in advance.