Environment:

macOs Monterey 12.1 java -version openjdk version "11.0.9.1" 2020-11-04 LTS OpenJDK Runtime Environment Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS) OpenJDK 64-Bit Server VM Corretto-11.0.9.12.1 (build 11.0.9.1+12-LTS, mixed mode)

I tried to run the below command : jar -cfm Log4jHotPatch.jar Manifest.mf *.class

Got the error below:

java.io.FileNotFoundException: Manifest.mf (No such file or directory) at java.base/java.io.FileInputStream.open0(Native Method) at java.base/java.io.FileInputStream.open(FileInputStream.java:219) at java.base/java.io.FileInputStream.(FileInputStream.java:157) at java.base/java.io.FileInputStream.(FileInputStream.java:112) at jdk.jartool/sun.tools.jar.Main.run(Main.java:267) at jdk.jartool/sun.tools.jar.Main.main(Main.java:1681)

Am i missing something.

When trying to run

java -cp <java-home>/lib/tools.jar:Log4jHotPatch.jar <java-pid>

with JDK 8, I only get the message

Error: Could not find or load main class <PID>

what could i be doing wrong?

We are seeing logging to to stdout which affects parsing of our outputs.

I have tried:

   <sysproperty key="log4jFixerVerbose" value="false" escape="false"/>

However, we are still getting the log messages like:

Loading Java Agent version 1 (using ASM6).
Patching class org.apache.logging.log4j.core.lookup.JndiLookup (com.amazon.cloud9.launcher.BootstrapClassLoader@29444d75)
Transforming org/apache/logging/log4j/core/lookup/JndiLookup (com.amazon.cloud9.launcher.BootstrapClassLoader@29444d75)

Any idea why the sysproperty might be ignored? Thanks!

Ensures any log messages from the hot patch are separate from any user program output that may be being used elsewhere.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Issue #, if available:

Description of changes:

This change allows this project to be built using Maven. Project build/config based on the build.gradle file.

Tested using

build-tools/bin/run_tests.sh target/Log4jHotPatch.jar ~/amazon-corretto-8.282.08.1-linux-x64
build-tools/bin/run_tests.sh target/Log4jHotPatch.jar ~/amazon-corretto-17.0.1.12.1-linux-x64

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This patch modifies AccessController to temporarily ignore security manager checks which can fail in very restricted environments, as when setting properties. It also avoids reflection.

This is a POC and work in progress! I haven't updated the file layout so the build and tests didn't had to be changed as well. All the tests except the security manager test pass (and I hope we can make the latter pass as well).

In general, this PR should be merged with the file layout and refactoring changes of #39.

The ideas behind this change is as follows:

• Divide the agent into a bare minimum agent part, a "payload" part (i.e. a "patcher" or "transformer") and a driver which injects the agent together with a "patcher" into the target VM.
• The agent uses a custom class loader to load the payload (i.e. the transformers) which implements the Patcher interface.
• The agent manages the transformers through the general Patcher interface methods install(),uninstall() and getVersion()
• calling install() on a Patcher will register all its transformers and retransform the affected classes in the dynamic use case
• calling uninstall()on a Patcher will unregister its transformers and retransform the previously transformed classes back to their initial state.

The current change contains three patchers, EmptyPatcher, PatcherV1 and PatcherV2 which all implement Patcher. EmptyPatcher is special in that it contains no functionality and has version number "0". PatcherV1 and PatcherV2 both implement the current Log4j patch with version numbers "1" and "2" respectively.

The main driver (i.e. Log4jHotPatch.main()) has a hard-coded version of the patcher it will deploy. This is currently PatcherV1 for testing reason. In reality this should be the latest patcher (i.e. the one with the highest version number). The driver attaches the target JVM and checks for the version of the transformer it already has. If there's no transfomer installed or if the transformer in the target JVM is smaller than the current transformer in the driver, the driver will deploy its transformer into the target. In the target this will trigger uninstallation of the old transformer and installation of the new one.

EmptyPatcher (i.e. version "0") is treated special in the driver. It can be used to overwrite any installed transformer in the target, effectively undoing all previous transformations. This can be used to "downgrade" the transformer e.g. by going from "2" to "0" to "1".

Apart from deploying the "default" transformer, the driver can be configured to deploy any available transformer by specifying its name with the patcherClassName property on the command line (e.g. -D'patcherClassName=Log4jHotPatch$PatcherV1').

Because all references to the custom class loaders and transformer are deleted in the agent once a new transformer will be installed, the old ones can be collected and unloaded by the JVM.

This change is split in three commits to help with the reviewing process, but should be squashed before merging

This change splits the code for the project into multiple java files, decoupling them:

• We have one class that represents the agent (keeping the premain and agentmain methods from the original patch
• One class to start the jar and attach to the vm (keeping the main method and the methods it calls
• One class that represents the actual bytecode transformation for the patch as Log4j2NoJndiLookup
• Some other utility classes

With this change, it will become less dangerous to import other patches that we might want the tool to support.

The command gives me the following error:

# ./gradlew build --warning-mode all

> Configure project :
The AbstractArchiveTask.version property has been deprecated. This is scheduled to be removed in Gradle 8.0. Please use the archiveVersion property instead. See https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:version for more details.
        at build_34dfj8owb22lztkrt6bemzzq5$_run_closure2.doCall(/root/hotpatch-for-apache-log4j2/build.gradle:17)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)
The AbstractArchiveTask.archiveName property has been deprecated. This is scheduled to be removed in Gradle 8.0. Please use the archiveFileName property instead. See https://docs.gradle.org/7.1/dsl/org.gradle.api.tasks.bundling.AbstractArchiveTask.html#org.gradle.api.tasks.bundling.AbstractArchiveTask:archiveName for more details.
        at build_34dfj8owb22lztkrt6bemzzq5$_run_closure3.doCall(/root/hotpatch-for-apache-log4j2/build.gradle:34)
        (Run with --stacktrace to get the full stack trace of this deprecation warning.)

FAILURE: Build failed with an exception.

* Where:
Build file '/root/hotpatch-for-apache-log4j2/build.gradle' line: 46

* What went wrong:
A problem occurred evaluating root project 'hotpatch-for-apache-log4j2'.
> java.lang.NullPointerException (no error message)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s

Should I add debug output as well?

Apache Oozie job intermittently fails after applying the hotpatch. This is because Oozie will list and the print the file and directory name in the current directory. On the other hand, the hotpatch creates attach_pid file in the current directory to attach the JVM and then delete it. When the attach_pid file is deleted between the listing and the printing, the Oozie job fails by NoSuchFileException. The below is the stacktrace:

java.nio.file.NoSuchFileException: ./.attach_pid1517
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
    at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.nio.file.Files.readAttributes(Files.java:1737)
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
    at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
    at java.nio.file.Files.walkFileTree(Files.java:2706)
    at org.apache.oozie.action.hadoop.LocalFsOperations.printContentsOfDir(LocalFsOperations.java:59)
    at org.apache.oozie.action.hadoop.LauncherAM.printDebugInfo(LauncherAM.java:293)
    at org.apache.oozie.action.hadoop.LauncherAM.access$100(LauncherAM.java:55)
    at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:221)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
    at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
    at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)

The mitigation is to disable the hotpatch on the Oozie server.

I plan to provide a fix to Oozie to retry the directory listing, however, it is worth creating the issue here for searchability. Thanks.

Steps to reproduce :

If you try to install any one of these procps-ng/net-tools/iputils before installing corretto, then log4j hotpatch will report following error while installation, however installation completed. Dockerfile :

FROM amazonlinux:2 
RUN yum update -y
RUN yum install -y procps-ng net-tools iputils
RUN amazon-linux-extras enable corretto8
RUN yum install -y java-1.8.0-amazon-corretto-devel

$ docker build -t log4j .

Installation of log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch reports following error :

Failed to get D-Bus connection: Operation not permitted
Created symlink /etc/systemd/system/multi-user.target.wants/log4j-cve-2021-44228-hotpatch.service, pointing to /usr/lib/systemd/system/log4j-cve-2021-44228-hotpatch.service.
Failed to get D-Bus connection: Operation not permitted
warning: %post(log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package log4j-cve-2021-44228-hotpatch-1.1-12.amzn2.noarch

As a workaround, you can try installing procps-ng/net-tools/iputils these after corretto.

The hot patch created a lot of /tmp mount points. Since my customer collects metrics on mount points, these /tmp mount points cost customer a lot on these unwanted metrics. Could these mount points be disabled ?

Note: All instructions here assume Java 11 (OpenJDK), but I was also able to reproduce the issue with Java 8 (OpenJDK).

Steps to reproduce

• Download an official release of the Jenkins generic Java package (.war).
• Start Jenkins with java -jar jenkins.war.
• Build Log4jHotPatch and run java -Dlog4jFixerVerbose=true -cp Log4jHotPatch.jar Log4jHotPatch ${JENKINS_PID}.

Expected results

Note: These are the actual results with a local (non-official) build of Jenkins:

The Log4jHotPatch agent is attached successfully.

Actual results

The Log4jHotPatch agent cannot be attached:

com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize
	at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:165)
	at Log4jHotPatch.loadInstrumentationAgent(Log4jHotPatch.java:228)
	at Log4jHotPatch.main(Log4jHotPatch.java:301)
Error: couldn't loaded the agent into JVM process 546677

The Jenkins logs show:

Exception in thread "Attach Listener" java.lang.SecurityException: class "Log4jHotPatch"'s signer information does not match signer information of other classes in the same package
	at java.base/java.lang.ClassLoader.checkCerts(ClassLoader.java:1151)
	at java.base/java.lang.ClassLoader.preDefineClass(ClassLoader.java:906)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1015)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:431)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Agent failed to start!

Evaluation

Stepping through the problematic frames in the debugger, I do not think Jenkins is at fault here. Jenkins contains some classes in the unnamed package namespace:

$ jar tf jenkins.war
[…]
ColorFormatter.class
JNLPMain.class
LogFileOutputStream$1.class
LogFileOutputStream.class
Main$FileAndDescription.class
Main.class
MainDialog.class
[…]
META-INF/JENKINS.RSA
META-INF/JENKINS.SF
META-INF/MANIFEST.MF
[…]
$

Note that META-INF/JENKINS.RSA and META-INF/JENKINS.SF are only present in our signed official releases; a local build will not contain these. The presence of these classes and signatures means that the system class loader associates the signatures with the unnamed package namespace. Then, when Log4jHotPatch attempts to load a class into the same unnamed package namespace without a signature, class loading fails.

I do not think Log4jHotPatch should make any assumptions about whether or not it is free to load unsigned classes into the unnamed package namespace. Perhaps Log4jHotPatch should use its own package namespace instead.

With CVE-2021-45105 a malicious user can cause a DoS which in most scenarios will lead to JVM restart.

After restart there's a time window when an attack against CVE-2021-45046 or CVE-2021-44228 could be launched before the agent has attached to the JVM process.

Would it be possible to extend this solution to cover this scenario as well?

Although -DformatMsgNoLookups=true prevents lookups directly in the message, Format Lookups are possible when reading a property from the ThreadContext/MDC in the pattern for the message. On certain scenarios, this can cause a StackOverflow through recursive lookups as described on CVE-2021-45105.

This patch disables lookups in Message Pattern by patching LiteralPatternConverter.

The patch for LiteralPatternConverter is not enabled by default and can be enabled using the following parameters

patcherClassName=com.amazon.corretto.hotpatch.patch.impl.set.Log4j2PatchSetWithDisableLookups

Hi, The patch has been deployed in AL1 on one of our server via "yum update --security", and we had a problem with it. We have the following error in the tomcat log avoiding any web applications to start on this server :

java.io.FileNotFoundException: /tmp/agent693101429317028124.jar (Aucun fichier ou dossier de ce type) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.(ZipFile.java:228) at java.util.zip.ZipFile.(ZipFile.java:157) at java.util.jar.JarFile.(JarFile.java:169) at java.util.jar.JarFile.(JarFile.java:106) at sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:93) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getJarFile(JarURLConnection.java:89) at org.apache.tomcat.util.scan.FileUrlJar.(FileUrlJar.java:48) at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:34) at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1957) at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1932) at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1917) at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1322) at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:878) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:388) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5566) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1017) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:993) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

In fact, this server has a specifiticy : there are 20 concurrent JVM / Tomcat instances running and launched via chkconfig services... I'm not sure but this could be the root cause of the pb... We applied the "sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill" to deactivate the HotPatch, and it worked. So it confirms that the issue is due to the Log4j HotPatch. I think you should publish a fix for this because I assume we won't be the only impacted customers.

A precision : we use amazon-corretto-8.302.08.1-linux-x64 under AL1

Best Regards.