This is the very first draft of my work on contributing log4j2-logstash-layout into Log4j core. Please note that this is still a work in progress. I want to have your feedback and support on the following issues which I will share below.

Goals

Below is the list of goals I plan to deliver in order:

• [X] Update manual
• [X] Replace Uris with LoaderUtils
• [X] Add directive for "nanos of epoch millis", "millis of epoch seconds", etc.
• [X] Move pattern, timeZone, locale parameters into timestamp key level.
• [ ] Add benchmark to log4j-perf
• [X] Rename log4j-layout-jackson-json-template to log4j-layout-json-template
• [X] Replace TLAs with a user-provided 3-way caching strategy

I will update this list upon progress.

Module and package name

I chose JsonTemplateLayout as the new module name, which clearly communicates the intent. I am not totally convinced with the package name o.a.l.log4j.jackson.json.template.layout though. I have my doubts about whether we can remove Jackson dependency in the future or not. This ambition does look neither feasible, nor practical in the short-term. That said, from an API perspective, why shall an internal detail like jackson get exposed in the package name? I am considering to place it under o.a.l.log4j.layout.json.template. Any ideas?

Documentation

LogstashLayout has quite some configuration knobs which renders it inconvenient to explain it in /src/site/asciidoc/manual/layouts.adoc. I have skimmed through the source code with the hope of finding a similar documentation pattern, but failed to do so. Would somebody mind shedding some light onto the convention I should follow here, please?

MapMessage#getFormattedMessage() JSON encoding bug

As reported in LOG4J2-2703, MapMessage#getFormattedMessage() is incorrectly formatting Objects for JSON. As a workaround in LogstashLayout, I had a LogstashLayout.Builder#mapMessageFormatterIgnored property, which I have dragged into JsonTemplateLayout.Builder too. The property instructs the layout to ignore MapMessage#getFormattedMessage() and do its own magic to properly encode the MapMessage into JSON. Shall I keep this ad-hoc fix or rather resolve LOG4J2-2703 first?

Performance tests

I have some test utility classes that I want to leverage in performance tests, but test classpath of log4j-jackson-json-template module is not accessible by log4j-perf module. What is the work around here?

What to do with JsonLayout?

I do not intend to implement JsonLayout by leveraging JsonTemplateLayout, at least, not now. I would rather just leave it be and get deprecated eventually. Put another way, is this PR the right place to address this issue?

Missing features compared to JsonLayout

You might have noticed that JsonTemplateLayout misses certain features (e.g., compact, complete, header, footer, etc.) compared to JsonLayout. This is a deliberate decision of mine. If you have any objections, please chime in.

Thread locals

Even though the existing TLA usage of LogstashLayout is dragged in, I am going to replace it with an object pool. I have spotted ThreadLocalVsPoolBenchmark and will start from there.

I've hard coded the limit to 10. If you feel the limit needs to be configurable (via system property or the log4j config?), let me know and I'll adjust.

I couldn't find a good way to keep the call in line 987 to the protected method rather than the private method, while also keeping track of the recursion count. I'm not sure if that call is relied on by subclasses, I don't think the Javadoc promises a call to that method for each variable resolved.

A pull request for https://issues.apache.org/jira/browse/LOG4J2-928.

This PR is not complete - it lacks tests and benchmarks (I'm going to add them in the next few weeks), but the core implementation is reviewable already.

Proposed fix for LOG4J2-2610 rollover issue:

• Fix for Windows Filesystem Tunneling issue, rapid rollover issue.
• Added FileUtils.updateFileCreationTime() method.
• Updated FileManager and RollingFileManager to allow the file creationTime attribute to be updated on rollover (when it doesn't match initialTime).
• RollingAppenderSizeWithTimeTest.testAppender() was failing on Windows before any changes. Rapid log rollover was causing overwrite of data in some cases (instead of writing to a new log file), resulting in lost logging messages. Mitigation was to apply a 1ms sleep in RollingFileManager.createFileAfterRollover().

Note: This is not ready PR, there are no unit tests for new feature - please review before i'll invest more time into it.

• Changed AbstractJacksonLayout.convertMutableToLog4jEvent to protected Object wrapLogEvent(LogEvent event)
• Added Extras as KeyValuePair[] to JsonLayout
• When extras as specified, LogEvent is wrapped in new LogEventWithAdditionalFields class, which merges LogEvent serialized data with provided extras

This is generally wanted feature, for example see here: https://github.com/ggrandes/log4j2-simplejson https://github.com/majikthys/log4j2-logstash-jsonevent-layout

How it works:

<JsonLayout>
  <KeyValuePair key="hostName" value="${hostName}"/>
</JsonLayout>

Inserts into each generated JSON hostName key:

{
"timeMillis":1505228896690,
"thread":"main",
"level":"INFO",
"loggerName":"org.springframework....",
"message":"Refreshing ...",
"endOfBatch":false,
"loggerFqcn":"org....",
"contextMap":{},
"threadId":1,
"threadPriority":5,
"hostName":"wp21257b"
}

This is accomplished by tricks with Jackon json serializer:

public static class LogEventWithAdditionalFields {
    // ....
    @JsonUnwrapped // Deserializes actual LogEvent into root object
    public Object getLogEvent() { return logEvent; }

    @JsonAnyGetter // Adds everything from the map into root object
    public Map<String, Object> getAdditionalFields() { return additionalFields; }
}

Add a new LuceneAppender which writes logging events to a lucene index library.The log4j2.xml configuration is as follows: <Lucene name="lucene" ignoreExceptions="true" target="/target/lucene/index" expiryTime=“1296000”> <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n"/> <IndexField name="time" pattern="%d{UNIX_MILLIS}" type="LongField"/> <IndexField name="level" pattern="%-5level" /> <IndexField name="content" pattern="%d{HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n"/> </Lucene>

this appender relies on the Lucene 5.5.4 version. this patch adds the corresponding test cases.

The current default forces Win platforms to install an obscure jansi native library (for pretty print console messages) or always set this property to true, even for trivial unit tests.

Hello all,

please accept this small patch from a long time user of log4j: it will enable using log4j 2.x in several applications here at @six-group.com!

This proposal allows creating detached Markers as specified by the IMarkerFactory interface. A detached marker allows the caller to inject non-constant data that can be used to implement structured logging (my use-case). Because of this, they cannot be cached in Log4jMarkerFactory.markerMap or converted.

Please let me know if you require additional modifications (e.g. unit tests perhaps?)

This change set aims to replace statically bound JSON template resolver factories with plugins. Along this pursuit, following side kicks are introduced:

• Added support for inline stack trace element templates in exception and exceptionRootCause resolvers.
• Added PluginUtil class.
• Updated TypeConverterRegistry to handle competing converters for the same type.

Judging from the json-template-layout.html generated via ./mvnw site -DskipTests=true -Dmaven.doap.skip=true, there is an AsciiDoc puzzler waiting to be addressed: usage of xref:event-template-resolver-exception[] and xref:event-template-resolver-exceptionRootCause[] renders garbled text and link.

This PR implements a dedicated RedisAppender in log4j2. More details can be found in the log4j2 ticket here: https://issues.apache.org/jira/projects/LOG4J2/issues/LOG4J2-2400?filter=allopenissues

This pull request will allow us to print a common suffix to each line of the stack trace of a throwable event.

The suffix can be configured with normal patterns (But patterns that can handle Throwable will be auto removed from here, because the rendered effect will be confusing with them).

For example, pattern %xEx{suffix(%X{mdc_key})} will append the MDC value to key mdc_key to each line of the stack track.

But for pattern %xEx{suffix(%xEx)}, the nested %xEx pattern will be ignored.

Given pattern "app-%d{yyyy-MM-dd}{timezone}.log.gz" treat second pair of curly braces as timezone option.

If timezone option is not set, then use default timezone as before. If timezone option is set, but it's parsing fails, then use GMT+0. (same DatePatternConverter behavior as in other places)

Warning! These changes are based on the work delivered in #1145 replacing the maven-changes-plugin. Hence, that PR needs to be merged first!

This PR attempts to simplify the site generation. Highlights from the changes are as follows:

1. all Maven reporting is disabled
2. Coveralls, JaCoCo, DOAP, PDF configurations are removed
3. enforce-upper-bound-deps Maven enforcer rule is fixed
4. /pom.xml is overhauled (developers are simplified, contributors are removed, unused configurations/properties are removed, etc.)
5. maven-site-plugin is disabled for all modules except /pom.xml
6. all <module>/src/site/markdown/index.md[.vm] files are moved to /src/site/markdown/<module>.md[.vm]
7. build instructions are updated

As a result, ./mvnw site takes less than 30s on a decent laptop.

We use the method FrameworkUtil.getBundle from OSGi, but this method was added only in OSGi 1.5, this causes NoSuchMethodError if older version of OSGi is used.

I would suggest to consider OSGi being unavailable for OSGi version < 1.5 or when that method is not available.

This is a draft of the changes to the Support page to disclose the relationship of some PMCs with TideLift.

Personally I don't mind disclosing the names of the people involved, if the other PMCs prefer it this way.

Bumps maven-dependency-plugin from 3.3.0 to 3.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.