We see a class loader memory leak by using Jackson: on redeployment of our application in WebSphere we see an increase of heap usage of a couple of hundred MB's en after several redeployments heap usage becomes close to the heap size and garbage collection takes a lot of CPU. Note that most extra heap is taken by the BufferRecyclers retaining char[][]'s.

SoftReferences may help to prevent out of memory errors, it doesn't help for gc overhead (including long compaction pauses.) In addition, the BufferRecycler classes of previous deployed versions of the app are still in the ThreadLocals of all threads of the threadpool and prevent the classloader with all its classes to be unloaded. See here: https://stackoverflow.com/questions/17968803/threadlocal-memory-leak for more on classloader leaks.

We would like Jackson to release/remove the BufferRecyclers from the ThreadLocals on shutdown, by calling a shutdown method on e.g. JsonFactory. See also: http://java.jiderhamn.se/2012/02/26/classloader-leaks-v-common-mistakes-and-known-offenders/

(migrated from http://jira.codehaus.org/browse/JACKSON-39 -- note, high vote count)

(suggested by Dimitri M on user list)

There are use cases where it'd be good to be able to feed input to parser, instead of trying to provide an input stream for parser to read from. This would cover use cases where input comes in chunks; for example, part of logical document in one chunk, then after a delay (perhaps in a separate request) another one and so forth. In these cases it may be difficult to implement InputStream (Reader etc) abstraction; instead, it would be better if application could feed (push) data to parser. But if so, parser must be able to indicate cases where no data is YET available (but may become available).

This is similar to how Aalto Xml processor (http://www.cowtowncoder.com/hatchery/aalto/index.html) operatesd in its async mode. However, since Json is much simple format than xml, implementation might be simpler.

Based on my experiences with Aalto, implementation is a non-trivial thing however. One problem is that even UTF-8 decoding needs to be somewhat aware of chunk boundaries, so in the end a separate parser may be required: this because current parser uses blocking to handle these split cases. A smaller problem is that of indicating "not-yet-available" case – this can probably be handled by introducing a new member in JsonToken enumeration.

InternCache is used in jackson for json property names only and solves the problem of extra memory footprint. In the same time it uses String.intern - that is not an issue rather than misusage of it. The purpose of using string.intern is jvm specific: usually it maintains string literal pool (and covers internal jvm cases).

There are several known drawbacks of using intern:

• String pool Hashtable is non-resizable table. That means: we suffer of hash code collisions when table will have items more than its size (suffer as much as many items in string.intern pool)
• The other drawback is involve GC into that: string intern pool is subject for minor collection too. So, app suffers of using string.intern implicitly via GC phases.

There is no use cases of check the equality of strings using == and therefore no any reasons of using String.intern - the biggest profit is to have string deduplication (already achieved).

patch

===================================================================
--- src/main/java/com/fasterxml/jackson/core/util/InternCache.java  (revision 489becbfb28a41980f0d5147d6069b30fa3b5864)
+++ src/main/java/com/fasterxml/jackson/core/util/InternCache.java  (revision )
@@ -58,7 +58,7 @@
                 }
             }
         }
-        result = input.intern();
+        result = input;
         put(result, result);
         return result;
     }

Test case

The general idea behind syntenic json is used in test (a flat json structure with 10000 properties with prefix someQName) is provide lots of different property names to trigger usage of string.intern within InternCache - that is close to real apps use cases where amount of unique property names is on a scale from hundreds to thousands.

{
    "someQName0": 0,
    "someQName1": 1,
    ....
    "someQName9999": 9999
}

JMH test

Let's measure performance of a single json parse with interting and w/o it: PerfInternCache.java

Benchmark                   Mode  Cnt     Score     Error  Units
-PerfInternCache.intern      avgt    5  4098.696 ± 164.484  us/op
+PerfInternCache.noIntern    avgt    5  2320.159 ± 204.301  us/op

GC

Another test is to measure how intern is affecting implicitly via GC: handle 10k the same jsons as we use in previous test (the use case is very very close to real one in real apps like web service / microsevices): InternCache GC timings java test

Run it with -verbose:gc -XX:+PrintGCDetails -Xloggc:gc.log

and after that get the total GC pause time $ grep "GC" gc.log | grep "Times: " | awk '{S+=$8}END{print S}'

-intern      0.1907254 +- 0.00469 sec
+w/o intern  0.07665   +- 0.00498 sec

Conclusion

Using intern harms application performance as explicitly via more expencive InternCache.intern and implicitly via GC. In the same time we keep memory footprint on the same low possible level.

Jsoniter project (https://github.com/plokhotnyuk/jsoniter-scala) has many impressive performance optimizations; linked f.ex from here:

https://www.reddit.com/r/java/comments/darehu/jackson_release_210/f1ysf1e/

Of ones included, number-parsing would be relevant for this repo.

EDIT: also see (from the comment below)

"Unrelated to jsoniter but this recent port of Lemire's Double parser:

https://github.com/wrandelshofer/FastDoubleParser

and the original paper https://arxiv.org/abs/2101.11408 also relevant"

Heya, updated to 2.4.1, and started to get the following exception:

java.lang.NullPointerException
    at com.fasterxml.jackson.core.sym.BytesToNameCanonicalizer$Bucket.access$000(BytesToNameCanonicalizer.java:1187)
    at com.fasterxml.jackson.core.sym.BytesToNameCanonicalizer.findBestBucket(BytesToNameCanonicalizer.java:1049)
    at com.fasterxml.jackson.core.sym.BytesToNameCanonicalizer._addSymbol(BytesToNameCanonicalizer.java:858)
    at com.fasterxml.jackson.core.sym.BytesToNameCanonicalizer.addName(BytesToNameCanonicalizer.java:667)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.addName(UTF8StreamJsonParser.java:2127)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.findName(UTF8StreamJsonParser.java:1998)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseMediumName(UTF8StreamJsonParser.java:1580)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(UTF8StreamJsonParser.java:1527)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:693)
    at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:50)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:270)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:308)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:308)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readOrderedMap(AbstractXContentParser.java:258)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrdered(AbstractXContentParser.java:213)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrderedAndClose(AbstractXContentParser.java:228)
    at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:119)
    at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:101)
    at org.elasticsearch.index.mapper.DocumentMapperParser.parseCompressed(DocumentMapperParser.java:177)
    at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:400)
    at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:266)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.processMapping(IndicesClusterStateService.java:419)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyMappings(IndicesClusterStateService.java:360)
    at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:179)
    at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:425)
    at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

This seems to happen with very large documents, with many different field names. I tried loading the document and recreating the failure, but it doesn't seem to fail... . I suspect maybe this is caused by many different threads parsing at the same time? I do not know.

For reference, here is a sample document that it fails on: https://dl.dropboxusercontent.com/u/2136051/jackson_npe.txt.

It would be useful if the public methods in com.fasterxml.jackson.core.io.JsonStringEncoder declare its input parameter as CharSequence rather than String.

To me, it seems to be trivial to fix since the code only uses methods on the input parameter which are in CharSequence (length and charAt).

Bundles need to include extra metadata to support java.util.ServiceLoader inside an OSGi framework. Jackson's bundles are all missing this metadata.

In theory, it should just be a matter of applying Bnd's @ServiceConsumer and @ServiceProvider annotations in the correct places as described here, where these annotations can be provided like this:

diff --git a/base/pom.xml b/base/pom.xml
index 6235a67..4972957 100644
--- a/base/pom.xml
+++ b/base/pom.xml
@@ -30,9 +30,17 @@ of Jackson: application code should only rely on `jackson-bom`
          parent pom, and then also allow override as need be
       -->
     <jackson-bom.version>${project.parent.version}</jackson-bom.version>
+
+    <version.bnd.annotation>6.3.1</version.bnd.annotation>
   </properties>
 
   <dependencies>
+    <dependency>
+      <groupId>biz.aQute.bnd</groupId>
+      <artifactId>biz.aQute.bnd.annotation</artifactId>
+      <version>${version.bnd.annotation}</version>
+      <scope>provided</scope>
+    </dependency>
     <dependency> <!-- all components use junit for testing -->
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>

I am struggling to wrap my head around your myriad of separate repositories, and so you may have a better way of providing this dependency.

Relates to https://github.com/wrandelshofer/FastDoubleParser/issues/25

Should lead to better performance on newer JDKs. Up until now, the code that was copied into jackson-core was optimised for Java8.

Shade the fastdoubleparser classes and bundle into jackson-core jar.

Please can you strictly name the jackson modules so that automatic naming is not used in JPMS

This will enable the following functionality in module-info.java

	requires transitive com.fasterxml.jackson.annotation;
	requires transitive com.fasterxml.jackson.core;
	requires transitive com.fasterxml.jackson.databind;

etc.. This way boot/base modules do not need to explicitly define the automatic module name in the libraries

The moditect plugin can perform this function for JDK 7 and JDK 8 builds via the following https://github.com/moditect/moditect

I haven't created a PR because it would be needed across all the projects, but it is very simple to implement.

	<plugin>
				<groupId>org.moditect</groupId>
				<artifactId>moditect-maven-plugin</artifactId>
				<version>1.0.0.Beta1</version>
				<executions>
					<execution>
						<id>add-module-infos</id>
						<phase>package</phase>
						<goals>
							<goal>add-module-info</goal>
						</goals>
						<configuration>
							<overwriteExistingFiles>true</overwriteExistingFiles>
							<module>
								<moduleInfoFile>
									src/moditect/module-info.java
								</moduleInfoFile>
							</module>
						</configuration>
					</execution>
				</executions>
			</plugin>

This will allow users to specify multiple type parameters, as well it will allow them to easily mix simple (URL) and compound type (List<URL>) parameters with ease. Finally, by having JavaType extend java.lang.reflect.Type we will be able to tunnel JavaType through Jersey's Client.get(Type) method.

See issue #41 for more information.

I have a TypeIdResolver implementation which works around generic type erasure to allow marshaling/unmarshaling of a parameterized type. This works fine when the underlying parameterized type maps directly onto a JSON type e.g. java.lang.Double but when it's a type that needs to be converted to a JSON string e.g. java.time.LocalDate then my TypeIdResolver is asked for a type and I can either return the String "java.time.LocalDate" or a null value.

If I return a null then TypeSerializerBase#handleMissingId currently does nothing as per https://github.com/FasterXML/jackson-databind/issues/633 but it looks like the control of how to write the typeId is now handed off to JsonGenerator#writeTypePrefix @ https://github.com/FasterXML/jackson-core/blob/master/src/main/java/com/fasterxml/jackson/core/JsonGenerator.java#L1061

It would be great if writeTypePrefix did nothign if typeIdDef.id is null - I don't want to monkey-patch this unless it's going to make it into master...

FilteringParserDelegate when parsing ID_END_ARRAY or ID_END_OBJECT of the skipped element (when _headContext.isStartHandled() == false should be able to exit the loop and continue parsing next tokens.

This logic was changed accidentally removed in https://github.com/FasterXML/jackson-core/commit/7db467ddec7c2899038249b55695b7e44c7b5c3e#diff-f6642caef61e0c403f51a6150ecf45263034fca5002782fd02eacd01e53fe549L694

closes https://github.com/FasterXML/jackson-core/issues/882

When the last element in array is an array or object and that element is skipped, the FilteringParserDelegate will end up in a loop from which it cannot exit. This means that the rest of the input will be skipped too. This results in incorrect JSON.

This behaviour exists since 2.9+ I believe this is due to https://github.com/FasterXML/jackson-core/commit/7db467ddec7c2899038249b55695b7e44c7b5c3e#diff-f6642caef61e0c403f51a6150ecf45263034fca5002782fd02eacd01e53fe549L694 where the if (gotEnd) conditions where removed. I think this should be added as currently the logic is:

 boolean gotEnd = (_headContext == buffRoot);
                    boolean returnEnd = gotEnd && _headContext.isStartHandled();

                    _headContext = _headContext.getParent();
                    _itemFilter = _headContext.getFilter();

                    if (returnEnd) {
                        return t;
                    }

and that means that it can only exit when _headContext.isStartHandled() is true. For skipped elements this is false.

This can be easily reproduced with this testcase

 @Test
    public void testCustomIncludesWithMultipleObjectsInArrayMissLast() throws Exception {
        var factory = new JsonFactory();
        var baseParser = factory.createParser("{\"foo\":[{\"bar\":\"baz\"},{\"bing\":\"boom\"}]}");
        var filteredParser = getFilteredParser(baseParser, filter("foo", "bar"));
        var writer = new StringWriter();
        var generator = factory.createGenerator(writer);
        Assertions.assertTrue(filteredParser.nextToken().isStructStart());
        generator.copyCurrentStructure(filteredParser);
        generator.flush();
        Assertions.assertEquals("{\"foo\":[{\"bar\":\"baz\"}]}", writer.toString());
        //Expected :{"foo":[{"bar":"baz"}]}
        //Actual   :{"foo":[{"bar":"baz"} 
    }

cc @tvernum who coauthored the fix and the testcases

I did the following test, is it possible without adding @JsonFilter("baseFilter") get the result I expected

import com.fasterxml.jackson.annotation.JsonFilter;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationConfig;
import com.fasterxml.jackson.databind.ser.impl.SimpleBeanPropertyFilter;
import com.fasterxml.jackson.databind.ser.impl.SimpleFilterProvider;
import lombok.Data;
import lombok.SneakyThrows;
import org.junit.jupiter.api.Test;

import java.util.Date;
import java.util.UUID;


public class JacksonIgnoreFieldTest {
    private static String[] IGNORE_FIELD = {"delFlag", "createBy", "createTime"};
    private static SimpleBeanPropertyFilter MY_FILTER = SimpleBeanPropertyFilter.serializeAllExcept(IGNORE_FIELD);

    private static ObjectMapper objectMapper() {
        return new ObjectMapper();
    }

    @SneakyThrows
    @Test
    public void serializationConfigTest() {
        ObjectMapper mapper = objectMapper();
        SerializationConfig serializationConfig = mapper.getSerializationConfig().withFilters(new SimpleFilterProvider().setDefaultFilter(MY_FILTER));
        mapper.setConfig(serializationConfig);
        User user = new User();
        System.err.println(mapper.writeValueAsString(user));
        // expect: {"userId":"9f452b3a-5962-45a9-b16c-e5cb92c3baf1_userId","userName":"fa566a27-f855-4c21-9dda-2b48590f0944_userName"}
        // actual: {"userId":"9f452b3a-5962-45a9-b16c-e5cb92c3baf1_userId","userName":"fa566a27-f855-4c21-9dda-2b48590f0944_userName","delFlag":"0","createBy":"cf6cfb21-5f14-4b4a-8268-f453ce991fdb_createBy","createTime":1672133883455}
    }

    @SneakyThrows
    @Test
    public void writerFilterProviderDefaultFilter() {
        ObjectMapper mapper = objectMapper();
        mapper.writer(new SimpleFilterProvider().setDefaultFilter(MY_FILTER));
        User user = new User();
        System.err.println(mapper.writeValueAsString(user));
        // expect: {"userId":"9f452b3a-5962-45a9-b16c-e5cb92c3baf1_userId","userName":"fa566a27-f855-4c21-9dda-2b48590f0944_userName"}
        // actual: {"userId":"9f452b3a-5962-45a9-b16c-e5cb92c3baf1_userId","userName":"fa566a27-f855-4c21-9dda-2b48590f0944_userName","delFlag":"0","createBy":"cf6cfb21-5f14-4b4a-8268-f453ce991fdb_createBy","createTime":1672133883455}
    }

    @SneakyThrows
    @Test
    public void addFilterAddAnnotation() {
        ObjectMapper mapper = objectMapper();
        mapper.setFilterProvider(new SimpleFilterProvider().addFilter("baseFilter", MY_FILTER));
        Role role = new Role();
        System.err.println(mapper.writeValueAsString(role));
        // expect: {"roleId":"313cb8b2-0568-410c-b987-2e4f6e5e3cfb_roleId","roleName":"cb88e09f-2b49-4544-be7d-beea9187c2b0_roleName"}
        // actual: {"userId":"2000c23d-c9f0-42f5-b599-263a1b5e41ff_userId","userName":"c9a221f2-1b5a-4ebf-bba2-8119878c32a4_userName","delFlag":"0","createBy":"7d4fc42c-461a-42b1-ad3d-5f3a2dfb0215_createBy","createTime":1672134420799}
    }

    @SneakyThrows
    @Test
    public void addFilterNoAnnotation() {
        ObjectMapper mapper = objectMapper();
        mapper.setFilterProvider(new SimpleFilterProvider().addFilter("baseFilter", MY_FILTER));
        User user = new User();
        System.err.println(mapper.writeValueAsString(user));
        // expect: {"userId":"9f452b3a-5962-45a9-b16c-e5cb92c3baf1_userId","userName":"fa566a27-f855-4c21-9dda-2b48590f0944_userName"}
        // actual: {"userId":"2000c23d-c9f0-42f5-b599-263a1b5e41ff_userId","userName":"c9a221f2-1b5a-4ebf-bba2-8119878c32a4_userName","delFlag":"0","createBy":"7d4fc42c-461a-42b1-ad3d-5f3a2dfb0215_createBy","createTime":1672134420799}
    }

    @Data
    public static class User {

        private String userId;
        private String userName;
        private String delFlag;
        private String createBy;
        private Date createTime;

        public User() {
            userId = UUID.randomUUID() + "_userId";
            userName = UUID.randomUUID() + "_userName";
            delFlag = "0";
            createBy = UUID.randomUUID() + "_createBy";
            createTime = new Date();
        }
    }

    @Data
    @JsonFilter("baseFilter")
    public static class Role {
        private String roleId;
        private String roleName;
        private String delFlag;
        private String createBy;
        private Date createTime;

        public Role() {
            roleId = UUID.randomUUID() + "_roleId";
            roleName = UUID.randomUUID() + "_roleName";
            delFlag = "0";
            createBy = UUID.randomUUID() + "_createBy";
            createTime = new Date();
        }

    }

}

https://github.com/openjdk/jdk/pull/9410

Even if these changes are merged soon, they will only go into Java 20 or 21. If we inline them in jackson-core, even Java 8 users benefit. jackson-core already has fast float and double writing code that is borrowed straight from Raffaello Giulietti's fantastic work speeding up Java number support.

Probably best to wait until the PR above is ready to be merged.

Based on https://github.com/wrandelshofer/FastDoubleParser/issues/32