Uber-project for (some) standard Jackson textual format backends: csv, properties, yaml (xml to be added in future)

This is WIP, creating a PR for discussion on the approach.

As discussed in #219, the toml parser landscape in java is mediocre. For this reason, I decided it was worth rolling our own, though I'm open to feedback on that decision.

TOML has an ABNF grammar available that I assume to be authoritative. The ABNF parser generator landscape is also not great in Java, so I decided to transform it by hand (potentially with errors?) into a jflex lexer file (toml.jflex), which generates a fairly efficient lexer using the jflex maven plugin. From that, I implemented a basic recursive descent parser (Parser.java). This approach also has the benefit of not requiring an external runtime dependency like antlr, unlike some existing parsers.

I have created a test class that includes all examples from the TOML docs. Most already work, but there are some samples that don't yet conform, mostly parsing successfully when parsing should fail.

I do not expect serialization to be a big problem, so I am keeping that for the end.

A few things are still outstanding:

• Should the toml parsers be encapsulated? Right now they are package-private. Most of jackson offers some extendability, but in my opinion, here it is beneficial to have the freedom to swap the parser implementation in the future.
• Is ObjectNode really the best model to parse to at the start?
  • A full streaming solution is out because of the out-of-order nature of TOML
  • We need to somehow remember which objects are "explicitly" and "implicitly" declared to fully conform to the spec.
• How to handle datetime? The TOML types map well to java.time, but not sure if this can be stored properly in an ObjectNode or how it's emitted by the parser I'll add eventually.
• How to handle integer types? There's some flags on ObjectMapper for this. TOML has some requirements here ("Arbitrary 64-bit signed integers (from −2^63 to 2^63−1) should be accepted and handled losslessly", "Floats should be implemented as IEEE 754 binary64 values."), so it might be feasible to just use long and double, but we could also use BD/BI like the code does right now.
• What's the license for the toml spec? Using test cases from it may require a note in the project dir, though it should not affect the license of the final project, since we only use it for tests and generating code.
• TOML has a compliance test suite, but I can't find the actual test files :) may be worth including in our test suite.
• 2.13 backport? Maybe once this is done...

I want to create a map from properties using JavaPropsMapper, which works as long as the keys for the map don't contain dots.

I have something like this:

    Config expResult = new Config(
            new Config.Server(17, "/path", "param1", "param2", "Wobble")
            , ImmutableMap.<String, String>builder().put("x.y", "y").put("a.b", "b").build()
    );

    System.setProperty("server.arg", "param1");
    System.setProperty("map.x.y", "y");

And when I run that through the JavaPropsMapper it falls over because it creates: {"map":{"x":{"y":"y"}}} when what I want is: {"map":{"x.y":"y"}} (I want to have domain names as keys)

It would be nice if there was some way to escape the dots in property names so that they weren't considered to be delimiters. I could probably make this work by configuring a completely different delimiter, but that would just confuse anyone wanting to work with this.

Things I've tried, any of which would be an OK solution for me, are:

    System.setProperty("map[x.y]", "y");
    System.setProperty("map.[x.y]", "y");
    System.setProperty("map.x\\.y", "y");
    System.setProperty("map.'x.y'", "y");
    System.setProperty("map.x'.'y", "y");

During the last years I did a lot of CSV reading (with and without Jackson). When parsing this type of data with Jackson, I often serialize rows to maps, when there is some sort of dynamics in the data that is impractical to handle with POJO mappers.

This is a typical example:

CsvSchema headerSchema = CsvSchema.emptySchema().withHeader();
MappingIterator<Map<String, String>> it = mapper
                .readerForMapOf(String.class)
                .with(schema)
                .readValues(input);
while (it.hasNext()) {
  Map<String, String> row = it.nextValue();
  String xVal = row.get("x");
  String yVal = row.get("y");
  String temperature = row.get("temperature");
  // do something with the values
  ...
}

The usual situation is that all the valid header values stem from a fixed set of strings, so there is an immediate association with Java Enums. I would like to do something like this:

enum MyEnum {
  X, Y, TEMPERATURE, HUMIDITY, ... ;
}

static MyEnum parse(String s) {
  ... // a separate method or an implementation detail of the enum class
}

...

CsvSchema headerSchema = CsvSchema.emptySchema().withHeader();
MappingIterator<EnumMap<MyEnum, String>> it = mapper
                .readerForMapOf(MyEnum.class, h -> parse(h), String) // specify enum type, mapping function header->enum, value class
                .with(schema)
                .readValues(input);
while (it.hasNext()) {
  EnumMap<MyEnum, String> row = it.nextValue();
  String xVal = row.get(X);
  String yVal = row.get(Y);
  String temperature = row.get(TEMPERATURE);
  // do something with the values
  ...
}

Besides rigour, Enum headers / EnumMaps have other advantages, such as internal optimizations for speed and memory footprint (compared to HashMap or LinkedHashMap).

Would that proposal be something that is within the scope of this library?

Not sure I've chosen the right repository to report the issue, but there's a similar one: #65

I can't see a "dataformats-text" explicit dependency so here's how I use it:

compile 'com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.9.9'
compile 'com.fasterxml.jackson.core:jackson-databind:2.9.9.3'

Below are tests proving it fails to parse numbers with underscores.

    private static class LongHolder {
        private Long v;

        public Long getV() {
            return v;
        }

        public void setV(Long v) {
            this.v = v;
        }
    }

    private static class IntHolder {
        private Integer v;

        public Integer getV() {
            return v;
        }

        public void setV(Integer v) {
            this.v = v;
        }
    }

    @Test
    public void testYamlUnderscores() throws IOException {
        ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
        mapper.readValue("v: 1000000", IntHolder.class); // ok
        mapper.readValue("v: 1_000_000", IntHolder.class); // ok
        mapper.readValue("v: 1000000", LongHolder.class); // ok

        // throws: com.fasterxml.jackson.databind.JsonMappingException: For input string: "1_000_000"
        mapper.readValue("v: 1_000_000", LongHolder.class);
    }

    @Test
    public void testJsonUnderscores() throws IOException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.readValue("{\"v\": \"1000000\"}", IntHolder.class); // ok

        // throws:
        // com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `java
        // .lang.Integer` from String "1_000_000": not a valid Integer value
        // at [Source: (String)"{"v": "1_000_000"}"; line: 1, column: 7]
        mapper.readValue("{\"v\": \"1_000_000\"}", IntHolder.class);

        mapper.readValue("{\"v\": \"1000000\"}", LongHolder.class); // ok

        // throws:
        // com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type `java
        // .lang.Long` from String "1_000_000": not a valid Long value
        // at [Source: (String)"{"v": "1_000_000"}"; line: 1, column: 7]
        mapper.readValue("{\"v\": \"1_000_000\"}", LongHolder.class);
    }

Hi,

I spotted a performance regression in snakeyaml introduced in version 1.18+. This issue has been reported on the snakeyaml mailing list here and seems to affect the parsing of large text values.

It affects jackson-dataformats-text starting version 2.9.1 as it depends on snakeyaml 1.18. Version 1.19 is also problematic but version 1.17 is ok.

I understand that this is not a jackson-dataformats-text issue but I'd like 1) to raise awareness on this regression and 2) to ensure that jackson-dataformats-text dependency on snakeyaml will be updated to a newer version that fixes the regression (if such version is released)

Hi,

Properties may have a prefix for an object, for example:

a.b.c.name =
a.b.c.id =
x.y.z.name =
x.y.z.color =

It would be nice if a prefix can be specified, so that only these properties with the prefix will be loaded, prefix will be stripped and then business as usual. When writing, the prefix will be added.

This will allow reading system properties and complex file properties using this interface quite easily and replace many of the complex properties processing.

If this is acceptable, I can workout a patch.

Thanks!

jackson-dataformat-yaml 2.8.11

read file config like this:

 gemfireSwitchForRegions:
    maxConnections: '20'
    user: 'OFF'
    legalEntity: 'ON'

----
java

ObjectMapper objectMapper = new ObjectMapper(new YAMLFactory().enable(Feature.MINIMIZE_QUOTES));
JsonNode jsonNode = objectMapper.readTree(inputStream);
SequenceWriter sw = objectMapper.writer().writeValues(System.out);
sw.write(jsonNode);

----
output:

  gemfireSwitchForRegions:
    maxConnections: 20
    user: OFF
    legalEntity: ON

read file config like this:

 gemfireSwitchForRegions:
    maxConnections: '20'
    user: OFF
    legalEntity: ON

----
java

ObjectMapper objectMapper = new ObjectMapper(new YAMLFactory().enable(Feature.MINIMIZE_QUOTES));
JsonNode jsonNode = objectMapper.readTree(inputStream);
SequenceWriter sw = objectMapper.writer().writeValues(System.out);
sw.write(jsonNode);

----
output:

  gemfireSwitchForRegions:
    maxConnections: 20
    user: false
    legalEntity: true

why in the first case it will convert it will output OFF/ON, but when it reads OFF/ON, it will change to false/true?

When reading given CSV with jackson-dataformat-csv 2.11.4

name
Roger
Chris

using following snippet

CsvMapper csvMapper = new CsvMapper();
csvMapper.configure(CsvParser.Feature.FAIL_ON_MISSING_COLUMNS, true);
CsvSchema csvSchema = CsvSchema.builder().setUseHeader(true).setReorderColumns(true)
        .addColumn("name").addColumn("age").build();
List<Person> persons = csvMapper
        .readerFor(Person.class)
        .with(csvSchema)
        .<Person> readValues(csv)
        .readAll();
...
class Person {
    public String name;
    public int age;
}

doesn't throw a CsvMappingException although age column is missing in CSV.

A backwards-incompatible (with jackson 2.9.5) change was introduced in snakeyaml 1.20, so anyone trying to use the latest released versions of jackson and snakeyaml would run into a problem:

java.lang.NoSuchMethodError: org.yaml.snakeyaml.events.MappingStartEvent.<init>(Ljava/lang/String;Ljava/lang/String;ZLorg/yaml/snakeyaml/error/Mark;Lorg/yaml/snakeyaml/error/Mark;Ljava/lang/Boolean;)V

    at com.fasterxml.jackson.dataformat.yaml.YAMLGenerator.writeStartObject(YAMLGenerator.java:489)
    at com.fasterxml.jackson.core.base.GeneratorBase.writeStartObject(GeneratorBase.java:286)
    at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:626)
    at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
    at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3207)

Another release - maybe another CVE - unclear from release notes

https://bitbucket.org/snakeyaml/snakeyaml/wiki/Changes

https://bitbucket.org/snakeyaml/snakeyaml/issues/547/restrict-the-size-of-incoming-data

Hello there

I wanted to report that a key containing a colon : followed by new line chars \n does not seem to be parsing correctly. For example the following test case reproduces the error

    @Test(expected = JsonParseException.class)
    public void testYamlMultiLineNotCanonical() throws Exception {
        String key = "SomeKey:\n\nOtherLine";
        ObjectMapper jsonMapper = new ObjectMapper();
        ObjectNode jsonRoot = jsonMapper.createObjectNode();
        jsonRoot.put(key, 302);

        YAMLFactory yamlFactory = new YAMLFactory();
        ObjectMapper yamlObjectMapper = new ObjectMapper(yamlFactory);
        yamlObjectMapper.configure(JsonParser.Feature.STRICT_DUPLICATE_DETECTION, true);
        StringWriter swriter = new StringWriter();

        yamlObjectMapper.writeValue(swriter, jsonRoot);

        ObjectMapper readMapper =  new ObjectMapper(yamlFactory);
        ObjectNode readJsonRoot = (ObjectNode) readMapper.readTree(swriter.toString());
        Assert.assertEquals(readJsonRoot.get(key).asInt(),302 );
    }

I would have expected the parser to quote the key as \n and : are supported yaml content.

There is work around for this: Force canonical output.

        yamlFactory.configure(YAMLGenerator.Feature.CANONICAL_OUTPUT, true);

That however could have other side effects. Maybe I am missing some other configuration that would do the same?

Thoughts, comments are welcomed.

I am generating a yml file with the below settings:

ObjectMapper mapper = new ObjectMapper(new YAMLFactory()
	.enable(YAMLGenerator.Feature.LITERAL_BLOCK_STYLE)
	.enable(YAMLGenerator.Feature.MINIMIZE_QUOTES)
);

This will write multiline strings as following:

    - replaceInFiles:
        replacement: test
                     123

However if i put a space between "test" and the following line break it will quote the string as if MINIMIZE_QUOTES was disabled:

    - replaceInFiles:
        replacement: "test \n\n123\n"

Hi - are you able to comment on the usage of snakeyaml regarding CVE-2022-1471? Is the team able to make a statement like this one? https://github.com/spring-projects/spring-boot/issues/33457#issuecomment-1336643760 Thanks!

|    |    |         +--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.14.0
|    |    |         |    +--- com.fasterxml.jackson.core:jackson-databind:2.14.0 (*)
|    |    |         |    +--- org.yaml:snakeyaml:1.33 -> 1.32
|    |    |         |    +--- com.fasterxml.jackson.core:jackson-core:2.14.0 (*)
|    |    |         |    \--- com.fasterxml.jackson:jackson-bom:2.14.0 (*)

There's an issue in the archived repo on this. I already checked the API docs. However, I did not find a way to change the decimal separator other than annotating my POJO or modifying its getters. The latter is what I have currently implemented.

Is there another way to change the decimal mark? I am not very familiar w/ the API.

I create an ObjectWriter like this:

CsvMapper mapper = new CsvMapper();
CsvSchema schema = mapper
  .typedSchemaFor(MyPojo.class)
  .withColumnReordering(true);

ObjectWriter csvWriter = mapper
  .writerWithSchemaFor(MyPojo.class)
  .with(schema);

When trying to parse a CSV with missing cells (in present columns) into a Kotlin Data class with fields that have default values, I found (https://github.com/FasterXML/jackson-module-kotlin/issues/605) that I needed to combine CsvParser.Feature.EMPTY_STRING_AS_NULL and KotlinFeature.NullIsSameAsDefault to achieve this. Looking through the test cases, it looks to me as if the CSV handling of missing columns could be improved. Quoting myself from the above mentioned ticket:

the usage of default parameter should be the default when parsing a CSV and encountering an empty cell, regardless of whether or not EMPTY_STRING_AS_NULL is used, in my opinion.

That is, because CSVs don't have explicit nulls (like JSONs do). If I'm not mistaken, missing fields in JSONs are parsed in a way that default values on the respective field are used. So far so good. However, in CSVs, there is a difference between missing columns (defaults are used) and missing cells in present columns (requires me to mix feature flags from CsvParser and Kotlin module to achieve this. I'm not sure about the situation for Java/POJOs. Looking at the existing tests,

MissingColumnsTest::testDefaultMissingHandling() handles missing columns the way I would like missing cells (in a present column) to be handled, but NullReadTest::testEmptyStringAsNull330() ensures empty cells are handled as null even if there is a default value.

As a infrequent user, I find the different handling of missing/present cells/columns confusing. In my opinion, all three of the following assertions should succeed or there should be a single flag to make it so (something like EMPTY_AND_MISSING_AS_DEFAULT or so):

    static class PojoWithDefault {
        public Integer id;
        public String value = "default";
    }

    public void testDefault() throws Exception {
        CsvSchema headerSchema = CsvSchema.emptySchema().withHeader();

        PojoWithDefault missingColumn = MAPPER
                .readerFor(PojoWithDefault.class)
                .with(headerSchema)
                .<PojoWithDefault>readValues("id\n"
                                             + "1").next();
        Assert.assertEquals(missingColumn.value, "default"); //succeeds

        PojoWithDefault missingCellInPresentColumnWithoutComma = MAPPER
                .readerFor(PojoWithDefault.class)
                .with(headerSchema)
                .<PojoWithDefault>readValues("id,value\n"
                                    + "1").next();
        Assert.assertEquals(missingCellInPresentColumnWithoutComma.value, "default"); //succeeds

        PojoWithDefault missingCellInPresentColumnWithComma = MAPPER
                .readerFor(PojoWithDefault.class)
                .with(headerSchema)
                .<PojoWithDefault>readValues("id,value\n"
                                    + "1,").next();
        Assert.assertEquals(missingCellInPresentColumnWithComma.value, "default"); //fails
    }

Ordered maps were introduced in YAML 1.1 and the spec discourages considering key order to be significant otherwise:

object: !!omap
- property: "value"

With jackson 2.13.3, this is parsed as a Map<Object, List<Map>> while I had expected it to be parsed as a Map<Object, LinkedHashMap> instead. Besides being a bit odd and unexpected, serialising the parsed object produces

object:
- property: "value"

Notice the missing !!omap in the output.

I don't know how I would want to explicitly request !!omap serialisation - I'm certainly not suggesting clobbering the key parse order with HashMap when !!omap was not explicitly specified just so that LinkedHashMap suddenly means !!omap, even though that approach would technically be compliant with the spec. Perhaps a configuration flag to select YAML 1.0 or YAML 1.1+ serialisation of LinkedHashMap would make sense? And maybe wrapper/delegate map objects to explicitly override the !!omap type for a specific LinkedHashMap instance regardless of the flag?