Java serialization library, proto compiler, code generator

Hi David,

I have a problem right now using your parser. Hope you can help me identify the cause and solve it.

Problem Background: In my message proto file, there are several fields. Almost all the fields, including primitive type field, message field and enum field, have comments (doc). Since I’m generating the documentation for the proto file, I want all comments of fields to show up.

Problem: Only comments of primitive type field (int, string, datetime) show up right now, comments of both message field and enum field don’t show up, even though there are comments for the message and enum field in the proto file.

I'm thinking that maybe all comments of MessageField and EnumField are not parsed through by the parser, and that's the reason it keeps giving me null values.

Regards, Liang

Protobuf allows some fields to be encoded packed:

https://developers.google.com/protocol-buffers/docs/encoding#optional

This patch will allow protostuff to read these fields.

here is my test code:

client code:

class ClassB
{
    @Tag(1)
    private String b1;

    ClassB()
    {
        this.b1 = "b1";
    }
}

class SuperClassA
{
    @Tag(1)
    private String sa1;
}

class ClassA extends SuperClassA
{
    @Tag(101)
    private String a1;

    @Tag(102)
    private ClassB a2;

    public ClassA(ClassB a2)
    {
    }
}

server code:

class ClassB
{
    @Tag(1)
    private String b1;

    ClassB()
    {
        this.b1 = "b1";
    }
}
class SuperClassA
{
    @Tag(1)
    private String sa1;

    @Tag(2)
    private ClassB sa2;

    public SuperClassA(ClassB sa2)
    {
        this.sa1 = "sa1";
        this.sa2 = sa2;
    }
}

class ClassA extends SuperClassA
{
    @Tag(101)
    private String a1;

    @Tag(102)
    private ClassB a2;

    public ClassA(ClassB a2)
    {
        super(a2);
        this.a1 = "a1";
        this.a2 = a2;
    }
}

Protostuff Version:1.0.7 I use the GraphIOUtil.writeTo on server site to serialize the object, and then on client site use client code to deserialize the object. Then it will throws IndexOutOfBoundsException.

but if i use ProtostuffUtil to serialize and deserialize the object, it will pass successfully. I want to know whether is this a bug of the GraphUtil, and if this is a bug, which version have been resolved ? here is the exception stack:

java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
    at java.util.ArrayList.RangeCheck(ArrayList.java:547)
    at java.util.ArrayList.get(ArrayList.java:322)
    at com.dyuproject.protostuff.GraphCodedInput.mergeObject(GraphCodedInput.java:98)
    at com.dyuproject.protostuff.runtime.RuntimeUnsafeFieldFactory$14$1.mergeFrom(RuntimeUnsafeFieldFactory.java:828)
    at com.dyuproject.protostuff.runtime.MappedSchema.mergeFrom(MappedSchema.java:188)
    at com.dyuproject.protostuff.runtime.ObjectSchema.readObjectFrom(ObjectSchema.java:627)
    at com.dyuproject.protostuff.runtime.ObjectSchema.mergeFrom(ObjectSchema.java:312)
    at com.dyuproject.protostuff.GraphCodedInput.mergeFrom(GraphCodedInput.java:153)
    at com.dyuproject.protostuff.CodedInput.mergeObjectEncodedAsGroup(CodedInput.java:271)
    at com.dyuproject.protostuff.CodedInput.mergeObject(CodedInput.java:239)
    at com.dyuproject.protostuff.GraphCodedInput.mergeObject(GraphCodedInput.java:108)
    at com.dyuproject.protostuff.runtime.RuntimeUnsafeFieldFactory$15$1.mergeFrom(RuntimeUnsafeFieldFactory.java:904)
    at com.dyuproject.protostuff.runtime.MappedSchema.mergeFrom(MappedSchema.java:188)
    at com.dyuproject.protostuff.GraphIOUtil.mergeFrom(GraphIOUtil.java:91)

thanks

We really appreciate this project.I wanted to provide a preview of our 0-copy version of protostuff. It obviously needs a lot of work to merge but I wanted to make you aware of it.

Usage: https://github.com/kshchepanovskyi/protostuff-rpc-example

Fixed issue with multiple module outputs and PluginProtoCompiler Add 'service_block' for service generator templates

Reading objects like Strings in Java causes a lot of garbage that can easily cause large application to suffer from unnecessarily long GCs.

This patch allows for the usage of an externally maintained ByteBuffer to be filled from the input, so developers can have the freedom to maintain ThreadLocal instances of that ByteBuffer for example and re-use it, avoiding unnecessary allocations.

RuntimeEnv#loadClass(String) explicitly uses current thread's context classloader instead of Class#forName.

In my use case, where I have a polymorphic field, the class that I want to load is loaded on a separate class loader. As such, I'm unable to deserialize it in this context, because the class is not found on the thread's context class loader.

I created a hotfix library jar to depend on, and it fixed my issue. Simply returning Class.forName(name) instead of the context class loader call in that method.

Reading objects like Strings in Java causes a lot of garbage that can easily cause large application to suffer from unnecessarily long GCs.

This patch allows for the usage of an externally maintained ByteBuffer to be filled from the input, so developers can have the freedom to maintain ThreadLocal instances of that ByteBuffer for example and re-use it, avoiding unnecessary allocations.

Motivation

Java 8's built in String class no longer accepts 3-byte surrogates / 6-byte surrogate pairs. This causes issues when UTF characters, such as emoji's, are present in a string. Since Protostuff encodes using 3-byte surrogates / 6-byte surrogate pairs, deserialization using Java's String class results in a corrupted String.

The reason why Java 8 no longer supports 3-byte surrogates / 6-byte surrogate pairs is that it has tightened it's implementation to follow the standard more rigorously (see here for details). By following the standards more closely, Protostuff should be compatible with more external sources. For example, Protobuff running on Java 8.

Changes

• Support 3-byte surrogates / 6-byte surrogate pairs in Java 8, which allows for communication with older version of protostuff.
• Encode surrogate pairs using Standard UTF-8, instead of 6-byte surrogate pairs to avoid corruption for decoders without 3-byte surrogate / 6-byte surrogate pair support.

Effects

This change should be fully compatible with previous versions of Protostuff, as well as Protobuff's Java implementation (and most likely other language implementations as well). The serialized output should be more in line with what standardized UTF-8 decoders expect.

DataInputStream, however, will not the ability to reliably decode Strings that contain 4-byte surrogates pairs, as they have no support for 4-byte encoded characters. There are many ways around this, however, so it is not a huge concern.

A message field of type uint32 that actually uses the sign-bit, appears to be serialized to JSON as a negative number.

For example, if the corresponding java int has this value:

0xFAE15000

protostuff-json will serialize this to a value -85897216, whereas it really is 4209070080.

This caught me by surprise, as I was not expecting that behaviour in a non-binary format like JSON. I can work around it for my use case by doing some post-processing, but was wondering whether it's a deliberate choice on your end.

thanks

I can't build project in my favourite IDE. There is an issue with protostuff-me - IntelliJ Idea does not support different language versions for single module - 1.3 for main and 1.7 for tests.

Are there any solutions for this issue?

P.S. I understand that this question is only about convenience and I always can run mvn clean install from command prompt.

When there is an reference loop in the object map, the JsonIOUtil.toByteArray method will result in StackOverflowError. Following are the steps to reproduce this phenomenon.

`public class UserVO {

public String name;

public UserVO userVO;

public T t;

}

@Test public void test1() throws IOException { UserVO u1 = new UserVO<>(); u1.name = "u1-name"; u1.t = 43d;

    UserVO u2 = new UserVO();
    u2.name = "u2-name";
    u2.userVO = u1;

    u1.userVO = u2;

    DefaultIdStrategy STRATEGY = new DefaultIdStrategy(IdStrategy.DEFAULT_FLAGS
            | IdStrategy.PRESERVE_NULL_ELEMENTS
            | IdStrategy.MORPH_COLLECTION_INTERFACES
            | IdStrategy.MORPH_MAP_INTERFACES
            | IdStrategy.MORPH_NON_FINAL_POJOS);
    Schema<UserVO> schema = RuntimeSchema.getSchema(UserVO.class,STRATEGY);

    String s = new String(JsonIOUtil.toByteArray(u1, schema, false));
    System.out.println(s);

    UserVO fooParsed = schema.newMessage();
    JsonIOUtil.mergeFrom(s.getBytes(), fooParsed, schema, false);
    System.err.println(fooParsed);

}

Following is the top part of the error stack

java.lang.StackOverflowError at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeFieldName(UTF8JsonGenerator.java:186) at com.fasterxml.jackson.core.json.JsonGeneratorImpl.writeStringField(JsonGeneratorImpl.java:202) at io.protostuff.JsonOutput.writeString(JsonOutput.java:407) at io.protostuff.runtime.DefaultIdStrategy.writePojoIdTo(DefaultIdStrategy.java:507) at io.protostuff.runtime.DerivativeSchema.writeTo(DerivativeSchema.java:111) at io.protostuff.JsonOutput.writeObject(JsonOutput.java:507) at io.protostuff.runtime.RuntimeUnsafeFieldFactory$14$1.writeTo(RuntimeUnsafeFieldFactory.java:1134) at io.protostuff.runtime.RuntimeSchema.writeTo(RuntimeSchema.java:475) at io.protostuff.runtime.DerivativeSchema.writeTo(DerivativeSchema.java:121) at io.protostuff.JsonOutput.writeObject(JsonOutput.java:507) `

I would like to use map defined in .proto v2 file to generate protostuff code. But with no luck. Have someone encounter that before, how are you overcome this?

I opend morph_non_final_pojos option, but it is write className string to the final bytes, It is too large when use the colloctions or maps, so I want to replace my defined class id(int) to className, how to do it?

NPE happens in my app. pls see the code below:

method : io.protostuff.runtime.RuntimeEnv#loadClass

@SuppressWarnings("unchecked")
static <T> Class<T> loadClass(String className)
{
    try
    {
        return (Class<T>) Thread.currentThread().getContextClassLoader()
                .loadClass(className);
    }
    catch (ClassNotFoundException e)
    {
        try {
            return (Class<T>) Class.forName(className);
        } catch (ClassNotFoundException e1) {
            throw new RuntimeException(e);
        }
    }
}

when i use netty GlobalEventExecutor to support thread asynchronization, GlobalEventExecutor would setContextClassLoader((ClassLoader)null),

so Thread.currentThread().getContextClassLoader() is null ,it will lead to NPE.

beside ,i have see this issue : https://github.com/protostuff/protostuff/issues/223

I serialized a Java object, in which the field of integer type is used, and the int32value is used in proto3. When I tried to deserialize using protobuf, I failed.What did I do wrong?

Java:

public class JavaObject {
    private Integer value1;
}

Protobuf:

message ProtoMessage {
    google.protobuf.Int32Value value1 = 1;
}

version: 1.8.0 seems same problem as issue#264 at io.protostuff.runtime.HasDelegate, line 39. change parameter handler from null to ArraySchemas.GENERIC_HANDLER

public class CollectionWithArrayItemThrowNpeTest {
    @Test
    public void test() {
        try {
            DefaultIdStrategy idStrategy = new DefaultIdStrategy(IdStrategy.DEFAULT_FLAGS | IdStrategy.COLLECTION_SCHEMA_ON_REPEATED_FIELDS);
            idStrategy.registerDelegate(new ProtobufCalendar());

            ClassThrowNpeWhenDeserialize originMsg = new ClassThrowNpeWhenDeserialize();
            originMsg.value = new ArrayList<Calendar[]>();
            Calendar[] vArray = new Calendar[1];
            vArray[0] = Calendar.getInstance();
            originMsg.value.add(vArray);
            RuntimeSchema<ClassThrowNpeWhenDeserialize> schema = RuntimeSchema.createFrom(ClassThrowNpeWhenDeserialize.class, idStrategy);

            byte[] bytes = ProtobufIOUtil.toByteArray(originMsg, schema, LinkedBuffer.allocate(512));
            ClassThrowNpeWhenDeserialize msg = schema.newMessage();

            ProtobufIOUtil.mergeFrom(bytes, msg, schema);
        } catch (NullPointerException e) {
            e.printStackTrace();
            Assert.fail();
        }
    }

    static class ClassThrowNpeWhenDeserialize{
        List<Calendar[]> value = new ArrayList<Calendar[]>();
    }

    public static class ProtobufCalendar implements Delegate<Calendar> {

        @Override
        public WireFormat.FieldType getFieldType() {
            return WireFormat.FieldType.FIXED64;
        }

        @Override
        public Calendar readFrom(Input input) throws IOException {
            Calendar calendar = Calendar.getInstance();
            calendar.setTimeInMillis(input.readFixed64());
            return calendar;
        }

        @Override
        public void writeTo(Output output, int number, Calendar value, boolean repeated) throws IOException {
            output.writeFixed64(number, value.getTimeInMillis(), repeated);
        }

        @Override
        public void transfer(Pipe pipe, Input input, Output output, int number, boolean repeated) throws IOException {
            output.writeFixed64(number, input.readSInt64(), repeated);
        }

        @Override
        public Class<?> typeClass() {
            return Calendar.class;
        }

    }
}