I'm not sure this is due to OpenJDK, but here's what I've found.

The Android N preview now uses OpenJDK, and when running my app using FST on the emulator using the latest N preview system image, I get the following crash:

Caused by: java.lang.NoSuchMethodException: newInstance [class java.lang.Class, long]
  at java.lang.Class.getMethod(Class.java:2016)
  at java.lang.Class.getDeclaredMethod(Class.java:1995)
  at org.objenesis.instantiator.android.Android18Instantiator.getNewInstanceMethod(Android18Instantiator.java:53)
  at org.objenesis.instantiator.android.Android18Instantiator.<init>(Android18Instantiator.java:38) 
  at org.objenesis.strategy.StdInstantiatorStrategy.newInstantiatorOf(StdInstantiatorStrategy.java:86) 
  at org.objenesis.ObjenesisBase.getInstantiatorOf(ObjenesisBase.java:91) 
  at org.nustaq.serialization.FSTObjenesisInstantiator.<init>(FSTObjenesisInstantiator.java:38) 
  at org.nustaq.serialization.FSTConfiguration$4.getInstantiator(FSTConfiguration.java:372) 
  at org.nustaq.serialization.FSTClazzInfo.<init>(FSTClazzInfo.java:129) 
  at org.nustaq.serialization.FSTClazzInfoRegistry.getCLInfo(FSTClazzInfoRegistry.java:130) 
  at org.nustaq.serialization.FSTClazzNameRegistry.addClassMapping(FSTClazzNameRegistry.java:98) 
  at org.nustaq.serialization.FSTClazzNameRegistry.registerClassNoLookup(FSTClazzNameRegistry.java:85) 
  at org.nustaq.serialization.FSTClazzNameRegistry.registerClass(FSTClazzNameRegistry.java:81) 
  at org.nustaq.serialization.FSTConfiguration.addDefaultClazzes(FSTConfiguration.java:773) 
  at org.nustaq.serialization.FSTConfiguration.initDefaultFstConfigurationInternal(FSTConfiguration.java:445) 
  at org.nustaq.serialization.FSTConfiguration.createAndroidDefaultConfiguration(FSTConfiguration.java:375) 
  at org.nustaq.serialization.FSTConfiguration.createDefaultConfiguration(FSTConfiguration.java:438) 
  at org.nustaq.serialization.FSTConfiguration.createDefaultConfiguration(FSTConfiguration.java:433) 
  at org.nustaq.serialization.FSTConfiguration.getDefaultConfiguration(FSTConfiguration.java:185) 
  at org.nustaq.serialization.FSTObjectOutput.<init>(FSTObjectOutput.java:74) 
  at ...

I have a class Called DateEncoderSerializer which extends FSTBasicObjectSerializer My intention is to set the DateTimeFormatter (Joda) on it before returning the DateEncoder object, since DateTimeFormatter doesn't implement Serializable.

I marked the relevant DateTimeFormatter fields in my DateEncoder class as "transient", which allows the DateEncoder to be successfully serialized/deserialized when no Serializer is registered with the FSTConfiguration object. However, of course my field is uninitialized because I have to set it to be "transient" to avoid exceptions.

When I register the Serializer:

fastSerialConfig.registerSerializer(DateEncoder.class, new DateEncoderSerializer(), false);

I get the following stack trace?

I've tried multiple variations of out.writeObject(...), out.defaultWriteObject(...) in combination with the associated in.readXXX() methods and I keep getting Exceptions?

Like I said, without the registration, the DateEncoder object serializes/deserializes fine but I need to be able to set the DateTimeFormatter on it before it is usable??

Any ideas why the out/in write/read methods are failing?

java.io.IOException: java.lang.NullPointerException at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:243) at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:1122) at org.numenta.nupic.network.NetworkSerializerImpl.deSerialize(NetworkSerializerImpl.java:142) at org.numenta.nupic.network.JavaFstNetworkSerializationTest.testHTMSensor(JavaFstNetworkSerializationTest.java:498) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) Caused by: java.lang.NullPointerException at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:353) at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:708) at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:562) at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:708) at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:562) at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:370) at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:327) at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:307) at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:241) ... 26 more

Map<String, Integer> m1, m2;

m1 = new LinkedHashMap<String, Integer>();
m1.put("a", 1);
m1.put("BB", 1);
m1.put("ccc", 1);
m1 = Collections.unmodifiableMap(m1);

System.out.println(m1.keySet());

FSTConfiguration fstConf = FSTConfiguration.createDefaultConfiguration();
ByteArrayOutputStream out = new ByteArrayOutputStream();
fstConf.encodeToStream(out, m1);
m2 = (Map<String, Integer>) fstConf.decodeFromStream(new ByteArrayInputStream(out.toByteArray()));

System.out.println(m2.keySet());

Output is:

[a, BB, ccc]
[BB, a, ccc]

FSTVersion: 2.45

class FOO implements Serializable {
    FOO()
    {
    }
    FOO[] foos;
}

FOO[] test = new FOO[1];
test[0] = new FOO();
test[0].foos = test;

test will serialize, but it will fail to deserialize: deserialize failed: java.io.IOException: java.io.IOException: unable to ressolve handle 0 <foos of FOO> 111 : 0 I'm suspicious of FSTObjectInput.instantiateArray it calls objects.registerObjectForRead only AFTER the array is fully read

FSTObjectInput.readArrayNoHeader also calls objects.registerObjectForRead but with a pos that has advanced slightly, so it's not the right handle for the array.

In the previous bug I reported with FSTIdentity2IdMap, my real data leads to an intermittent instance of this deserialize failure, but only after FSTIdentity2IdMap.resize fails.

In reflecting on the other issue, while it's wrong not to find an already written object (because it got lost by FSTIdentity2IdMap.resize), I still wasn't sure why the resulting data wouldn't deserialize. It seems like it should deserialize, but have improper extra copies of objects.

That's what led me to this issue.

However, the structure of my data, in the cases that work (most of the time, when the hash function get luckier), I don't understand why the deserializer works, as the same self-references exist.

I need to be sure that my data serializes and deserializes correctly 100% of the time. I feel pretty certain FSTIdentity2IdMap.resize is broken (see my other post), and it seems like there's at least one more issue with Arrays, but I've only got it 80% understood at this point.

I'm seeing very intermittent issues with FSTObjectOutput (when iterating over serializing 100's of objects). I kept thinking: shouldn't it be completely deterministic? In the course of trying to debug it, I stumbled upon the use of SoftReference in FSTConfiguration. So then it occurred to me, oh, I wonder if the intermittent nature stems from the behavior of the Garbage Collector?

So, I experimented with 2 things:

1. putting System.gc() in the loop. this makes it a lot slower, but failures still occurred intermittently. of course, this call isn't guaranteed to actually do anything.
2. removing the use of SoftReference in FSTConfiguration and just holding references to the Objects directly. In this configuration, everything appears to be stable! Promising!

I noticed that you had this comment: /** * reuse heavy weight objects. If a FSTStream is closed, objects are returned and can be reused by new stream instances. * the objects are held in soft references, so there should be no memory issues. FIXME: point of contention ! * @param cached */

Which suggests (perhaps) some uncertainty about the use of SoftReference?

I don't know your code well enough to decide whether it's safe for me to use my modification (this is a utility that runs and exit, so I'm not concerned about leaking memory, but I am very concerned about reliability),

Have you seen problems with this? My test case can't be shared directly, but I've spent enough time on this problem, that I could work towards developing a standalone example. But before I spent more time on it, I thought it was worth asking you about it.

Thanks, Brad

I use fst for memcached, but application restart throw error

Exception in thread "pool-3-thread-1" ameba.exception.AmebaException: java.io.IOException: java.lang.NullPointerException
    at ameba.cache.util.Serializations.asObject(Serializations.java:57)
    at ameba.cache.util.Serializations.asObject(Serializations.java:70)
    at ameba.cache.Cache.get(Cache.java:365)
    at com.xxxx.fiaame.resources.basedata.SecurityService.findStockSecurityList(SecurityService.java:27)
    at com.xxxx.fiaame.resources.common.CacheService.refreshStockList(CacheService.java:129)
    at com.xxxx.fiaame.resources.common.CacheService.doCache(CacheService.java:73)
    at com.xxxx.fiaame.resources.common.CacheFeature$1.run(CacheFeature.java:28)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.NullPointerException
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:241)
    at ameba.cache.util.Serializations.asObject(Serializations.java:55)
    ... 9 more
Caused by: java.lang.NullPointerException
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:350)
    at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:683)
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:537)
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:363)
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:325)
    at org.nustaq.serialization.serializers.FSTArrayListSerializer.instantiate(FSTArrayListSerializer.java:63)
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadWithSer(FSTObjectInput.java:481)
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:359)
    at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:683)
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:537)
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:363)
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:325)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:305)
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:239)
    ... 10 more

Hello,

I am getting an error when I serialize a private nested class, because its parent (the one where the inner class is defined in) is not serializable. I was not expecting that the parent class would also be serialized, this seems a bit odd ... and possible not intended? Moving the particular class to its own type resolved the issue, but we might have more cases like this yet to be detected.

This is on version 1.55 (JRE 6 compatible version).

Please advise on the best way to proceed.

thanks!

Hi, I have issue with your library. Unfortunately, I can not post here any source code to be more specific. It seems, when you deserialize my object which is not marked as Serializable, you read incorrect class name from stream which contains also data which are no more part of class name.

During my investigation I find out that in FSTStreamDecoder.readStringUTF() you are using readFInt() which returns incorrect value. In general, this can be also problem of serialization process, who knows. For more information please contact me at: stehlik.michal(at)gmail.com.

Although stated as Android compatible, I ran into an exception today:

java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[na:0.0]
            at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:875) ~[na:0.0]
            at org.hive2hive.core.security.FSTSerializer.deserialize(FSTSerializer.java:59) ~[na:0.0]
            at org.hive2hive.core.security.H2HDefaultEncryption.decryptAES(H2HDefaultEncryption.java:43) ~[na:0.0]
            at org.hive2hive.core.network.data.vdht.EncryptedVersionManager.get(EncryptedVersionManager.java:178) ~[na:0.0]
            at org.hive2hive.core.network.data.UserProfileManager$QueueWorker.run(UserProfileManager.java:203) ~[na:0.0]
            at java.lang.Thread.run(Thread.java:841) ~[na:0.0]
     Caused by: java.io.IOException: java.lang.RuntimeException: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:240) ~[na:0.0]
            at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:873) ~[na:0.0]
    ... 5 common frames omitted
    Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:327) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:304) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:238) ~[na:0.0]
    ... 6 common frames omitted
    Caused by: java.lang.NoSuchMethodError: sun.misc.Unsafe.putBoolean
            at org.nustaq.serialization.FSTClazzInfo$FSTFieldInfo.setBooleanValue(FSTClazzInfo.java:858) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:634) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:515) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:657) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:515) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[na:0.0]
            at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:323) ~[na:0.0]
    ... 8 common frames omitted

Unfortunately, the Unsafe implementation of Android is incomplete and not all methods are available.

I tested with FST v.2.21 and Android 4.4.4. For the compilation, Android Studio with Java 7u71 is used.

Hi,

I'm trying to clone (deep copy) a complete object graph fetched from DB using Hibernate. The thing is that each entity should be considered as a new one ('transient' in Hibernate lingo). Therefore I had the plan to set each entity's ID to 'null' and it's version property to zero.

My setup is currently like this:

first the global config:

Some 'hook' to ensure that we're working with a 'real' object instead of a Hibernate proxy:

Set a callback to erase each ID and version property for each entity that contains a @Conditional annotation:

The 'ProxySafeObjectOutput' is working fine; However, I'm experiencing issues with the callback that is invoked upon deserializing the object graph. Almost each entity in the graph contains a reference to an other one; it also contains lot of cyclic references where each reference is annotated with '@Conditional'. I'm constantly having 'NullpointerException's upon deserialing; I'm sure that it's related to the '@Conditional' annotation because if I remove it everything works (besides resetting Id and version fields).

Is there an other way (some hook perhaps) that I can use to check if an entity field should be skipped?

I'm invoking the deep clone as follows:

Any ideas?

First of all, thanks for this great serializer! I'm using it for every project I find a reason for because it is one of the best and most convenient one.

Today, I updated to the newest version (2.19) and also switched to the latest BouncyCastle version (bcprov-jdk15on:1.51). Serializing and deserializing a keypair generated with bouncy castle provider seems to run into an error. For your convenience, I created a Github Gist here.

Do I need to use a custom serializer to prevent this error (i.e. serializing keypairs manually)? Thanks for your help in advance!