Serialization

Serialization Configuration and Tuning

As was previously mentioned, Dempsy is built on a set of core abstractions. One of those abstractions is Serialization. This means the entire serialization scheme for the framework is pluggable. If you're interested in plugging in custom serialization then see the section on Framework Development and specifically the section on the Serialization Api.

Dempsy comes with two serializer implementations. The first is straightforward java serialization. The second is Kryo, which is the default if you don't specify it in the Application Definition or Cluster Definition.

Java Serialization

Note: Currently the default implementation for the Serializer is Kryo. Kryo, when used correctly, can be 8 to 10 times more efficient, in terms of both network traffic and raw serialize/deserialize performance, than straight Java serialization. Please do not use Java serialization for any production systems that require efficiency unless you plan on hand optimizing all of your message serialization. You have been warned.

As the name implies, Java serialization is simply an implementation of the core abstraction that uses standard java serialization. Therefore, in order for it to work, all of your message classes and their attributes need to implement java.io.Serializable. If they don't you will get send failures in Dempsy.

In order to enable java serialization you simply provide the implementation to the Application Definition (which is over-ridable in the Cluster Definition as described in the Application Definition section. An example follows:

...
<bean class="com.nokia.dempsy.config.ApplicationDefinition">
   <constructor-arg value="ApplicationName"/>
   <property name="serializer">
       <bean class="com.nokia.dempsy.serialization.java.JavaSerializer"/>
   </property>
   ...

Kryo Serialization

If you don't do anything, Kryo will be used for the default serialization. However, Kryo can be made much more efficient with a little bit of configuration.

Note: In order to use the defaults, every class that gets serialized must have a default constructor. Kryo will work with private default constructors as long as the security settings allow for it. For more details on Kryo's requirements for object creation, see the Kryo documentation on object creation

Class Registration

The easiest way to optimize Kryo is to provide the configuration with an ordered list of all of the classes that will be serialized as part of the application. The order of the list should be most-frequently serialized to least frequently serialized. Note: it is critical that all senders and receivers have the same list in the same order and the best way to make this happen is to make this list part of the Application Definition.

In order to list the classes you will provide the instance of Dempsy serializer in the Application Definition just like in the case of the Java Serializer except we will include additional parameters. For example, the Application Definition for the Word Count example could look like this:

<bean class="com.nokia.dempsy.config.ApplicationDefinition">
   <constructor-arg value="WordCount"/>
   <property name="serializer">
      <bean class="com.nokia.dempsy.serialization.kryo.KryoSerializer" >
         <constructor-arg>
           <list>
             <bean class="com.nokia.dempsy.serialization.kryo.Registration" >
               <constructor-arg value="com.nokia.dempsy.example.userguide.wordcount.Word" />
             </bean>
             <bean class="com.nokia.dempsy.serialization.kryo.Registration" >
               <constructor-arg value="com.nokia.dempsy.example.userguide.wordcount.CountedWord" />
             </bean>
           </list>
         </constructor-arg>
         ...

Note the Word class is listed prior to the CountedWord class. It's safe to say, given the nature of the Word Count application, that there will be many more Word messages than CountedWord messages and therefore, Word is listed first.

By default the Kryo serializer will work with unregistered classes, however, there is a way to tell the Kryo serializer that all classes must be registered. Since registered classes are so much more efficient, during development it is a good idea to set this option. Setting this options causes serialization to fail on any unregistered class. This will require the developer to register the class to resolve the problem and thereby assure that all classes are registered.

To enable this option you set the following property:

...
      <bean class="com.nokia.dempsy.serialization.kryo.KryoSerializer" >
       ...
          <property name="kryoRegistrationRequired" value="true" />
          ...

Advanced Configuration

Dempsy provides a hook into the Kryo serializer that allows for the application developer to get access to the underlying Kryo implementation and thereby opens up all of the underlying Kryo options to the developer. You will need to be familiar with Kryo itself in order to use it but if you want access, you need to implement the com.nokia.dempsy.serialization.kryo.KryoOptimizer interface.

The implementation of the com.nokia.dempsy.serialization.kryo.KryoOptimizer interface will be given the central com.esotericsoftware.kryo.Kryo instance both before any registration happens, and then again afterward. Things to keep in mind:

1. If you want to modify or add to the com.esotericsoftware.kryo.Kryo instance before any Registrations are applied, implement the void preRegister(com.esotericsoftware.kryo.Kryo kryo) method to do that. Otherwise implement it to do nothing.
2. If you want to modify or add to the com.esotericsoftware.kryo.Kryo instance after all of the Registrations are applied, implement the void postRegister(com.esotericsoftware.kryo.Kryo kryo) method to do that. Otherwise implement it to do nothing.
3. Dempsy's Kryo based serializer pools com.esotericsoftware.kryo.Kryo instances since they are not thread safe. They are created 'as needed' and therefore the KryoOptimizer will be used each time a new com.esotericsoftware.kryo.Kryo instance is created and pooled.

To provide your implementation of the KryoOptimizer you set the property on the Serializer as follows:

...
      <bean class="com.nokia.dempsy.serialization.kryo.KryoSerializer" >
       ...
          <property name="kryoOptimizer">
              <bean class="com.mycompany.myapplication.MyOptimizer" />
          </property>
      ...

Next section: Monitoring