Low Latency

This is a specialized topic, and will only be useful developers who need to do extreme optimization in Java!

The vast majority of Java applications do not need to be optimized for low latency. There are however certain situations when you need to handle an exceptionally high volume of data and need squeeze every last ounce of performance out of Java. There are a number of techniques to optimize for low latency, but they place significant restrictions on the code you can write.

Garbage Collection

The garbage collector is a fundamental feature of the JVM. It is incredibly useful, but is also a black box and without careful tuning it can trigger the nightmare 'stop-the-world' garbage collection.

When optimizing a Java application for low latency, it is import to make every effort to minimise garbage collection.

Allocate Objects At Startup.

So our first optimization technique is to allocate the objects you need for your service into heap memory at startup. Garbage collection locates objects which are no longer referenced and deallocates them from heap memory. By allocating all objects during startup and maintaining strong references to them, the garbage collector can not collect them!

Object Pooling

The objects should be stored in a pool. When a new object is needed, an existing one is retrieved from the pre-allocated pool of objects. When that object is no longer needed, it is returned to the pool. This is a very unusual technique in Java, as the garbage collector is (usually) fundamental to Java development.

// During start up we create and populate the pool.
ObjectPool<T> pool = new ObjectPool<>();
for (int i=0; i<1000000; i++) {
   pool.add(...);
}

// When we need an object, we do NOT create a new one, we get it from the pool
T instance = pool.next();

// When finished with the object, we must release it back to the pool for re-use
// We must not allow it to be collected
pool.release(instance);

There is one other important aspect to this technique. The pre-allocated objects must be mutable. Working with primitive and enum fields is straight forward, but there will likely need to be an object pool for every other object type used!

A key issue is that it is highly unlikely you will be able to use classes like String. Unless you already know all possible values a string can have (and they can fit in memory), you will need to create a mutable String equivalent data structure and pool it instead.

This technique is not for the faint-hearted, and should only be necessary where data is being handled at exceptionally high volume (e.g. FX price updates).

Multithreading

Another fundamental feature of Java is its ability to perform multithreading. Performing operations in parallel is exceptionally useful and allows for applications to handle many different activities at the same time, improving performance significantly over single-threaded execution.

Unfortunately the usual approach to threading results in a large amount of context-switching, an operation that is very costly when you are trying to optimize for low latency.

One Thread Per Core

So our second optimization technique is to restrict the number of active threads to the number of available cores. Threads need to be coded with great care to prevent them context switching at all if possible.