Worth checking out are Java libraries made by Real Logic guys:
Aeron - Efficient reliable UDP unicast, UDP multicast, and IPC message transport
Simple Binary Encoding (SBE) - High Performance Message Codec
Agrona - High Performance data structures and utility methods for Java