There are service architectures where putting them behind an ELB is not an ideal scenario (an example would be RabbitMQ; works great with haproxy in front of it, not so much with ELBs). ELBs aren't always a perfect fit, and depending on what sort of connection lifecycle or scaling velocity you require, direct addressing of service instances might be required.
WebSocket. ELB adds latency and is harder to debug.
Have you considered using NLBs instead?
That was before NLBs. But yeah, NLBs might be a good choice now.