Thanks! We're using binary websockets to route the actual video frames. There's also an instance of haproxy sitting in front of everything. But our server is still choking every one in a while, heh.
Got my latency down to < 1s (same city) with an IP cam with built-in MJPEG stream, served by node.js as an ever updating static jpg. On the client side I then used a simple requestAnimationFrame-script to update the image source as often as the client would allow.
We're located in Flint, Michigan and using a DigitalOcean box hosted in NYC. I haven't run any latency tests, but I'd ballpark the number to be in the 250-500ms range. I was blown away that it worked as well as it did.