Hacker News Reader

Comment by zozbot234 | Hacker News Reader

> All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud.

It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out", but that's only really feasible for relatively simple workloads where you just seek to do away with the whole issue of keeping different nodes in sync on a real-time basis, and accept the resulting compromises. In many cases, that's just not going to be enough.

View on HN

There are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe.

Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multiple exchanges around the world.

Source: I used to be head of Global Latency Monitoring for a HFT.

Hey I’d love to pick your brain about what it takes to do this kind of work coming from a web/systems engineering background. Shoot me an email at samheutmaker at gmail if you’re open to it.

I'm curious why you moved to LLNL from HFT?

Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.

It sounds like the opposite direction happened here.

Umm, most workloads aren't so compute intense that cpu interconnect speed would have a huge impact. The vast majority of rented computer tasks will be for things that need to communicate with external services (payments, region synchronizing, general api state comms, etc). In these cases, I doubt that your code is tuned tightly enough to reveal interconnect latency (as opposed to cache misses and swaps). Besides, hooking up interconnect runs gets ludicrously complicated and expensive fast. Like, $50k in cables and a week of experimenting with numa zones and kernel settings for a small cluster. I can only imagine the headaches of orchestration on national lab clusters.

Source: I have set up, and currently manage a small cluster which is always pegged. I also collaborate with folks at a couple of the national labs for scientific research.

Infiniband and friends aren't cpu interconnects, they're alternatives to ethernet. Network latency is important and it's not that hard to write code whose performance is bounded by io latency/throughput.

Well they can be both, so I can see how parent might have gotten confused.

True enough.

> it's not that hard to write code whose performance is bounded by io latency/throughput

I suspect that most people don't, though. How many applications are built in Ruby, Python, or PHP, using some very productive but inefficient framework? Those are all very fine tools in their own way, and are the right tool for the job in many contexts. But they're unlikely to saturate IO.

Those technologies, or mostly IB, were more important when Ethernet speeds lagged behind. IB was first to 100G. Now that Ethernet has caught up, IB has almost no advantage besides a tiny latency decrease. Even mellanox makes more Ethernet than IB products now.

I thought process-to-process, Ethernet has still several times larger latency than IB?

We're talking about a few hundred nanoseconds usually. If that matters, then IB is still better.

Most business workloads simply aren't CPU bound, nor amenable to parallel computation. The public clouds would offer it if they had demand for it.

Surely network latency (and perhaps throughput) is just as relevant to IO-bound and transactional workloads though? At least if they need to be served by a multiple-node cluster.

Reply on news.ycombinator.com