Hacker News Reader

Hacker News

Comment by lenova | Hacker News Reader

Full thread

lenova

Nebula (a distributed mesh overlay network first developed at Slack) released their own network benchmarks today:

https://www.defined.net/blog/nebula-is-not-the-fastest-mesh-...

This is in response to this oft-cited benchmark post by Netmaker:

https://medium.com/netmaker/battle-of-the-vpns-which-one-is-...

Previous discussions on encrypted overly mesh networks here:

"Would we still create Nebula today?" (Oct 2023) https://news.ycombinator.com/item?id=37871534

View on HN

It should be noted the Netmaker benchmarks were ran in a shared cloud environment, so the throughput related metrics must be thrown out. You have no visibility into what other workloads are running on the same physical host or how your test hosts are physically connected. As a result you could be measuring software X with two instances on the same bare metal and software Y across two instances in different buildings (yes instances can move buildings without you knowing).

A proper test would be done with bare metal machines with 10/40/100G NICs and a controlled network plane free of other traffic.

Not to defend the netmaker article too much (its pretty crappy), but the noise of shared cloud infra is not unmanageable, it just needs to be factored in in the plan and analysis. Just repeating the tests couple of times, preferably in different regions should give pretty meaningful results. Also recognizing the scale of things, noisy neighbors are extremely unlikely to cause order of magnitude difference, especially when operating at < 1gbps scale. Lastly, just using good instance types helps a lot here too, the T types netmaker used were pretty much the worst possible option. The table in the docs provides guidance:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-...

Note how some types have "up to X Gigabit", some are just "X Gigabit", and for t2.micro that netmaker used the docs say just "low to moderate" :D

But all that being said, I think it is valuable to benchmark things in noisy real-world conditions too and not only in pristine lab conditions. It definitely requires bit more effort in the analysis side to understand whats going on. And benchmarking without good analysis is pretty bad regardless if it was done in public cloud or some dedicated lab.

One more thing, from the docs you can also infer the level of overcommit of the networking of the host by looking at the documented perf of .metal instances. For example m6in.8xlarge has 50 Gbps networking and m6in.metal has 200 Gbps. But also m6in.metal is equivalent to m6in.32xlarge in terms of capacity, and 32xlarge is 4 times bigger than 8xlarge. So from that you can easily tell that m6in.8xlarge has no overcommit for networking because the network and CPU/mem allocation match exactly. At the same time you can tell that m6in.large does have network overcommit; m6in.large is 1/64th of the .metal instance in terms of size but 25 Gbps × 64 = 1600 Gbps, 8 times more than the 200 Gbps of the .metal instance!

Finally EC2 has controls for instance "tenancy" and "placement"; you can ensure that your instances are run on hosts dedicated to your account, again avoiding host-level noisy neighbours. Of course both of these aspects only control the physical host, the networking infra outside the host is afaik almost completely out of control. But at the same time I'd be surprised if there is any significant level of congestion within an AZ.

This is all to say that there is whole spectrum of things between "using random public instances with no consideration" and "set up private dedicated hardware lab", its not just binary choice.

Reply on news.ycombinator.com