No love for tinc[1]?
It's the granddaddy of mesh networking, long before Wireguard, and while it's not quite zeroconf, it's very simple to setup and maintain. It also runs on everything.
It's the granddaddy of mesh networking, long before Wireguard, and while it's not quite zeroconf, it's very simple to setup and maintain. It also runs on everything.
I actually looked into tinc, and really wanted to love it because of its simplicity.
Unfortunately though, it seems that the development scene around it has been stalled for several years now, and benchmark reports show that it can't keep up with cloud speed tests when compared with Wireguard:
- Wireguard: 390.3 Mbps
- Netmarker: 369.3 Mbps
- Tailscale: 62.5 Mbps
- ZeroTier: 56.8 Mbps
- Nebula: 38.4 Mbps
- Tinc: 34.7 Mbps
- OpenVPN: 22.3 Mbps
Source: [disclaimer, article written by Netmaker CEO, so implicit bias]
(https://medium.com/netmaker/battle-of-the-vpns-which-one-is-...)
I found this other benchmark[1] that places them much closer to native WG performance, with Netmaker still in the lead, and somehow faster than a direct connection. This is probably due more to Netmaker using the in-kernel WG, instead of the userspace implementation. I'm kind of curious to give it a try, TBH. :)
Hell, even OpenVPN is _much_ faster than that[2,3].
Also, yeah, tinc development has been slow for years now, but I didn't experience any issues with the prerelease versions. It's certainly behind the times now, but I also enjoyed how easy it was to configure and use.
[1]: https://techoverflow.net/2022/08/19/iperf-benchmark-of-zerot...
[2]: https://www.zerotier.com/blog/benchmarking-zerotier-vs-openv...
The folks over at the Nebula team had an interesting discussion regarding the the "battle of the VPNs" article published by Netmaker I sourced in my parent comment:
And really I've been using tinc for almost a decade and I didn't really see the benefit of changing. It's rock-solid.
With the exception of one thing: I use some central nodes on cloud VPSes and they can access everything. As far as I know a nebula lighthouse can't access any of the clients. So I've been meaning to give nebula another try.
But zerotier and tailscale aren't options for me because they rely on their cloud infrastructure. I only want stuff that's fully self-hosted.
There's a great tinc android client these days too.
Unfortunately, I cannot confirm. Sharing my experience:
I used tinc over multiple years on production servers and it would sometimes create netsplits that did not recover. I also suspect that there's a race or bug in re-keying, which also causes disconnects.
On the netsplit issue, it was me posting alone on the relevant issue [1] over multiple years without response. (I don't expect to get any from free-time maintainers, especially on hard-to-reproduce issues, but it's still important to know that such unsolvable hurdles exist.)
When I switched to Nebula, it improved this situation. But both Nebula and tinc max out at around 1 Gbit/s on my Hetzner servers, thus not using most of my 10 Gbit/s connectivity. This is because they cap out at 100% of 1 CPU. The Nebula issue about that was closed due to "inactivity" [2].
I also observed that when Nebula operates at 100% CPU usage, you get lots of package loss. This causes software that expects reasonable timings on ~0.2ms links to fail (e.g. consensus software like Consul, or Ceph). This in turn led to flakiness / intermittent outages.
I had to resolve to move the big data pushing softwares like Ceph outside of the VPN to get 10 Gbit/s speed for those, and to avoid downtimes due to the packet loss.
Such software like Ceph has its own encryption, but I don't trust it, and that mistrust was recently proven right again [3].
So I'm currently looking to move the Ceph into WireGuard.
Summary: For small-data use, tinc and Nebula are fine, but if you start to push real data, they break.
[1]: https://github.com/gsliepen/tinc/issues/218
[2]: https://github.com/slackhq/nebula/issues/637
[3]: https://github.com/google/security-research/security/advisor...
My tinc usage was far less demanding. Just a handful of nodes and light traffic, so I didn't experience any issues.
I did migrate to WireGuard about a year ago, and instead of a mesh network, I ended up with a hub-and-spoke configuration[1] which worked fine for my humble needs.
[1]: https://www.procustodibus.com/blog/2020/11/wireguard-hub-and...
I've reopened the issue so you can continue to document and look for solutions. :)