Ziti: Programmable network overlay and edge components for zero-trust networking | Hacker News Reader

Ziti: Programmable network overlay and edge components for zero-trust networking

github.com

156 pointstalonx42 comments

All of this somewhat recent new activity that exposes easier user defined networking makes me wonder about corporate Cybersec departments. Are they trying to keep all this stuff in a box, control it, etc? I know none of it is really new per se, but it is certainly easier to do now.

I know that some of it is fairly easy to detect, but Cyber also can't use the same old stranglehold techniques[1] they have in the past, because remote developers need to be able to use docker and other tools that use network overlays.

The old school approach of trying to block it all is based on, I assume, old style networks where the corporate office floor network has too much access to production. And so, the corporate VPN inherits too much access also, so it works similarly to your desk.

Perhaps this pushes more effort to make the VPN and office floor networks completely separate from anything important.

[1] For example, popular corporate VPN software products, like AnyConnect and GlobalProtect, are somewhat notorious for blocking things like Docker overlay networks by default.

I am a dev on the project... The kinds of people I engage with aren't necessarily trying "to keep all this stuff in a box, control it, etc?". The cool thing about OpenZiti is that it's a split vpn by default. Only the traffic you want going toward your services can go there from the users who are supposed to get to it (core tenants of zero trust principles).

I've seen some companies try to go the route you're thinking. There are definitely high-security type of solutions that want, and need that level of control over the network. They are building solutions around OpenZiti to enable the observability part of the equation you're hinting at.

Generally, though, OpenZiti is ideally targeting applications. We'd love it if every app on the planet had their own bespoke, ubiquitous network connection which the developers could control. The network engineers can control the underlay like they want while the developer can control access to the overlay/apps like they want. Ideally, that would be a best of both worlds situation.

> And so, the corporate VPN inherits too much access also, so it works similarly to your desk. This! 100% nailed it. That's why the split vpn aspect is so important.

True, the tools used by risk managers will have to adapt to overlays if overlays (software defined networks that look like encrypted noise on the wire) are as inevitable as they appear to be. It's kind of an arms race and it only makes sense to embrace what's happening and invest in those tools that bring the controls and insights that are needed by defensive security interests to do their job.

At the same time overlays move security closer to the app which shifts the purview left even further toward the developer. This might mean that developers will bear more of the burden of ensuring a secure deployment by choosing a good overlay and effectively operating that overlay in a way that satisfies Security and Compliance.

What you’re describing is the industry fad phrase “zero trust,” where the industry as a whole is shifting towards untrusted endpoints and networks with proper access control on every layer.

I think the shift away from trusted networks is long overdue and inevitable not only thanks to the needs of engineers and other employees, but especially since endpoints (and employees!) are so easily compromised anyway.

There’s a reasonable case to still block encrypted overlay networks from a defense in depth standpoint, since they could make detecting a compromise harder, but I think that at some point most places will stop bothering and shift towards monitoring systems instead of the network as well.

> [1] For example, popular corporate VPN software products, like AnyConnect and GlobalProtect, are somewhat notorious for blocking things like Docker overlay networks by default.

It’s not that they block them per se, they are just ignorant of them and so they can’t route them. Traditional VPNs rely on routing protocols like OSPF and BGP (with the occasional static route thrown in) to move traffic, and your docker overlays aren’t integrated with the netsec team’s routing infrastructure.

No, they actually block them by actively inserting their own routes at a higher priority and inserting firewall rules. You can fool around with LD_PRELOAD to neuter their functions that do this and get everything working end-to-end. They are blocking things that would otherwise be NATed and work just fine.

There's also workarounds for Windows using Powershell scripts and "Set-NetIPInterface -InterfaceMetric 6000" to lower the priority of the AnyConnect internal routes that break docker, or using wsl-vpnkit (as DockerDesktop does).

Have you written any of this up in more detail, or have relevant links to share?

Only if you configure them to do so. I created our Globalprotect solution and we use docker extensively without issues. It just sets a default route (0/1 and 128/1) and users just need to NAT from the docker network for it to work.

Well, "you" being the entity that controls the head end of the VPN, which controls settings not (always) tweakable by the end user.

Because I've found OpenZiti kind of hard to wrap my head around, I'm repeating my TL;DR from another recent discussion (https://news.ycombinator.com/item?id=32923851#32942158):

It is a meshed overlay with endpoint authentication and ACLs. Endpoints can be: application-embedded (think TLS) or system level (think PtP VPN) or routers to subnets (think routing VPN).

One thing that took me a long time to wrap my head around is: You can incrementally implement OpenZiti by: setting it up as a traditional VPN, then start putting individual server endpoints directly on OpenZiti, then put individual services directly on the fabric (application embedded).

I wished I can give you +5 upvoting, thanks for making clear on its purposes. I guess now there are two OpenZiti related articles on HN front page as a direct responses for the Cloudflare's elephant in the room zero trust eSIM news, the explanation is rather timely.

> Because I've found OpenZiti kind of hard to wrap my head around

It sounds like, from your description, that it’s somewhat like the (very HN popular) Tailscale VPN but it can also operate at “a higher level” and be built right into applications instead of needing OS networking privatives?

OpenZiti has some similarities to Wireguard/Tailscale... lets give s description of the former as thats then opensource vs opensource.

Wireguard is built to be a better VPN and really cares about "connecting machines" and not so much about connecting "services". OpenZiti cares about connecting "services" with zero trust networking concepts, including least privilege, micro-segmentation, and attribute-based access (though you can also set up a whole CIDR if you want). OpenZiti also uses the embedded identity to build outbound only connections into a mesh (think Cloudflare tunnels), so we can close all inbound ports. This can all be surmised as WG being 'default-open' whereas ZT is 'default-closed'.

Wireguard uses UDP and hole punching to build P2P connections, while OpenZiti uses TCP and a mesh overlay (with the outbound only at source and destination). This is how Tailscale implements Wireguard to ensure it works easily in all situations. It also allows you to control the internet routing and provide higher redundancy, resiliency and control for routing traffic according to policy (e.g., low latency or geo-restrictions).

Due to OpenZiti's uses of identity in the endpoints and fabric for routing, you also get a private DNS and unique naming (e.g., send from IoT endpoint service to IoT server rather than from 192.xxx.xxx.xx to 100.xxx.xxx.xx). This also means we do not need to use floating or static IPs, easily handle overlapping, no need for port forwarding.

Finally, where it really differentiates is that with OpenZiti you canstart with "network-based zero trust" (installing a router in private IP space) and progress to "host-based zero trust" (using an agent/tunneller), it also have a suite of SDKs to embed in apps themselves for "application-based zero trust". This allows it to run clientless, in serverless, in confidential computing and more.

P.S., Wireguard get a lot of well-deserved love! OpenZiti uses the Windows TUN (WinTun) that the Wireguard project made as (at least) part of our Windows tunneler. Thanks, Wireguard!

Yes, I think that's a fair statement. I'd say it's closer to ZeroTier, because ZeroTier has ACLs and the "libzt" that you can embed in applications. Though in the case of libzt you are embedding a full TCP stack and talking on ports, where I get the impression OpenZiti is more like a pipe with credentials.

Or: It's like Tailscale, if Tailscale had mutual TLS client/server baked into it as well as VPN.

> OpenZiti is more like a pipe with credentials

That's a better way to think about it, I can see that. The idea of authorizing before connect to an open, listening port is great, but not having a listening port at all in your app is so much more powerful imo. It makes it all that much harder for worms and ransomware and the like to propagate when there are no listening ports to attack. OpenZiti uses a full TCP stack in the tunneling apps - the agent-like things you would install for brownfield type of things but for app embedded zero trust it doesn't need any of that.

Tailscale would also need to natively provide segmentation, least privilege, attribute based access, endpoint posture checks, and do authentication/authorisation before connectivity is established. I believe they recommend a firewall does some of this. Also the SDKs.

For the HN readers,

https://support.netfoundry.io/hc/en-us/articles/360019471912...

from the Ziti support pricing page:

* Standard - To be retired soon. Not applicable for the latest pricing plans ( since 2022)

Does this mean you are required to be using Ziti infrastructure just to use this product or its SDK? Or it is self-hostable?

That's the SaaS stuff. NetFoundry is the company behind OpenZiti but OpenZiti is 100% open source and you can go install it and run it all yourself.

https://openziti.github.io/ziti/quickstarts/quickstart-overv...

It's also fully featured, no nerfing etc

And fwiw - a better link that talks about the SaaS is actually the pricing link https://netfoundry.io/pricing/ for anyone interested in "not" hosting it themselves... :)

responses* to all the great discussion:

1. the problem solved by the openziti platform is making secure networking simpler and stronger for dev, ops and sec teams.

2. to solve this problem, openziti provides functions such as mtls; strong IDs with automated enrollment; outbound session initiation (initiated by client or server side and bridged by proxies...such that inbound firewall rulesets become deny-all); programmable cloud native overlay network which only accepts cryptographically authorized sessions.

3. app-embedded via OpenZiti SDKs means no separate agent required. your private overlay network goes wherever your app goes. other options include agents and gateways, as well as embedding in browsers, proxies, etc.

* i work for netfoundry - we built saas on top of openziti

What does "zero-trust" mean?

I'd presume I want a network I can trust.

what a great, great question. it's a horrible term. really it means you have "no trust of the network, so you verify it before allowing connections"... The term is basically the OPPOSITE of what it is.

You use a strong identity and you verify that cryptographically verifiable identity before allowing connections to your network, then on top of that you have a separate authorization tier for that identity to access services.

it's a horrible term... :)

A perspective from someone who hasn't been super involved in network security:

Historically, many folks would treat a VPN as something like a moat around the castle. If you're on the VPN, you can access stuff, and if you're not on the VPN, you cannot. However, this leads to a sort of ambient authority: break into the VPN, and you've broken into anything.

"Zero Trust" is a model closer to capability-based security than to ambient authority: just because you're inside the VPN does not automatically mean that you have access to the servers inside of it, you must also authenticate with them, specifically. It's "zero trust" because no user is trusted by default, not because you as a user cannot trust the network.

If you're interested in learning more about it in video (still need to write the content up in 'blog/article' form) you can watch me blather about it at the beginning of the talk i gave to developer week... https://www.youtube.com/watch?v=kVW2SOETP90

I actually use that castle/moat analogy and talk about how that's how vpn's work... it's a tried and true example. :)

Thus, zk is akin to all doors within the castle being locked and only the appropriate persons can access their respective areas: chefs -> pantry; knights -> armory, etc.

Aspects of it yes, that would be the principles of micro-segmentation and least privilege. There are other aspects of ZT principles, e.g., software-defined perimeters which OpenZiti delivers where this anology does not work. For example, SDP allows OpenZiti to do authentication/authorisation before any connectivity is established with outbound only connectivity as well as not even trusting the host OS if you use an SDK. I wrote a blog on this a few months back - https://netfoundry.io/demystifying-the-magic-of-zero-trust-w...

Zero Trust Network means a network without an implicit trust. Just because a device is present on a corporate office network doesn't mean it has access to all company resources.

It's very common to have site-to-site connections and allow everything in-between or slightly limit things in between. Sometimes it even just allow-listing certain IPs like office public IP or employees home IPs.

In a zero trust environment there are no such assumptions made, in fact, the only assumption that is made is that the network is already breached. No one is implicitly trusted, every access to a resource must authenticate, audit logs must be secured.

Since this was unbelievably hard to find: It's not a VPN, it's a TCP tunnel. https://news.ycombinator.com/item?id=32928222

ZeroTier alternative ? Has the team done any performance measures ?

Yes, it is a zerotier alternative but there are key differences in how we do somethings... in fact, its on my list of things to do for creating some of these comparisons.... in the mean time, here is some comments on Ziti vs others - https://www.reddit.com/r/selfhosted/comments/v1ymn5/when_pub....

We did do performance testing too ... OpenZiti is built for high performance - https://netfoundry.io/benchmark/benchmarking%20open%20source...

zero tier is layer 2 - openziti is layer 3, 4 or 7 depending on what you're doing. it's similar to zero tier in ways, but very, very different in others.

openziti's main goal is to bring application embedded, zero trust into applications but getting there is a long journey. that's why we provide "agents" like other "better vpns" like zero tier, wireguard, etc

Have you tested large numbers of endpoints? Sometimes performance bottleneck isn't the data throughput but the connection establishment phase.

Eg if a smart phone app rolled out to 10k+ users, or an IoT service with 100k+ devices, will the service be able to handle it?

A bunch of us (me included) used to work in the IoT world. You can *TRY* to simulate 100k assets using "20 or so" nodes but truthfully it's just never *really* the same. So, to be transparent, no. It's really, really, really hard to test 100k devices effectively in my experience. (happy to be told otherwise/taught what others have done). We ran hundreds of __actual__ machines simulating "thousands" of devices, but it's still NOT the same... We do perform scale testing though it's not me that does it...

So no, we haven't gotten to 100k actual deployed devices - yet. You can be the first! :)

As every developer will say, we "built it for scale". Many of us are also a bunch of ex-IoT devs, and we _have_ built systems like this in the past so we're really familiar with the types of issues that can crop up.

Verified users count, we're in the 5000 to 10000 range that I know of (as in, I am pretty sure we have networks of that size deployed out there in the wild). I'll ask what our "fabric" people have tested and how and get back to you if it's significantly different than what I know about.

Great response, thank you.

I'll add some more detail. There are a lot of different ways your application can break as it scales up. From something which is handles data flow, the three I tend to think about the most are: the model, throughput and number of connected clients.

We've tested the model with 100k identities with the operations that clients use: auth, listing services, creating sessions, etc, to make sure that the model scales reasonable well. We had to add additional denormalization and make some other tweaks, but now the controller scales relatively well for those operations. I'm sure there are still edge cases where it may break down, but we'll have to fix them if someone hits them.

We've done thoughput testing to make sure we can handle high throughput use cases. This also resulted in lots of changes, including reworking the end-to-end flow control. This is an area where we're happy with the progress we've made, and performance is in a reasonably good place, but where we have lots of ideas of how to continue to improve and will be continuing to test and iterate.

Having tons of connected clients (even without much traffic flowing) is it's own scaling challenge. We've done some amount of testing here. As part of the testing mentioned above we had to make some changes to make sure that slow clients didn't hold up fast clients. More generally,this is where things start getting complicated and very specific. You can have very different types of traffic flow, so it's hard to model anything generic. We have not done as much in this area because we've not seen any cases where we're memory constrained, which is the usual sign of a concurrent connections scaling bottleneck.

Hope that's helpful.

Thanks! From the benchmark report [1], it is not clear how much of the baseline wire performance is observed, the numbers in the table are anywhere from 30% to 90% of plain wire bandwidth.

As there are multiple overlay projects popping up - Tailscale, NetMaker, OpenZiti, NetBird, Nebula, ZeroTier, EVPN, etc, we should consider a baseline benchmark index, like [2].

P.S. We have been testing ZeroTier for VPN access and observe ~70% baseline wire bandwidth.

[1] https://netfoundry.io/benchmark/benchmarking%20open%20source...

[2] https://techoverflow.net/2022/08/19/iperf-benchmark-of-zerot...

Nice! Sounds like OpenZiti is your next one to put on that list? We would __LOVE__ to have a third-party do any kind of performance testing like that and report the results. Good -- or bad! It's important to be transparent in things like this, we believe that wholly.

If you want any help, we'd be happy to help you setup a network (if you need it), just ask over on the discourse!

Is this an alternative to:

- ngrok

- ssh tunnel

- tor

- libp2p

It's similar to them, sure. you could build an ngrok alternative on top of OpenZiti. In fact we might be working on something similar that you might seeing as a Show HN soon :)

SSH tunnels are similar in ways, yes but they'll suffer from the same "you need an open open port" problem that OpenZiti doesn't suffer from (excluding the overlay nodes, I can get into that if you want)

tor is an anonymization network - utterly different. With OpenZiti the network needs, and must know what identity is connecting. it's really not the same, very different. some similar privacy type of overlap might be out there, but in general that one feels really different.

libp2p is new to me. I have seen that coming up lately but I don't know much about it yet so i can't really comment on that one. sorry

The similarity with tor is that you can connect to any other computer via its public key, which corresponds to a .onion address.

If your computer's public key is xx73u2, then I can access it at https://xx73u2.onion/and/whatever/path/i/want, and tor handles all the encryption. From this perspective, anonymity comes as a bonus.

Oh well see - TIL that was a thing for tor :) Thanks for the education, I always appreciate that!!!

So it's similar in that way then, indeed. Including e2ee. I guess I need to brush up on tor's capabilities again. I'm sure there are other overlaps and other differences I'm just not all that aware of.

I have seen this comparison to Tor before. There are similarities related to the mesh overlay which OpenZiti operates. Sessions are routed according to a policy which can be security or performance-based (natively its routes to the lowest latency). Nodes can be in any location around the world. Packets, as they flow, are synthesised to look like 443 (i.e., cannot determine the app type based on ports) and have metadata encrypted so that the only thing visible if someone intercepts are the public IP of the next hop, nothing related to source, destination or that its OpenZiti type traffic.

The major difference is that Tor is designed to be a decentralised, anonymous internet whereas OpenZiti is a private overlay operated with trust and identity. But to an outsider to an OpenZiti network, they would know little about what was going on like Tor.

Reply on news.ycombinator.com