Open-source firmware
mullvad.net
mullvad.net
We (Private Internet Access) are working on something similar called zero access. In response to your call to collaborate with other VPN providers, I would love to get on the phone with you to trade notes.
My contact info is on my profile here! Hopefully looking forward to it!
As far as innovation goes we published our vision for the VPN industry a few months ago. It's transparent and independently auditable VPN servers, through the use of System Transparency: https://mullvad.net/en/blog/2019/6/3/system-transparency-fut...
If you are interested in it you are more than welcome to join the implementation effort when we have put some basic things in place, or do your own thing right away. Above all it's important that we start moving in that direction as an industry. We don't really care who gets there first. It is also a more interesting game than the currently dominating one of affiliate marketing using hyperbolical and unverifiable security claims.
In a few weeks I will do a talk at the Open Source Firmware Conference together with our technology partner 9eSec ( https://9esec.io/ ), which will coincide with publishing a website as well as more planning of the implementation effort. As far as collaboration goes I have discussed System Transparency at length with IVPN many times, whose values and vision I find very well aligned with Mullvad's. We have an ongoing conversation with them about this and other things.
Thanks for the invite to chat, I agree it doesn't hurt to have a one-on-one talk. Even if Mullvad and PIA doesn't turn out to be a good fit for collaboration we should definitely share ideas on our vision for the VPN services market. Our perspective is pretty much all in the blog post about System Transparency. Looking forward to reading more about your idea "zero access". I'll reach out soon!
We are absolutely interested in joining in the implementation effort and likely have a lot of research/code to bring forth. I agree, that security theater is a joke xD
Looking forward to hearing from you, Andrew
I'm not a firmware expert so take the following with a grain of salt. In general the more your target platform has in common with currently supported mainboards the easier it will be. First of all there obviously needs to be support for the instruction set architecture, then the CPU microarchitecture, and finally for the components on the specific mainboard you are targeting.
Given what has now been done by 9elements and others for SuperMicro boards based on Kabylake I expect more boards to be ported in the coming year.
However, if the OEM supports the coreboot port (e.g. by providing schematics and generally answering questions that come up) that's very helpful.
So, question to kfreds: how was the cooperation with Supermicro? Are they aware of your effort, were they helpful?
(disclosure: doing coreboot development for 12 years now, some of that professionally, now for Google's Chrome OS devices)
Many other VPNs have opted for not yet adopting Wireguard since they claim it's currently difficult or impossible to offer the same privacy guarantees, and also due to the unaudited/new nature of the code base.
Mullvad has offered Wireguard for a while though, and without any warnings about this topic. Has Mullvad found a way to provide similar guarantess?
Or is there another reason why Mullvad does not see this as a problem while other providers do?
As for the privacy issues with a "standard" WireGuard deployment we are well aware of them and are working with others to develop a standard which will hopefully be adopted by privacy-focused operators. Unfortunately we only highlighted them to others in January this year. Note that no criticism of WireGuard's privacy seems to be older than this email: https://lists.zx2c4.com/pipermail/wireguard/2019-January/003...
We have deployed solutions for some of WireGuard's privacy issues while others are in the pipeline. It should however be noted that some of the attack vectors are somewhat esoteric and quite unlikely, so we don't consider them critical. The devil is in the details, as you can read in my email above. On the whole we think WireGuard is where the industry should be heading, fast, for many reasons, and that's why we are spearheading it: https://mullvad.net/en/blog/2017/9/27/wireguard-future/
At the moment IVPN is slightly ahead of us in terms of privacy features in WireGuard, but they are the provider we probably have most in common with, and we frequently discuss projects like this one, so they are definitely the one we most expect competition and collaboration from: https://www.ivpn.net/knowledgebase/253/Using-WireGuard-for-P...
It has idiosyncrasies like:
* Jason refuses to implement or see point of binding to an ip/address. I do not know of a serious network service that cannot bind. I guess roaming to any interface is always the right thing to do and sharing a port number for separate services on separate ips should be done via NAT.
* If you assign an ip/route to a peer, that is already assigned to another peer, previous assignment is simply removed. Somehow, this is a feature and not a blunder >95% of the time. On a security product, it's better to keep going and load a kind of obviously wrong config, discarding conflicting assignments, than to raise an error. Or even a warning (a warning is/was planned).
* on Windows Jason invented tying network id to hash of entire key+address configuration. Cool idea, but I guess you're supposed to fiddle with firewall settings each time you change any peer key/ip, because it's now a new network.
* wireguard is marketed as a modern secure VPN. It has proofs and what not, but I haven't seen a specification of what are it's security guarantees. It (obviously) leaks packet size and timing, also ecn bits, maybe server time? Does it leak public keys, dscp markings? I guess you're supposed to read the whitepaper, maybe source code and keep an eye for any changes.
Bollocks.
$ cat netinet/ip_ah.c netinet/ip_ecn.c \
netinet/ip_esp.c netinet/ip_ipsp.c netinet/ip_spd.c \
netinet/ipsec_input.c netinet/ipsec_output.c | wc -l
6445
You're comparing apples and oranges. Most critics implicitly count IKE and other PKI-related bits as part of IPSec. But Wireguard doesn't offer anything comparable. So if you want to compare apples & apples, you can only compare the actual transport layer. Wireguard is still more elegant, but of course Wireguard has the benefit of hindsight and IPSec has 20+ years of operational fixes.What's happening is that because Wireguard doesn't include PKI people end up gluing together poor man's PKI with shell scripts and shoe strings. PKI is the hard part of key management, and because Wireguard doesn't attempt to solve it, it doesn't incur any of those costs. And the people who do try to solve it and get it horrible wrong, well I guess that's just not Wireguard's fault, right? How convenient.
Important work. Could you share more about your envisioned OSS toolchain for independent auditing of servers? What were the factors which lead to choosing this board, e.g. could other VPN providers contribute resources for extending this coreboot port to a newer chipset?
As for choosing this board, I alluded to it in a post down below. Note that I'm not a firmware expert though. Support for x86 was obviously already there, as well as for the microarch if I recall correctly, so "all" our technology partner 9eSec ( https://9esec.io ) had to do was support the specific board. Had we chosen another microarch or something with multiple CPUs we could have spent months on it. Just another example of how important it is to work with experts.
Every now and then I check the Mullvad blog to see if any new servers have been added, or features I should know about, etc etc. In essence, things that pertain to me as an end user. Stories like this, or new servers, WireGuard being added, etc.
I always thought it would be cool to see some sort of article or blog post with a 2019/2020 road map, however. Just so I can see things that are upcoming or know what’s being worked towards. With VPN services, sometimes it feels kind of stagnant, as the only appreciable updates are new servers or new features. Plus, since I don’t use the Mullvad app itself, updates to it don’t really pertain to me all that much.
Idk, I’m rambling, just food for thought. A roadmap blog post would be cool imo.
http://people.linaro.org/~rob.herring/openfirmware/1275/home... (look for 'song' there)
Crypto and VPN seem like ideal points to basically put a fork in x86 since they are not dependent upon raw CPU performance. It would seem that fracturing the x86 world by picking off individual use cases until x86 has very little left would be the best option (ie. exactly what x86 did to mainframes).
However, I have to take issue with this statement:
> This is the first time a modern off-the-shelf server platform gains coreboot support, and it is an integral part of realizing our vision of transparent and independently auditable VPN servers.
While this is the first modern coreboot system - it is not, by the implication, the first modern server system with fully open source firmware. In fact, I contend that the firmware isn't even fully open source, nor is it fully auditable - the ME taint is still present, as the article clearly states.
(EDIT: I did not read the original article terribly clearly - it is extremely clear in this regard - and it doesn't position the system as being "fully" open source. My mistake and I'm sorry. See comments below for response/apology in response to author's correction.)
In contrast, the Talos II server system, a POWER9 (powerpc64/LE) system, by Raptor Computing Systems has been shipping for well over a year now - and it has fully open source firmware - and by that I mean everything, supported and developed by both the community, Raptor Computing, other OpenPOWER Foundation members and IBM.
The only remaining blob on the base Talos II system is the blob firmware for the onboard NICs - and that's been fully documented and clean-room reverse engineered and the replacement open source project is just about completed (project ortega). It's already usable in beta form, but if it's an issue for you until them - disable them and use an add-in card.
However, unlike the Xeon platform referenced as being the first modern coreboot server platform, the Talos II, or any other OpenPOWER system, does not have an invasive management engine in every CPU you have zero access to with zero auditability.
It's not enough for the firmware to be open source, I want my systems documentation to look like this: https://wiki.raptorcs.com/wiki/Category:Documentation - so that I can write that firmware if I have to. I seriously doubt the supermicro board would include schematics, like the Talos II or Blackbird does, either.
That said, if instead the existing open source firmware, you really want coreboot - which the Talos II does not yet support - for a specific reason - it's being actively ported to the platform, but not out of necessity, instead it's an exercise in open source firmware diversity - as it should be.
If you want a fully auditable, owner controlled, open source platform for your VPNs, or for any other application, x86 based systems aren't it - because, by Intel's ME and AMD's PSP designs, they can't be.
Also, you can buy a Talos II Lite or a Blackbird off the shelf for much less. There are options.
Having open source coreboot on those machines and what it allows you do on top (linuxboot for example) can massively improve your operational security and its viable for a global VPN network.
Improving the state-of-the-art step by step on the platform that dominates the market will lead to more security gain overall then using what is essentially specialized hardware.
Please note that you might have misunderstood the article though. Specifically you seem to have missed these two passages:
> Modern Intel and AMD processors unfortunately still require some closed-source (and encrypted!) firmware in order to boot. This insanity is unfortunately very much a part of the technological foundations on which we rely daily.
> Today marks the day when (mostly) open-source firmware for off-the-shelf modern servers becomes an option.
Thank you for your transparency and forthrightness - and hard work in your involvement in porting coreboot to the new platform. The world needs more people like you.
In the future, I'll read more carefully - take some time to digest what I've read - and write in a tone which conveys a bit more charity than I have done.
I wonder what caveats come with running Coreboot on modern Intel servers. I thought Boot Guard was an issue?
- Signed, firmware developer.
Could the submitter or a moderator also change the title to "Open-source firmware is the future"?