Question: Is that done on an end-to-end basis? Or do they encrypt the links between data centers?
I want to encrypt a 10g ethernet and all the solutions look quite expensive.
Has anyone done high speed encryption (i.e. 10gbps/1500 byte packets) with strongswan or similar?
You are vastly underestimating AES-NI. Most modern Intel CPUs can handle that kind of encryption throughput (when paired with fast enough memory). Even my old Sandy Bridge-E CPU from 2011 gives me ~96 Gbps of AES throughput (with quad channel memory).
http://arstechnica.com/security/2013/12/we-cannot-trust-inte...
I'm not really sure how this would work for network traffic, I guess for disks I would expect CBC mode, which would require an IV? Is that right?
So could the hardware be made to expose the key, or IV's, or leak information in some unexpected way?
Perhaps feasible, but far far less likely than compromising the entropy of a random number generator.
You can setup IPsec tunnels on your application nodes and use that to get transparent encryption of all traffic, but that is pretty complicated to configure and manage. It's much easier to just have your different applications use an encrypted and authenticated transport when talking to each other over the network, whether they are in the same datacenter, or talking to a remote datacenter. This is how Google and Facebook do their cross datacenter encryption.
https://groups.google.com/d/forum/highspeedencryption
It's public, but please do join if you're interested.
I think it would benefit the community to have an open and central location for best practices of high speed encryption.
Epistasis, Phil21, others, please do join and let's continue the discussion and progress there!
Without it, it can still encrypt at at 4gbps.
I was going to suggest you could place a couple servers at each end, but you only need one low end server.
Anyone offering expensive solutions is doing you a disservice.
I don't know exactly what resources IPSEC takes, but you can definitely get the "or similar" part of the question.
AES-NI takes 3.5 cycles per byte. BLAKE2 takes 3.08 cycles per byte. That means that 2-2.5 cores can handle all the encryption and signing by themselves. Then you have 80% of the machine's resources free just to shovel packets in and out.
Yes, it ought to work quite easily, but in practice it doesn't yet, does it?
I'm planning to spin up a few servers in Amazon to test it there just to get some benchmarks. What kind of testing and experimentation have you done? In my use case I would put the traffic through a single machine across a ten gig link to another machine. So I could ensure that all the traffic passing through with it.
The practical use case is to encrypt traffic between data centers. I don't see a feasible way to do that on a megabyte at a time. It almost certainly has to be done on a per packet basis. Either at the application which is a little bit difficult to do in the short-term, or in some sort of IP sec or other similar solution.
No question that the right solution here is to make all the applications do encryption, but again, I need to have a practical solution now, and then the developers can put that in.
http://www.intel.com/content/dam/www/public/us/en/documents/...
The question remains on how to perform routing for this. It appears that running Quagga on both end points, speaking BGP to each other, will allow ECMP over the 6-12 IPsec tunnels. Each individual stream will still be capped at ~1Gbit, but at least the total throughput could be used and there's no traffic between the two endpoints that's not encrypted.
However, getting Quagga + BGP + ECMP + multiple IPsec tunnels + pinning tunnels to particular cores + setting up the RSS flows from the NIC to particular cores... Well it was a bit more than I could bite off at the time. The pieces feel sooo close to working, but it becomes a Rube Goldberg machine of systems software, with each piece operating at the limit of its design properties.
And honestly, I kind of want two orthogonal encryption technologies; TLS has bugs every other day it seems, and IPsec has always seemed a little bit suspicious, and really who knows if my random numbers are really random, so I wanted two layers.
Please do share if you make any progress!
Single-stream IPSec is still a major performance issue. Maybe someone out there is getting more than about 2.5Gbps performance in the real world per stream out there, but I haven't talked to them. This is with some pretty idealistic assumptions too - I would expect most folks real-world performance is orders of magnitude lower.
Scaling it over multiple cores does work, but introduces the complexity that you mention. This still completely ignores the major problem of the fact that single flows are stuck with early-2000's level network performance! With ethernet moving to 40g in high-performance applications, being stuck with 2gbps TCP connections inside the datacenter is not ideal and makes doing global-infrastructure-level ipsec a non-starter. I briefly spent some time considering going 100% ipsec for all inter-cluster communications for a product we were building, but the performance implications put that dead in the water almost immediately. Moving it up the stack into the application was far less expensive - but of course carries with it the cost that you are no longer operating in a "fail safe" environment.
You could make the argument that anything high performance needs to horizontally scale across multiple streams and links - but that's being too idealistic for the current state of technology. Sometimes you just really need to make that legacy NFS transfer go faster.
With the advent of hardware acceleration built into "commodity" CPUs these days - I really did expect more real-world progress on this front both on the linux side and custom asics (e.g. switches/routers). Given the number of colleagues/customers who have asked for similar solutions ("I want 10gbps single stream ipsec") it's not an uncommon problem folks are running across.
http://permalink.gmane.org/gmane.linux.network/280175
And here's some StrongSwan documentation about getting there:
I would expect it to be easier to encrypt and sign a megabit at a time instead of individual packets.
I'm having trouble picturing a situation where it has to be IPsec and it also has to be 10Gbps on a single box.
It's easy to pass 10Gbps of TLS packets over a link with both sides of the connection doing their own encryption. But if you want to secure against possible unencrypted packets passing over a link, you'd want layer 3 or layer 2 encryption as well. Layer 2 means abstruse hardware, layer 3 means encapsulation of some sort, be it IPsec or whatever. That encapsulation is what I believe that kernels can not currently handle, though they should theoretically be able to. FreeBSD has made some noises about improvements to parallelization to support multi-gigabit IPsec, but I haven't been able to test them.