Hacker News Reader

> In practice, I heard someone say that the biggest improvement in people’s privacy has been use of Gmail.

In practice, until Snowden happened, NSA was able to access all the Google's internal data as Google replicated in plaintext its whole datacenters through the links snooped by the NSA or the GCHQ.

http://www.slate.com/blogs/future_tense/2013/10/30/nsa_smile...

Correct. And now they encrypt all the inter-data center traffic.

Question: Is that done on an end-to-end basis? Or do they encrypt the links between data centers?

I want to encrypt a 10g ethernet and all the solutions look quite expensive.

Has anyone done high speed encryption (i.e. 10gbps/1500 byte packets) with strongswan or similar?

I believe end to end is simpler when you control all the ends and can tweak all the software, like they do. There's also an advantage of having very fast processors already provided.

You have to encrypt the transport at the application layer when you're talking about the scale of major internet companies--there are so many links going out of each datacenter that it's not feasible to do IPsec or similar lower level encryption. There aren't many hardware devices sold that can do the encryption at the 10-100+ Gbps speeds that datacenter links use (if any), and it's much easier to amortize the encryption cost across all your application cpus versus having a small number of devices or dedicated hosts doing crypto at a lower layer.

> There aren't many hardware devices sold that can do the encryption at the 10-100+ Gbps speeds

You are vastly underestimating AES-NI. Most modern Intel CPUs can handle that kind of encryption throughput (when paired with fast enough memory). Even my old Sandy Bridge-E CPU from 2011 gives me ~96 Gbps of AES throughput (with quad channel memory).

Should we trust the implementation of AES-NI in this context?

http://arstechnica.com/security/2013/12/we-cannot-trust-inte...

AES-NI and RDRAND are different beasts. AES-NI is deterministic, and thus much more difficult to practically backdoor.

Sorry if this is a dumb question, I'm just trying to think it through. It seems like it has to use the key you give it; it seems like the output stream has to be correct, or it just won't work.

I'm not really sure how this would work for network traffic, I guess for disks I would expect CBC mode, which would require an IV? Is that right?

So could the hardware be made to expose the key, or IV's, or leak information in some unexpected way?

Leaking an IV isn't a problem, but leaking the key would be a problem. If AES could be made to leak the key it would be a huge problem with the cipher itself. Otherwise leaking the key would be restricted to some extremely complex timing level behavior of the CPU, which would be difficult to make it all the way to the network level, without network card cooperation (though many 10gbps network cards are intel made).

Perhaps feasible, but far far less likely than compromising the entropy of a random number generator.

I was replying to a question about whether or not you could encrypt the links between the datacenters, where you typically you have large core/cluster routers that handle traffic entering and leaving your datacenter. I was talking about the fact that these routers have dozens or hundreds of 10-100 Gbps ports on them, and are simply not built to do line rate IPsec on each port. So you can't just turn on IPsec for the routers at your datacenter border and magically get encrypted transport between your datacenters.

You can setup IPsec tunnels on your application nodes and use that to get transparent encryption of all traffic, but that is pretty complicated to configure and manage. It's much easier to just have your different applications use an encrypted and authenticated transport when talking to each other over the network, whether they are in the same datacenter, or talking to a remote datacenter. This is how Google and Facebook do their cross datacenter encryption.

Companies like Ciena make line rate encryption cards that protect links. Not cheap, but not outrageously expensive for that level of gear.

With hardware acceleration, my quad core laptop can encrypt faster than 20gbps.

Without it, it can still encrypt at at 4gbps.

I was going to suggest you could place a couple servers at each end, but you only need one low end server.

Anyone offering expensive solutions is doing you a disservice.

Is that at the network level? My experience is that packet handling in Linux and FreeBSD does not make high speed IPsec feasible.

With things like kernel bypassing and bulk packet processing it's really not that hard to get to 10gbps on a single server. Heck, linux is getting close to ready for 100gbps https://lwn.net/Articles/629155/

I don't know exactly what resources IPSEC takes, but you can definitely get the "or similar" part of the question.

AES-NI takes 3.5 cycles per byte. BLAKE2 takes 3.08 cycles per byte. That means that 2-2.5 cores can handle all the encryption and signing by themselves. Then you have 80% of the machine's resources free just to shovel packets in and out.

It's easy to get 10Gbps packet handling on Linux or FreeBSD, it's easy to encrypt at >10Gbps on Intel, but I would love to know how to get both into a working situation IPsec solution at >10Gbps. Speculation isn't interesting, because clearly all the pieces should work, so I'm looking for anybody with actual experience with it working. I've tried, and been unsuccessful, and also unable to find anybody else on the internet that has been successful.

Is IPsec a hard requirement?

I would expect it to be easier to encrypt and sign a megabit at a time instead of individual packets.

I'm having trouble picturing a situation where it has to be IPsec and it also has to be 10Gbps on a single box.

The original question referenced strongswan, which is IPsec, and where my investigations have focused in the past. But as I understand it strongswan is just for key exchange, and the decryption is still performed in the kernel, and often single-threaded.

It's easy to pass 10Gbps of TLS packets over a link with both sides of the connection doing their own encryption. But if you want to secure against possible unencrypted packets passing over a link, you'd want layer 3 or layer 2 encryption as well. Layer 2 means abstruse hardware, layer 3 means encapsulation of some sort, be it IPsec or whatever. That encapsulation is what I believe that kernels can not currently handle, though they should theoretically be able to. FreeBSD has made some noises about improvements to parallelization to support multi-gigabit IPsec, but I haven't been able to test them.

I am in the same boat you are. I also appreciate your practical and concise point of view.

Yes, it ought to work quite easily, but in practice it doesn't yet, does it?

I'm planning to spin up a few servers in Amazon to test it there just to get some benchmarks. What kind of testing and experimentation have you done? In my use case I would put the traffic through a single machine across a ten gig link to another machine. So I could ensure that all the traffic passing through with it.

The practical use case is to encrypt traffic between data centers. I don't see a feasible way to do that on a megabyte at a time. It almost certainly has to be done on a per packet basis. Either at the application which is a little bit difficult to do in the short-term, or in some sort of IP sec or other similar solution.

No question that the right solution here is to make all the applications do encryption, but again, I need to have a practical solution now, and then the developers can put that in.

I've only messed with a single tunnel at a time. However, there's a whitepaper from Intel from several years ago that says that multiplexing over 6-12 tunnels should possibly work:

http://www.intel.com/content/dam/www/public/us/en/documents/...

The question remains on how to perform routing for this. It appears that running Quagga on both end points, speaking BGP to each other, will allow ECMP over the 6-12 IPsec tunnels. Each individual stream will still be capped at ~1Gbit, but at least the total throughput could be used and there's no traffic between the two endpoints that's not encrypted.

However, getting Quagga + BGP + ECMP + multiple IPsec tunnels + pinning tunnels to particular cores + setting up the RSS flows from the NIC to particular cores... Well it was a bit more than I could bite off at the time. The pieces feel sooo close to working, but it becomes a Rube Goldberg machine of systems software, with each piece operating at the limit of its design properties.

And honestly, I kind of want two orthogonal encryption technologies; TLS has bugs every other day it seems, and IPsec has always seemed a little bit suspicious, and really who knows if my random numbers are really random, so I wanted two layers.

Please do share if you make any progress!

I hesitate to post as my experience is a bit dated as well, but this topic continually seems to come up lately in my circles without a whole lot of authoritative information I've been hearing.

Single-stream IPSec is still a major performance issue. Maybe someone out there is getting more than about 2.5Gbps performance in the real world per stream out there, but I haven't talked to them. This is with some pretty idealistic assumptions too - I would expect most folks real-world performance is orders of magnitude lower.

Scaling it over multiple cores does work, but introduces the complexity that you mention. This still completely ignores the major problem of the fact that single flows are stuck with early-2000's level network performance! With ethernet moving to 40g in high-performance applications, being stuck with 2gbps TCP connections inside the datacenter is not ideal and makes doing global-infrastructure-level ipsec a non-starter. I briefly spent some time considering going 100% ipsec for all inter-cluster communications for a product we were building, but the performance implications put that dead in the water almost immediately. Moving it up the stack into the application was far less expensive - but of course carries with it the cost that you are no longer operating in a "fail safe" environment.

You could make the argument that anything high performance needs to horizontally scale across multiple streams and links - but that's being too idealistic for the current state of technology. Sometimes you just really need to make that legacy NFS transfer go faster.

With the advent of hardware acceleration built into "commodity" CPUs these days - I really did expect more real-world progress on this front both on the linux side and custom asics (e.g. switches/routers). Given the number of colleagues/customers who have asked for similar solutions ("I want 10gbps single stream ipsec") it's not an uncommon problem folks are running across.

For others who are interested, here's a thread that culminates in the 2.5Gbps number:

http://permalink.gmane.org/gmane.linux.network/280175

And here's some StrongSwan documentation about getting there:

https://wiki.strongswan.org/projects/strongswan/wiki/Pcrypt

Given the large interest and discussion, I've created a discussion forum here:

https://groups.google.com/d/forum/highspeedencryption

It's public, but please do join if you're interested.

I think it would benefit the community to have an open and central location for best practices of high speed encryption.

Epistasis, Phil21, others, please do join and let's continue the discussion and progress there!

SMTP with TLS doesn't leak any metadata except the connecting IP address. And with DMARC you can mandate it.

Calls for whole new protocols for email are usually made by people who don't understand the email ecosystem as it currently stands.

Can the TLS be enforced by the sender all the way through the chain to the destination mailbox?

Yes, as well as PGP or S/MIME. It just depends or your relay configuration. We've implemented it switch able. We've customers that are bouncing non PGP encrypted E-Mails.

The sender (the sender's service provider or organization) decides when to use TLS while the message bounces around internally, then (this is the typical long hop) the recipient's server and the sender's SMTP client negotiate, and then the recipient gets to decide.

The sender may have opinions, but as soon as the message is handed over to the recipient's server, that's it. That's the end of the sender's influence, and that's how it has to be on a global internet. People can connect without having their infrastructure approved by anyone else.

No, it can't. The receiver could be using a non secure SMTP server.

But SMTP with TLS is opportunistic -- and if we're talking "national state secrets" it's trivial for a state to MITM the connection and do a TLS downgrade attack.

Not if the receiving smtp server implements DMARC

Is DMARC like HSTS for SMTP?

DMARC doesn't have a STARTTLS policy. I think you mean DANE / TLSA.

In which case you're conceding security to governments anyways.

Sorry Tony, you're absolutely right. Got my acronyms wrong. Thank you :)

(The parent is Tony Finch, a very smart email guy who runs email for the university of Cambridge)

If you actually care about the data, it does nothing. It only guarantees that the front door is encrypted.

With the number of cloud anti spam and archive companies out there, you have no idea what happens between those providers and the mail servers.

I can't agree with this assessment. Personally, the NSA or pervasive surveillance use case is overblown and a low risk threat to me.

Anyone reasonably competent conducting the most cursory of investigations could figure out who I do business with in hours. The NSA boogeyman with omniscient network surveillance capability will come up with ways to find the source/destination of obfuscates messages anyway.

Sensitive data that I routinely handle is content. Employees improperly reading mail (on either sender or recipient side), or some data breach affecting mail servers, desktops, or relays are much higher risks.

PGP is a great way to encrypt ad hoc data at rest. I think the reason it isn't pervasive is that once you figure out what is really secret, you develop specific tools to transmit that data more seamlessly.

What about the chilling effect? People change their behaviors when they know they're being watched. It goes completely against the spirit of the US Constitution. How can I pursue happiness if I'm worried that what I write will be a liability in the future?

EDIT: grammar

I disagree with the surveillance society, but I can't control that policy.

So I choose to focus on what I can control.

I absolutely agree with "focus on what you can do." I think that we can also engage in the conversation as a community and network, which of course you are doing.

The main thing that stuck out to me about your post was "low risk to me." I have studied risk extensively, and I would like to point out that generally, risks are classified with ratings of both frequency and severity. I would categorize the NSA spying as moderate-high risk (to you) due to their extremely high frequency. Yes, the severity to you will be relatively moderate (chilling effect, etc), but the cumulative, unrelenting effect it has is incredible.

As I'm thinking about this more, it's not even a true "risk," because risk always entails uncertainty. You are certainly being affected by it. The true risk of the spying program is that somehow you are hung by your own words, which has an extremely high severity (prison aka slavery). I hope that this rings true to you. Rereading what I've written, it seems like I could be projecting a lot of my own feelings and experiences onto you, but this stuff is just so universal in its effects.

Thanks for discussing this! Have a great day.

My idea has been to have a system that works on top of email but sends out "fraud" email to other participants in the network. The encrypted payload gets unencrypted at each endpoint and has a marker as to whether it was, what i call a "fuzzer" email, in which case it's discarded - otherwise it gets graduated to the users inbox.

Under this system, Eve will see Alice exchanging encrypted messages with Bob and Carol but won't know if they're automated dummy messages or authentic communication.

It lowers the information fidelity through an intentional introduction of noise.

The "fuzzing model" has wide use cases: for instance, a web browser can programmatically "browse" a variety of websites so that Eve can't tell what Alice is truly looking at. Every user in a system will have "red flag" websites in the "auto browse" list.

In these discussions about the importance of metadata, I'm surprised that Bitmessage is rarely mentioned. The Bitmessage protocol is decentralized, trustless, and hides metadata such as the sender and receiver of messages.

I was really excited when I first read about bitmessage. As far as I can tell, it solves more problems than any other messaging protocol. It's a mystery to me why is doesn't get more attention.

Dark Mail turned out to be a joke. Apart from the silly fields like "political party", the fundamental security model's broken. It has essentially zero benefits of "just use Gmail". The suggested security level delegates security to the receiver's mail server. Basically if we just had a " require TLS with verified cert for SMTP" option, that'd do most of what Dark Mail does.

In fact, big companies already so this. When LM talks to the DOD over SMTP, you can bet they have mandatory TLS. Exchange offers this out of the box.

DM offers higher capabilities with no usability benefits over PGP or whatever.

Disclaimer: Since the beginning, I've said the DMA had a zero chance of success. The problem isn't solvable.

We've had mixminion[1] around for years, which handles backwards compatibility with aplomb.

[1] https://github.com/mixminion/mixminion/

Anyone on HN know what Signal does for envelope encryption? AFAICT it's the only crypto app that's OSS and still cares about usability.

I'm reasonably certain all communication with the routing servers is done via https. You could double check their source on GitHub

Backwards compatibility has nothing to do with it. No one can read GPG encrypted mail.

It isn't built into their clients, they don't by default get their own keypairs, it is simply straight up impossible to send encrypted email to even friends and family and have them be able to read it.

Everything else is window dressing to this problem.

How about S/MIME? At least, the programs that I use on a regular basis have it built in. (iOS and OS X Mail, Thunderbird, Outlook). It still does not help at all with the management and distribution of key pairs.

"In practice, I heard someone say that the biggest improvement in people’s privacy has been use of Gmail"

Technologically that might be true, but since Google is a third party they're required to provide their data to the government so practically, the technological considerations like SSL are absolutely meaningless.