Hacker News Reader

Comment by bo1024 | Hacker News Reader

I can't agree with that, this was on Google's on fiber connections between their own data centers, right? And no other company with multiple data centers encrypts all traffic between them, right? (maybe you'll find a small counterexample but no big one.) So I don't think this is "security 101".

View on HN

I work for a company bigger than Google, and we encrypt everything in flight between datacenters. It is security 101.

Does your company have dedicated, unshared, fibre between those datacenters?

Consider the recent passwords leak from Adobe: they stored passwords in a dedicated unshared datacenter. Does this make a good security decision to encrypt passwords instead of using a hash because nobody should have been able to access these encrypted passwords? I really don't think so.

There are problems with your analogy.

1. Data at rest (Adobe) vs data in travel (Google).

2. Software Hack vs Hardware hack

The Adobe data was sitting on a server in a datacenter, it was accessible from the internet on some level. The Google data was taken, apparently, from a dedicated, google owned, unshared link (quite likely a fibre-optic tap)

The methodologies, skill levels and required hardware for the penetrating the above two types of setup are wildly different.

I blame adobe for getting a server hacked, it happens a lot and and they ignored a lot of body of knowledge built up over the years. I do not blame Google for getting their inter-datacentre links physically compromised by a security agency of the US government.

Nor do I blame them for (incorrectly, as it turns out) deeming that an unlikely scenario and therefore giving it low priority.

I would blame them for not doing anything about it now that they know it is happening but that does not seem to be the case.

(I fully expect companies to encrypt data between datacentres if they are not on dedicated unshared links)

I hear what you are saying but I think there are similarities. In both cases there was an assumption "X is safe" and then the thinking have stopped. I've heard different version of how the data was taken from the google's link and some ideas were pretty low-tech. The data links have been compromised in the past not only by NSA (search for "Operation Ivy Bells" if you haven't heard this story before) but also by criminals or even competitors.

Yes we do. Still use encryption.

> And no other company with multiple data centers encrypts all traffic between them, right?

Indeed they do! From personal experience, Cisco was hawking its TrustSec inter-DC encryption solution five or six years ago, even over dark fibre.

If you believe the threat is a government agency splicing private, unshared fiber to capture your traffic between data centers, why in the world would you trust equipment from Cisco (who lists "Government" as one of the industries they sell to) to protect you from that?

Well I certainly understand your point, but the question was 'are big companies encrypting their inter-DC traffic' and the answer is 'yes', even if it's backdoored without their knowledge.

Google's inter-dc links are way too big for any appliance type of thing to encrypt. Like most things at Google the scale of their network is incomprehensible to most people.

> Google's inter-dc links are way too big for any appliance type of thing to encrypt

Frankly, no.

There are numerous network devices that can handle AES-256 on 10 Gbps links, as a matter of routine, whilst doing 'mundane' switching for the day job.

If you have the money there are dedicated hardware that can handle the same at 100 Gbps. IP Cores is one from memory that produces the circuitry for that. They can throw compression in there as well if you like.

Encrypting data links isn't magic. Google just didn't do it.

Those were very amusing tiny numbers you wrote in your post.

100Gbps is nothing for a company at the scale of Google. They are probably closer to 10-100Tb/s on there backbones.

You don't need appliances here as they can't handle the load, build the encryption into your application.

In addition to the numbers already posted, you can remember that Google decided to add encryption to the data links in September.

This is not true. Quoting from http://www.washingtonpost.com/business/technology/google-enc...

"Google’s encryption initiative, initially approved last year, was accelerated in June as the tech giant struggled to guard its reputation as a reliable steward of user information amid controversy about the NSA’s PRISM program,"

Well, I do :) Moreover, I encrypt all the traffic even inside the same data center.

could you share which technology are you using to encrypt all the traffic?

* Mid-tier servers: standard HTTPS with nginx

* Database: SSL connections for MySQL

* Memcached, Gearmand, and other tools that don't have built-in SSL support: simple home grown message level encryption (AES256)

And of course, there are VPN tunnels between data centers in addition to the above.

Thanks.

> And of course, there are VPN tunnels between data centers in addition to the above.

Could you please be more specific on the VPN solution that you are using? How do you manage the shared keys? How do you make sure 'system administrators can't easily read the traffic?"

We use Cisco appliances for VPN (a few different models) and indeed there is a shared key that we have to input manually. However, after the key is entered (and configs saved) in order to decrypt the traffic one would need to print Cisco configs which is a very unusual operation that would be logged and then alerts will fire, audits will catch it, etc.

Just out of interest how do you transfer the key between datacentres for setup? Same person travels between them? PGP encrypted email? Or over the phone?

Phone is I think an obvious (and now clearly wrong choice) although maybe always suspect if you are concerned with dark fibre . The endpoint security of a device generating and transmitting the key now also being a risk. How far up the chain do you worry?

An airgapped device to generate the key and a single person travelling between datacentres seems the secure (although costly) solution. Obviously if TSA/customs remove device from them for inspection or connect it to anything it needs to be thrown away (or moved to insecure duties) and the setup process restarted.

I think public key encryption with long enough key is a pretty safe bet these days. Of course, NSA might have new non-public discoveries in math/crypto that might make public encryption obsolete. Or they might have a device form Area 51 that breaks any encryption. However, I haven't seen any evidences of this yet.

>> in order to decrypt the traffic one would need to print Cisco configs

Which could be done legally via 'Cisco Service Independent Intercept (SII)' built into IOS to comply with CALEA (Communications Assistance for Law Enforcement Act). And not so legally via user-escalation exploits within the same service.

Anyway, props for making the effort. I too am interested in your key exchange methods.

Amazon does that.

Reply on news.ycombinator.com