Google releases stats on email encryption in transit | Hacker News Reader

Google releases stats on email encryption in transit

google.com

183 pointsMatt_Cutts50 comments

A fun experiment for someone who isn't afraid of having their ISP, colocation provider, or VM provider terminate their account and/or sue them because ZOMG LOTS OF OUTBOUND TCP/25 CONNECTIONS MEANS A SPAMMER IS YOU:

• Take Google's list of mail domains they see (they offer a CSV file for download) and look up all of the mail exchangers for all of the domains.

• Remove duplicates from the list of MX hostnames.

• Write a short program that does the following for each of those mail exchangers:

- Connect to tcp/25, send a valid EHLO, and see if the server indicates that it supports STARTTLS.

- If it doesn't, disconnect and move on to the next hostname.

- Send STARTTLS and start a TLS session. Put a NULL cipher at the top of your client's ciphersuite list to see if anyone bites. Record the certificate chain the server sends.

- See if the leaf certificate is valid and if you can make it actually chain up to a trusted CA via the server's certificate chain.

- Disconnect.

You now have a list of some of the largest mail exchangers on the Internet, whether or not each one supports STARTTLS, and whether or not each one supplies a (valid) certificate that actually chains up to a CA anyone trusts. Write a blog post containing pretty graphs for these statistics. Poke fun at the most egregiously broken TLS configurations, certificates, and certificate chains you see. Idly wonder if the proliferation of (sometimes completely broken) opportunistic TLS with SMTP and the relative scarcity of authenticated TLS with SMTP will push passive attackers into becoming active attackers so they can continue to easily grab messages in-flight. Bemoan the broken CA system. Drink heavily. Dream.

I'm lost. Where is the "profit" step ?

If it makes major providers start bothering with proper SSL configuration with a not-self-signed cert for their mail services maybe that would be a profit: things become slightly more secure for some people.

The 99.9% freak me out.

The huge problem with email encryption right now is that you can't reasonably make it mandatory (as you can see, a lots of domain do not support it) so it can be easily be stripped à la sslstrip, and there is no such a thing as HSTS. (Again, please, build pinning into everything.)

So an active attacker able to MitM can simply turn encryption off, and those 99.9% make me wonder if that happened to the 0.1%...

> The huge problem with email encryption right now is that you can't reasonably make it mandatory

Many services can make it mandatory, as long as you're willing to take the hit to convenience, so at least your outbound email is always protected (and you get some level of canary for inbound email).

This is usually for some compliance rule or another because they want enterprise business, so even the mainstream ones like Google Apps[1] and Office 365[2] have support for requiring TLS.

[1] https://support.google.com/a/answer/2520500?hl=en

[2] http://technet.microsoft.com/en-US/library/jj723154%28v=exch...

That problem is quite common on the internet in general. Just think how common ftp still is! We constantly use public networks and then have to mitigate the fact that the packets are public. It would be better to treat encryption like a peering arrangement. When there is sufficient traffic between two large networks all packets should be routed via an encrypted tunnel.

While I generally agree with the concept, encryption isn't free, and will break pretty much every routing protocol used. It's easy enough to get a tunnel through transit when you control both ends, but neither the hardware nor the software that actually runs the peers supports it in any way. You'd have to set the link, then encrypt on top of that, costing a ton of cpu cycles, bandwidth overhead, and requiring something that can handle encrypting everything in real time and then pass it off to the actual edge connection. "Large" networks do 100Gbit plus, and I can't think of any way this is currently feasible to do.

I was really thinking in terms of coporate networks, which would be considered small. It is already common on such networks to have proxies/NAT/filtering that inspect and route everything in a very resource unfrinedly way.

I would propose creating single VPN tunnel between these corporate networks that all traffic destined for the other network is routed down. Just join individual private networks with encryped tiunnels. This would be completely transparent at the client level as the data would just follow a different path. It would avoid the startup cost of SSL or the higher CPU requirements of symmetric encryption. Such a setup could be required as a contract/tendering/supplier requirement.

Really if you have two companies that are sending data that may be sensitive why would you want any of the packets to ever be sent in plan view? Client side standards will never give a complete gurantee of that. It is much more enforcable to add the security at the network level.

STARTTLS seems to work like that joke in Anchorman: It works 60 percent of the time...every time.

Worse, it works 100%... 99.9% of the times.

The other times, it's when someone wants it not to work.

My guess is dem pesky 99.9% sla's and routing around the downtime in some way.

> The huge problem with email encryption right now is that you can't reasonably make it mandatory

Sure you can, see pgp.

Really, its nice to see they are encrypting coms between each other but I don't really see how this affects me. If I want something secure I will use pgp. Email encryption, has and, always will be a client side problem. There is nothing the middle men and women can do for you that you can reasonably trust.

If google really wants to solve this problem they will make pgp features native to gmail. I'm guessing they're heading that way with the chrome extension being discussed here at the minute[1].

[1] https://news.ycombinator.com/item?id=7842696

But Google likes to read your email so they can target ads. Even they don't want you to encrypt on the client.

Don't want and don't support are two different things. Google have lots of ways to get ad signal.

Leading the way looks a lot better than dragging your feet on something that is going to happen whether they want it to or not. At least they can say "hey we lead the way on this, even if it was 3 decades too late"

> that is going to happen whether they want it to or not

I highly doubt this. This sounds like wishful thinking.

Pessimism, just as illogical as optimism.

As in, care to explain why you so highly doubt it? I have little more than anecdotal trends as evidence.

It also gets tricky doing spam detection and search if all email is encrypted on the client.

Or easier, if mostly the people who send ham know your public key.

Maybe they don't care about your email(they have plenty of data) as long nobody else can use it for ads ?

Google is more than capable of delivering a user a reasonably targeted ad in gmail without reading the message body.

Capable but unwilling.

But email providers don't need end user cooperation to implement intra-provider encryption, i.e. encryption of intermediate hops before it's finally read. (If you email another gmail user, Google can keep the whole voyage encrypted, and likewise if you email a Yahoo user after they agree to encrypt email between the services.)

Yes, such schemes are leaky, as they allow the attacker to read it at any link between providers that haven't cooperated to set up a secure link. And it probably doesn't change the "expectation of privacy" issue, as the provider will be applying encryption to the plaintext, but it decreases the attack surface without requiring end user action.

65% Out, 50% In overall.. that's actually a lot better than I was expecting to see.

It's really cool to bring up an issue such as this, but... the positive impact of SMTP over TLS is limited and low value. Honestly, who gives a hoot that constant contact newsletters are sent in cleartext?

I need to worry about where my organization's email goes, because quite simply, people do dumb things. Things like put payment information in email. Or PII subject to HIPPA or some other regulatory regime.

Secure SMTP gives me no solace whatsoever, because even if I ensure that my receiving party's email gateway is secure, I have no assurance that any relaying happening behind that gateway is secure to the standards that I need to meet. For example, under many corporate security policies, MPLS tagged connections between offices is deemed secure -- they trust the telco. My organization requires an encrypted link or fiber owned and managed by the organization.

IMO, you're better off operating under the assumption that your email is a postcard, because it is. Sensitive information should be shared in this order: in person, in a registered paper letter (the US government permits material up to the secret level via this means), first class mail, on the phone, via internal IT systems to other internal recipients, via homing pigeon, email.

Given the number and state of the certificate barons and their document-as-stolen marbles, SSL is not really encryption though, is it?

I thought this was going to be GPG stats. I bet Google do keep them... and I bet they have an uncanny correlation with people downloading their Gmail and leaving the service, or being well placed within the IT industry and vocal opponents of many of Google's recent policies.

I'm confused as to why anything isn't 100%. I guess I could maybe understand 0%, but how can it be 50%? I would imagine you either encrypt all email or you don't encrypt any email. What is the in-between?

Large installations with heterogeneous clusters, accreted during many internal reigns and rounds of partially-integrated M&A.

Probably because it allows connections that aren't perfectly secure, such as supporting anything less than 256 bits.

That wouldn't explain why Google would have difficulty.

It implies some kind of inconsistency in configuration or something on the servers Google's sending email to?

Transit encryption requires that both sides are game. Google always supports it, but the server on the other side may not.

With my small personal domain I was just using a self-signed cert until a few weeks ago, and I could see that some sending servers quit and sent in plain text after encountering a self signed cert, others delivered regardless.

Take Cisco.com for example. There are a lot of systems there that are sending out emails. Employees and newsletters are probably encrypted but only make up a small portion of their outbound mail. Things like ticket updates are going to result in a huge volume of email. Cisco may have (wrongly) decided that because the ticketing system is a trusted source it doesn't need to be encrypted. So some of their email is going directly out from the tool and some is going through a smart host. This is probably why it's <50%.

I wonder what's happening on weekends?

Every weekend, the percentage of encrypted outbound mail decreases, but the percentage of encrypted inbound mail increases. If you click on "90 days", you can see that this trend has been going on for quite some time.

Could this be due to the difference between the usage patterns of work email vs. personal email? Or could it be a specific email service provider (or providers) who do something differently during weekends?

On weekdays, lots of people using Gmail and Google Apps to send work emails to each other which are more likely to accept encryption. But on weekends, inbound commercial, non-work related emails (such as newsletters and promotions) drops off, leaving the percentage of personal emails higher.

I'm in Asia, and checked that segment.... pretty much 0% encrypted except for yahoo (but yahoo japan is unencrypted). I really wish they would be more forward thinking. Even if the network architecture is ahead in Asia, the services feel so much behind!

Why is hotmail between 50% and 90%?

That's a good question. I've been involved with some research, outside of Google, on this exact topic. Because we don't have the same data sets that Google has, we have a slightly different methodology for gathering data and we found that hotmail (or any Microsoft mail provider) never even has the option to encrypt if you want to. We always though it was strange that they didn't, and this new data seems to suggest that there may be something else going on.

One possibility that was already pointed out is heterogeneous architectures. It's possible that different systems in hotmail, maybe in different geographic regions or across national borders, use encryption.

Paypal 0%

So how do I manually check that my host supports this? I checked the list of domains and although I know I have sent mail to gmail address my host does not appear.

Interesting to see peaks on weekends. Guessing more private (person-to-person) traffic, so more traffic to/from domains that encrypt?

During the weekdays there's more business communication, newsletters etc. from corporate servers that always lags behind in terms of featurfullness and security.

I'm so pleased to see that (at time of writing) 4/35 comments are about this feature of the data. It's really encouraging to know that the HN audience is sufficiently academic to spot features like this, wonder about them, question them, and arrive at interesting insights. It leaves me keen to continue treating HN as one of my main new aggregators knowing that bullshit data and visualisations will be detected and treated as such!

I cannot see abv.bg and mail.bg

Funny that meanwhile they still won't support XMPP encryption...

Maybe because they're dropping support for XMPP altogether.

This is ironic, as Gmail has never supported email encrypted (or even signed) using S/MIME and digital certificates.

This post is referring to the secure transport of plain text emails between email providers. I.e. protecting from a mitm between gmail.com and hotmail.com when somebody sends a normal email from one to the other.

See the other gmail announcement today: https://code.google.com/p/end-to-end/

I still think DarkMail will be better than any PGP-based solution. But of course DarkMail doesn't actually "exist" yet, so I guess PGP wins by default for now.

Reply on news.ycombinator.com