What are the privacy implications of this? Will Cloudflare be able to see GitLab traffic? Was/is Fastly able to before? And is there any reason to trust one over the other?
Gitlab is still the team that didn't notice that three of their three independent backup mechanisms had not been working for weeks to months. And while wholly cloning Github is legal, I started having serious doubts about their ethical instincts when they repeatedly tried to incite people against Github for cloning minor features.
Beyond that: it depends. I tend to believe large vendors are worse for privacy of generic consumers, because they have the scale where it pays off to sell data or do "big data" analysis. They are better for individual high-value targets because they have more. to lose, and less to win, relative to whatever they could expect by screwing you over. Protection against rogue employees is also something that happens in larger teams, if at all.
So, this is a really good point. Someone made the smart decision within Gitlab to outsource some of the harder problems to an outside party with serious experience on the topics rather than trying to script their way out of the challenge themselves.
> Protection against rogue employees is also something that happens in larger teams, if at all.
ref: amazon w/r/t capital one for a good example of this exact failure mode. (Can't tell if we're in agreement; your statement is a bit ambiguous, but given "if at all," I'm assuming we're on the same page)
> Thompson's prior role at the company didn't lend her any insider access in this case.
Thanks for the fact check!
I just don't get why the internet neither forgets nor forgives. A startup in it's early days makes mistakes, the only real mistake Gitlab made was be fully transparent about theirs.
You vouch for CF (and no beef against them), but you know nothing of their internal privacy or reliability pitfalls. That warm fuzzy feeling you have about CF is a result of their carefully groomed PR coupled with 0% opacity on internals. Again, not saying CF has any issues, but if they did, you wouldn't know
Trust takes a lifetime to build and a moment to collapse.
EDIT: toned down
Cloudflare will see All The Things™ as TLS termination at Cloudflare is required for Cloudflare to be able to analyze web traffic and block common attacks. Just based on that one sentence, it seems like a value play more than anything.
That they care enough for the defense-in-depth is probably a good sign, but that also shows a bias in favor of security over privacy. Depending on your personal or corporate privacy preferences/politics, that may or may not be desirable, but it's also for you to decide if any resulting risks are acceptable.
My hope is that Gitlab pressed Cloudflare on the process changes that took place after the WAF outage last year.
---
(this comment has been heavily edited since it was originally published)
Unless there's been a serious homomorphic encryption breakthrough, it's impossible to impose layer 7 intrusion prevention if layer 6 encryption is passthrough.
They subscribed to consolidate the listed offerings (CDN, WAF, DDoS protection). That WAF is mentioned at all is a tip that Cloudflare must decrypt and analyze Gitlab's traffic before re-encrypting it for delivery to Gitlab's application servers.
A. Uploading your own certificate - this just uploads it and CF still does HTTPS interception.
B. Spectrum - Cloudflare will proxy the actual port 443 TCP connections (which might be the plan for SSH connections to GitLab that are not altssh.gitlab.com). The issue is that this only does layer 4 IP reputation DDOS protection; no WAF, etc is done so this most likely isn't what they will do for the main gitlab.com site.
We opted to use Cloudflare Spectrum to facilitate our change. There is no way to mix Spectrum and non-Spectrum on one hostname, but Spectrum allows us to choose between TCP, HTTP, and HTTPS pipelines.
While we intend to TCP proxy port 22 on gitlab.com and 443 on altssh.gitlab.com, this also allows us to use Cloudflare's HTTP(s) content processing pipeline on ports 80 and 443 on gitlab.com. In the latter case Cloudflare will be terminating TLS, performing WAF, CDN and Workers duty, then re-encrypt and send the request to our origin.
You can look at some visualization of this here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
- What are the privacy implications of this? Theoretically, any 3rd party in line of traffic (in this case it is Cloudflare) has the ability to monitor and alter traffic. This is especially true when said 3rd party does inspections into TLS traffic. The privacy implications are, that they can see and alter any traffic between the client and the origin. But in the case of Cloudflare and GitLab, that is desired, as we want to take advantage of Cloudflare's Web Application Firewall to further protect us from 0-days and other threats, where mitigation using traditional methods might cause a delay, that cloud lead to exploitation of those, causing bigger harm. My personal opinion on this (it does not necessarily reflect the view of GitLab) is, that Cloudflare is a vendor I personally trust in terms of adhering to what they say in regards to privacy. While they, as any other vendor, could be breached, I see the benefit of their Security solution bigger, than the breach-risk.
- Will Cloudflare be able to see GitLab traffic? Yes, partially. SSH is not inspected by CF, despite being proxied by them. HTTP can be inspected by anybody on the net and yes, thus CF, too. HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin) for some reasons: - CDN: They need to know which resource you request in order to serve it from a cache - WAF: Once enabled (not from the start) it will scan requests for malicious content and either block or challenge the request - Workers: A highly integrated FaaS platform, we can use to dynamically authenticate requests to cached resources and other logic we may want to execute on the edge of Cloudflare's network. Despite all that Cloudflare does not log raw requests. On our end, we only see the information about where the request came from, user agents, etc. You can find out more about what Cloudflare logs over here: https://blog.cloudflare.com/what-cloudflare-logs/
- Was/is Fastly able to before? Fastly is in the same position as Cloudflare will be in the future. Right now https://about.gitlab.com is served via fastly pointing to an origin, which we control. If fastly would be breached, the attacker could control the contents of about.gitlab.com. Static assets are also hosted via fastly and are also subject to be possibly altered by an attacker.
- And is there any reason to trust one over the other? The decision within GitLab was made purely on technical grounds. Cloudflare was the only solution meeting our criteria to keep serving traffic via SSH on the same host as we do right now. Speaking for myself again, I trust Cloudflare and have personally been a customer of theirs for a long time 2012-ish). While I do not have that kind of relationship with fastly, I believe both vendors to be trustworthy in doing their best to protect the privacy of their users.
You can find more information about how we are going to deploy Cloudflare here: https://gitlab.com/gitlab-com/gl-infra/readiness/tree/master...
> ..inspections into TLS traffic..
> SSH is not inspected by CF, despite being proxied by them
> HTTPS will be terminated by Cloudflare (and re-encrypted when talking to our origin)
As someone who works on the web and does use Cloudflare, I guess I knew these facts, but I must admit I have some doubts about the security and privacy implications, what they mean in practice.
It comes down to trust, and while I believe CF is an excellent actor in regards to privacy, this need for trusting a third-party makes me uneasy.
If CF (or anyone really) is in the middle, then couldn't we say that SSH and HTTPS are not (or less) secure?
> If CF (or anyone really) is in the middle, then couldn't we say that SSH and HTTPS are not (or less) secure?
I guess that depends on how you define 'secure'.
The confidentiality aspect of security is reduced, by letting a 3rd party (Cloudflare) inspect the traffic. However, as stated before, they do not log contents keep metadata briefly. And I have no reason to believe they do otherwise.
Utilizing CF's technology to help prevent a possible breach of GitLab's servers however, strengthens security in my book.
Speaking of breaches. IMO a breach (whether it be at Cloudflare or at GitLab) would be horrific in its own right. But, if an attacker manages to compromise TLS traffic by attacking Cloudflare, they would gain access to that traffic and thus any authentication credentials within it. The blast radius would be limited. Breaching GitLab.com however, has the potential of putting all customer's data at risk. As cruel as it sounds, but I'd rather have a potential breach in front of GitLab. Well, ideally none at all.
I do understand the concerns about having to trust a 3rd party. And as you have pointed out, it comes down to trust. By publishing this information beforehand we want to make it easier for everyone to keep their trust in us, build new trust with Cloudflare, as well as to live up to our values (transparency specifically in this case). And it is my firm belief, that speaking candidly in these matters is important.
I suppose I'm coming to grips with the security (and maybe more about privacy) implications of Cloudflare's immense popularity.
The fact that both Cloudflare and GitLab are so up front, open and transparent about the compromises and advantages involved does ease my worry somewhat.
That's a good point about the upside for GitLab users, that a potentail breach at CF would be limited to that layer, and not GitLab itself.
All in all, the article (and your explanation) has convinced me that this is a net positive.
---
EDIT: To be honest, the point about Cloudflare being able to inspect HTTPS traffic still doesn't sit well with me. I suppose I'll need to study deeper how this works, before I'm persuaded.