DanielDent

They could, and as a stopgap measure that might work, but..

(1) Some of the collaboration features (e.g. work on Gitlab itself) depend on having everyone on the same instance.

(2) Gitlab.com gives them a nice dogfood-esque environment for what it's like to actually operate Gitlab at scale. If they are having problems scaling it, then potentially so are their customers. Fixing the root cause is usually a good thing and is often an imperative to avoid being drowned in technical debt.

(3) It moves the problem in some respects. Modern devops techniques mostly allow the number of like servers to be largely irrelevant, but still.. the more unique instances of Gitlab, the more overhead there will be managing those instances (and figuring out which projects/people go on which instances).

It's a simple approach which I'm sure would work, but it also means a bunch of new problems are introduced which don't currently exist.

I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale.

Nothing important of mine is allowed to live exclusively on Gitlab.com.

It seems like they are just growing too fast for their level of investment in their production environment.

One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much disruption if I needed to (yay open source!). Which I ended up forced to do on short notice when their CI system became unusable for people who use their own runners (overloaded system + an architecture which uses a database as a queue. ouch.).

Which put an end to what seemed like constant performance issues. It was overdue, and made me sleep well about things like backups :).

A while back one of their database clusters went into split brain mode, which I could tell as an outsider pretty quickly... but for those on the inside, it took them a while before they figured it out. My tweet on the subject ended up helping document when the problem had started.

If they are going to continue offering Gitlab.com I think they need to seriously invest in their talent. Even with highly skilled folks doing things efficiently, at some point you just need more people to keep up with all the things that need to be done. I know it's a hard skillset to recruit for - us devopish types are both quite costly and quite rare - but I think operating the service as they do today seriously tarnishes the Gitlab brand.

I don't like writing things like this because I know it can be hard to hear/demoralizing. But it's genuine feedback that, taken in the kind spirit is intended, will hopefully be helpful to the Gitlab team.

A lot of stuff from this made it's way into the chromebook. There's a verified boot process, hardware assisted key management, rollback protection, ...

And it's all open source and nicely documented for anyone who cares to look. With a bit of work you can actually create your own chain of trust and run your own verified boot process.

It's very cool.

Thank you for adding this detail... what you said makes WAY more sense to me now.

Circuit breakers aren't just there for the amusement value of watching a clustered system go into split-brain mode...

I.e. you would also want to be sure that your wiring was rated for 100% utilization, and that other circuit-breaker-like functions exist.

Fire is an actual thing, and figuring out the best way to recharge a halon system isn't exactly what you want to be doing.

You can certainly do some math/estimates based on looking at individual component specifications, but I like using a power meter (built in to some PDUs - which is a feature worth having, or you can buy one, they are quite inexpensive).

A system at idle vs full CPU vs full cpu + all disk will produce very different measurements.

Also keep in mind 80% derating - many electrical codes will state that an X amp circuit should only be used at 0.8X on a continuous basis (and the circuit breakers will be sized accordingly).

This is such a good example of the difference between Strongly typed and Stringly typed.

It's an interesting idea, but attackers would switch to using javascript. And I don't think it's realistic for typical users to have javascript default off.

I'd be happy to give it a try. I suspect rclone is the wrong tool for the job, but I haven't looked closely at the rsync.net service.

It does seem to use S3 behind the scenes based on URLs I've seen. The data loss incidents I've seen reported have tended to be after outages. Amazon Cloud Drive seems to keep an index mapping amazon cloud drive filenames to S3 objects, I suspect the index entry was corrupted/lost/rolled back. While the object likely still existed in S3 somewhere, that doesn't do much if users don't any way of accessing it.

Rclone is great. I wrote an integration to use it with git-annex (https://github.com/DanielDent/git-annex-remote-rclone).

Some of the supported providers (e.g. Amazon Cloud Drive) have a reputation for days-long service outages. Some users of Amazon Cloud Drive have even reported files going missing on occasion.

But the great thing with git-annex is you can have your data on multiple clouds (in addition to being on your own equipment), so partial or complete loss of a cloud provider does not need to result in availability or durability issues.

(1) Cloudflare does do BGP-based protection (rerouting of a packet) on their enterprise tier.

(2) They really aren't adding functionality into the network. If I put a cluster of Nginx servers in front of my web server, or if I have Cloudflare do it for me, I don't see how that fundamentally changes anything.

(3) The Cloudflare approach of an application layer proxy is feasible for any protocol which can be proxied. I've worked on SMTP proxies which do to SMTP what Cloudflare does to HTTP. Cloudflare also has a DNS proxy. Proxying is a perfectly valid and often incredibly useful technique.

I think we fundamentally agree.

But I would point out a subtle distinction: I believe TCP/IP and DNS provide an adequate building block for what needs to happen in a backwards-compatible progressive enhancement way.

I distinguish this from IPv6 which - because of how we chose to implement it - has created a ton of chicken-egg issues with the incentives for adoption.

But do they actually alter the architecture of the internet?

Or rather, how is Cloudflare making things worse or substantially different?

Example plausible flows in the life of a packet or HTTP transaction:

Before: Packet goes Comcast->Level3->HE->Destination After: Packet goes Comcast->Cloudflare->HE->Destination

Note that Destination is the one choosing to replace Level3 with Cloudflare. If Cloudflare ceases to be a good choice, Destination can swap Cloudflare for an alternate provider.

The use case where Cloudflare handles the HTTPS decryption (MITM-as-a-service) is a little bit different. Although it's become quite common, that's not the only way Cloudflare can be used. And I agree that's something where it changes things for the worse in some cases (but if it's a case of "Amazon's machines decrypt it for you" vs "Cloudflare's machines decrypt it for you", I'm less clear that a new problem is being introduced by Cloudflare).

There's a lot of different technologies and ideas that need to come together in an easy to use way.

Three of the most important: * Blockchains * Protocol-level anti-DDoS changes (protocols specifically designed to make them less suitable for use in in DoS) * https://en.wikipedia.org/wiki/Content_centric_networking

A pre-funding project I am working will achieve DDoS resilience without gatekeepers like Cloudflare, if I am able to bring it to market.

It turns out that there's actually a ton of academic research on a number of very promising techniques, in addition to the approaches I'm hoping to develop.

I see the issue more as UI/UX/ease of use/cost of adoption. Right now a lot of approaches you could probably defend a Ph.D. on the topic. They need to instead become approaches developers choose because it's the easiest and most cost-effective way to deliver a high-quality service.

As a strong believer in the end-to-end principle, I don't understand what you mean.

They have:

* A reverse proxy service, open to anyone

* A network-level traffic volume management system, open to those that can afford it

I can come up with a ton of scenarios where both of these services actually help with the whole 'end-to-end' issue.

So far Cloudflare appears to be acting in a non-destroying way. They take as content-neutral a stance as they can, and have had to defend lawsuits as a result of that stance. They recognize that the internet only functions and they can only be part of the infrastructure of the internet if they take that stance.

The greater concern is this: Companies like Cloudflare should not be necessary. They often are. And, with today's internet architecture, companies like Cloudflare must operate at scale to be a useful service with pricing that people can afford.

In a world where we accept the Cloudflare model, we also risk a set of gatekeepers emerging, and that set is "People who can manage terabits/second of inter-network connectivity at PoPs all over the world". It's potentially a troublingly small small group of entities.

I think these types of issues are why apple has started removing some of the encryption of the firmware itself.

I'm not sure the distinction is likely to matter. A small number of sites which account for a significant chunk of all traffic, and many of those sites are compatible with non-legacy internet protocols.

Quick plug for my related project: https://github.com/QA2/public-suffix-metalist - pull requests welcome.

I wonder how this leak affects their "vulnerabilities equities process".

The publicly available data would suggest that thus-far NSA-hoarded vulnerabilities are definitively known to actors who appear willing to act against US interests.

Vendor disclosure means those vulnerabilities can be patched and US interests can cease being vulnerable, but could also confirm NSA awareness of vulnerabilities - which could in turn cause attribution concerns for past or present operations the NSA is undertaking or has undertaken using these vulnerabilities (in addition to providing additional credibility to the leaker).

What a tangled web.

I've seen photonic switching refer to a lot of different ideas. Sometimes it's just a remotely-controllable add-drop multiplexer.

Does this type of photonic switching differ from CWDM/DWDM?

Are routers putting the signal onto different wavelengths depending on where they want packet to go?

A lot of the equipment I've seen is largely described in broad marketing-friendly terms which makes it hard to understand why the equipment is actually more useful than existing approaches.

Ubiquity gear around the 1Gbps rate isn't anywhere close to $18-20k.

And for very short links, there are projects like http://koruza.net/ which use standard (cheap) 1Gbps or 10Gbps media converters for free space optical links.

I'd love to know about affordable microwave 10Gbps equipment, I'm not sure we've gotten there yet.

While I don't believe this type of thinking about IP addresses represents a sustainable approach to an open internet, https://www.maxmind.com/en/geoip2-anonymous-ip-database and/or https://www.maxmind.com/en/proxy-detection-service may be what you are looking for.

60ghz, 80ghz, and higher frequencies (i.e. free space optics operating at or near the visible light spectrum) change the story considerably.

As frequency goes up and wavelengths get pencil thin, issues like "do I have clear line of sight" and "is there fog today" are more of a challenge than interference from others using the same frequencies/wavelengths.

There's plenty of bandwidth available in RF, but every option has it's tradeoffs. There's a limited amount of spectrum available at frequencies with convenient propagation properties for certain use cases.

This is a really interesting 'real world' anecdote. Do you know of any related data available publicly? i.e. "on our network, X bits in Y terabytes end up with undetected corruption, meaning that approximately Z% of downloads of a 2GB file will have at least one bit error".

While I agree a credit card might theoretically offer recourse, in practice I think what would happen is I would end up locked out of my iTunes account and I would discover that all my past "purchases" would be better described as "donations".

But it's crippled. I didn't want to waste my time evaluating a product other than the one which was interesting to me.

I tried within hours. And tried again multiple times with intervals on the order of weeks.