Hacker News Reader

Comment by praneshp | Hacker News Reader

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more.

I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

View on HN

> GitLab cost everyone a day

GitLab isn't popular because of the stability of its cloud platform. It's popular because you can install your own instance for free practically anywhere with minimal effort.

I run GitLab CE on a box in my server closet for projects that involve livelihoods.

I know (because I use it and also host it at home, and like it). The point of my comment was HN's reaction.

Remember that HN is not a single-minded mob, during each incident a particular group was vocal on a given subject but that group may not have been the same individuals who were vocal each time.

Fair point, thank you.

You can also look at RhodeCode CE - supports git/mercurial/svn and has a really nice installer you can use to keep things up to date.

>It's popular because you can install your own instance for free practically anywhere with minimal effort.

If that is the reason why you use Gitlab, then why not try gitea or gogs? gogs is written in Go and provides a docker image or a drop in binary

I have been running Gitlab CE for around three years now and I am very happy with it. The interface it good. The access management works very good. And the best part? Keeping it updated is very easy on a ubuntu box. Just a couple of very easy commands.

I am so happy with Gitlab that I didn't even install Gogs even once to give it a try, though I know about it since it's initial days. For me Gitlab CE just works. Unless Gitea shows a 10x feature I am very unlikely to shift away from Gitlab CE.

GitLab is a lot more feature packed than Gitea/Gogs. Gogs is lightweight, good for personal projects, but if you're looking for something to deploy company wide with integrated everything, Gitlab is the way to go.

I think this is changing at a pretty fast pace ... well for Gitea anyways. It also looks like Gitea is lighting a fire under Gogs, as they appear to be iterating at a faster pace as well. Here's a very quick breakdown of what's going on.

Activity for the last 160 days. There were 175 commits to gogs and 720 commits to gitea.

https://gitsense.com/gogs-gitea/commits-160days.png

Activity for the last 60 days. There were 109 commits to gogs and 262 to gitea.

https://gitsense.com/gogs-gitea/commits-60days.png

https://gitsense.com/gogs-gitea/changes-60days.png

https://gitsense.com/gogs-gitea/changes-files-60days.png

The options and vendors directory are unique to Gitea and they account for a lot of the changes within the last 60 days. I was told the vendors directory is used to store dependencies but I don't know what the options directory is used for. And as the following shows, they account for a lot of the files touched, in the last 60 days.

https://gitsense.com/gogs-gitea/changes-options-vendor-60day...

Based on what I've read on Hacker News, the developer behind Gogs, tends to merge in changes in spurts, so it's hard to tell if this recent flurry of activity is a spurt or not. In this 365 days of activity, you can see the 3 spurts for Gogs so far.

https://gitsense.com/gogs-gitea/commits-365days.png

Regardless of whether or not Gogs will continue to develop at an increased rated, it looks like Gitea will.

Gogs is pretty sweet. After hearing about it here I installed it on my old home server with constrained memory because I couldn't meet Gitlab's (IMO) obscene memory requirements. It's perfect for my needs as a hobbyist and I even use it as a secondary Git server from work. It's also way, way faster than a self-hosted Gitlab or any public Git server, so even though my current server has more than enough memory to run Gitlab, I still use Gogs.

The only (minor) issue I've had with it was when I tried to push the OpenCV repository to it, just for the hell of it, on a heavily constrained VM (Debian with 256 MiB of RAM). The poor thing just couldn't handle it without crashing until I upped the memory to a couple gigabytes.

It might be related to git and not Gogs/Gitea itself.

I've found Gitea to handle larger repos fine after running the following:

  git config --global core.packedGitWindowSize 16m
  git config --global core.packedGitLimit 64m
  git config --global pack.windowMemory 64m
  git config --global pack.packSizeLimit 64m
  git config --global pack.thread 1
  git config --global pack.deltaCacheSize 1m

This reduces the memory used by git during certain operations.

The one who erred (and recovered) once is more valuable than the one who never made mistakes.

There is no basis for that assertion.

This is basic risk management. What matters is error survival, not absence of errors. Moreover, total absence of errors means accumulating dangerous risk. See NN Taleb, for example.

Sounds defeatist to me. Especially when we are not talking about literal moonshots, but backup and restore procedures. The IT equivalent of seatbelts.

I think at this point, recognising that software and software deployments invariably have bugs/issues is just realistic. The only setups that are free of them, are setups where the problems havent been discovered yet.

A well known library like curl, has had CVEs.

A not known library that also addresses the same circumstances as curl might have none listed.

Which would you prefer to use?

That's a cross purpose argument

The one with the CVEs because it has had eyes on the source enough to generate them. The devil you know and all that.

Exactly. They've made mistakes, and can recover from them.

The other hasn't had that chance.

Thus, try the first, as it is better tested in the world.

That's very different, though. I trust the company/organization that had to fire an employee for a horrible mistake. They are going to be very careful with who they hire or let near the code in the future, and had much more to lose than a single person.

But I don't trust the employee that was shown forgiveness for a horrible mistake. Some might learn to not repeat what they had done, while others learn that they can get away with things through the magical power of phrasing the situation in a positive light. And some might be mistake-makes-for-life. Not all people come out the other side stronger.

I see this play out with regularity in the board game hobby.

Manufacturer A: perfect product. Manufacturer B: omits some pieces, expedites replacements.

People will love and extol the amazing service virtues of B, not A.

There's another reasonable takeaway: GitLab will probably now prepare for this sort of thing better than the average competitor who hasn't been bitten by this.

Borges: "In Alexandria, it has been said that the only persons incapable of a sin are those who have already committed it and repented."

We'll certainly learn from this, post-mortem write-up issue is in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108

Every corporation ever has screwed up like GitLab if not more.

They are just better at hiding it, smoothing it over and lying.

That's Internet culture for you: being lovable trumps pretty much anything else.

Nah, no need to think about backup now.

Stuff like this only happens once :) haha,

To be fair, I suspect gitlab will a adopt a 'never-again' policy towards making sure their backups work.

>GitLab cost everyone a day

That's just nonsense. In most cases, the local repo should be more than enough to continue work.

Only for development right? If you were someone that handled issues for the team, or depended on webhooks triggering builds, you were still blocked.

I'm not trying to skewer Gitlab, I host an instance at home. I've also nuked 30% of the ports on our openstack cluster and screwed up everyone's day. I admire the transparency. I just wanted to call out HN's reaction, vs way smaller (technical) issues involving GitHub. But someone did point out downthread that HN's not a hivemind, so there is that.

The actual problem was with their DB, not the repo storage so I'm not sure what the actual fallout was, but people lost things like issues/merge requests, so I can see how workflow would be interrupted.

Yeah, kind of defeats the whole D part of DVCS when you centrally-host everything. GitHub is great for creating an open-source watering hole and becoming a de-facto standard for open source project hosting, but bad for mission-critical stuff if you're relying on something like that to be 100% available, especially when using a technology which is designed to be distributed.

It would have been really cool if GitHub had fully bought into the distributed model and built the wiki/issue tracker/etc such that they were included in the repository, so that they were also decentralized.

The "Plan B" is to have a local Gitlab instance and use the mirror feature to mirror it to Gitlab.com and/or vice versa.

You should always have 2+ production nodes in case one goes down.

Reply on news.ycombinator.com