Hacker News Reader

why is the website dependent on the database being up? Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable?

According to the outage Google Doc it's supposed to do this but it stopped working a little while ago:

16h33 | Yorick: Why have the 502s on GitLab.com switched to 500s?

16h46 | Slight concern that 500s have not yet gone away, but let’s focus on standing up postgres-04 first...

Right exactly, my first question would be why did this happen, because it could lead to some interesting insight about an unknown dependency.

> Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up?

You can't do anything with GitLab without the database being available.

> Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage?

We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind.

> My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer?

In the past we tested certain scenarios whenever necessary. For example, when introducing new parts of the database load balancer we'd test what happens if we terminate X or Y. However, we do not yet do periodic chaos monkey style testing, which has been on our todo list for quite a while.

> Do we at least have a practiced procedure for manual failover if not an automatic failover?

Failovers are actively worked on (https://gitlab.com/gitlab-com/database/issues/91, https://gitlab.com/gitlab-com/database/issues/86, https://gitlab.com/gitlab-com/database/issues/40). Right now our failover procedure is still quite painful, and it's easy to make mistakes. We held off using this approach for our maintenance because of these reasons, and also because a failover usually results in 2-3 hours of slow loading times. Ironically we ended up with just that.

> My third question would be what sort of alerts do we have, if any, on replication lag?

Plenty: https://gitlab.com/gitlab-com/runbooks/blob/810824765ec45534...

In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.

Wow sweet, I didn't actually expect a response! Nice work.

> We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind.

Or make it automatic when the database isn't available?

> In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.

Good to know. My next question would be "were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?"

> Or make it automatic when the database isn't available?

I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issues/4382 to keep track of things.

> were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?

I don't think you can detect corrupt WAL until it is actually corrupt. In those case the host won't start up, for which we already have various errors that cover that case (e.g. there's an error for when too few transactions are happening).

There also wasn't really any impending doom before we started things. Two issues that triggered this were:

1. A lack of `set -e` 2. Apparently when we stop a command using `gitlab-ctl`, it will time out if the process doesn't stop fast enough. I'm not sure what exit status we report in that case, but in this instance it resulted in the script continuing to run.

you should probably start using zalando/patroni on a k8s cluster. and scale the db in k8s and maybe use a lb on the outside of k8s with ingress/lb. it's way simplier than repmgr, etc.

I'm happy to be corrected here, but in response to:

I have to constantly be worrying about not putting "secret" things into the doc. Unless they have someone who's job is to take their real chat and transcribe it, in which case, that's great because that person is useful for a postmortem as well.

I think they did only have one person updating the document, from what I could as I was following the live updates - only very occasionally did I see someone other than 'Andrew' update the document.

> why is the website dependent on the database being up?

I believe they use "GitLab Pages" to host their main, public-facing website. If Gitlab.com (the app I mean) is down, then so is their website.