throwdbaaway

So, what do you think of terraform?

It's puzzling to me that we, as software developers, spend so much efforts trying to automate such one off deletion tasks, and the automation would inevitably go wrong and result in data loss.

> Because the GDM data is immutable and thus can tolerate staleness, the query was also updated to read from replicas as well as Vitess primaries.

Any data that is fronted by the memcached caching tier can tolerate staleness right? There is not much difference between a short TTL of 1s versus an async replication delay of 1s.

Abusing vitess primaries is the root cause of this incident, and a similar incident can happen even without any scatter query.

> innodb global lock architecture has improved greatly since 5.1. ancient bug report from harrison is not indicative of problem here.

No, this performance cliff does still exist in the current mysql 8.0 branch. The Contention-Aware Transaction Scheduling added to 8.0 doesn't help the queuing problem with hot row at all. So facebook either doesn't suffer from the problem, e.g. all their writes are append-only so there is no hot row, or prevents the problem from happening, e.g. with API rate limiting built on top of something like memcached.

The improvement in innodb lock architecture mostly comes from splitting the locks into more granular levels. However, updates to the same hot row would require the same lock at the page-level. It is already as granular as it can be. So the tag of the bug report is correct, this is a problem of "lock queuing".

No, you don't need to make any network call on every read with listen/notify. It will essentially be a local cache, working the same way as etcd watcher, using one postgres connection per machine.

It has been almost 15 months since the previous blog post:

> Unfortunately, if we want to move to running more than one CONTROL process simultaneously for high-availability and better release management, that means we no longer have exactly one process managing the data

Do you guys still only have 1 CONTROL process?

Sounds like postgres listen/notify could be viable for your high-read-low-write use case? Or it is not scalable enough for the fleet size?

Somehow you are very confident that this github issue can be avoided by horizontal sharding? I am not so sure about that. If my hypothesis is right (see my other comment in this HN discussion), then it can be rather trivial to bring down any MySQL instance or shard, such that the end result would be similar regardless of horizontal sharding.

I would also back otterley's position that there exists many other companies with larger non-sharded MySQL instances than github's. They may or may not have better reliability than github.

I gathered from another comment here that this mysql1 database is still fronted by ProxySQL? In that case, the advice remains the same: don't bother with the proxy layer, focus on mysqld instead.

They mentioned nothing about CPU nor disk I/O, with the immediate cause being "reaching maximum number of connections", i.e. the following error from vitess vttablet:

    ResourceExhausted desc = transaction pool connection limit exceeded

My unsolicited advice: this looks like a classic case of https://bugs.mysql.com/bug.php?id=53825, which remains unfixed after 12 years.

Here's the solution space:

* Oracle added NOWAIT and SKIP LOCKED to MySQL 8.0, and called it a day. This is a bad solution because it just changes from one extreme (unbounded queue) to another (no queue).

* AliSQL added Statement Queue via hint (their older implementation extented the UPDATE syntax instead), which should work great, but do require modifying the write queries: https://partners-intl.aliyun.com/help/en/doc-detail/144127.h...

* At $prevjob we added block-level queue within InnoDB, based on an older experimental patch from AliSQL which they didn't actually bring into production. No app change required.

* That bug was reported by Facebook, but there is nothing relevant in their MySQL fork. They probably outsource the lock queuing logic to their Memcached layer.

The problem with this bug is that, without a correct analysis, the knee-jerk reactions are typically:

* adding more CPUs

* throwing in even faster disks

* bolting on a sharding layer like vitess

* blaming your DBaaS providers (well, they are to be blamed if they intend to stick with "vanilla Oracle MySQL")

Question to Github: I see you running pprof on vttablet, but have you run perf on mysqld?

>> Just the command line description and the manual of pg_upgrade makes you want to run away screaming. It's not ok.

Why is it not ok? This is pg_upgrade:

> Major PostgreSQL releases regularly add new features that often change the layout of the system tables, but the internal data storage format rarely changes. pg_upgrade uses this fact to perform rapid upgrades by creating new system tables and simply reusing the old user data files. If a future major release ever changes the data storage format in a way that makes the old data format unreadable, pg_upgrade will not be usable for such upgrades. (The community will attempt to avoid such situations.)

This is mysql_upgrade:

> Each time you upgrade MySQL, you should execute mysql_upgrade, which looks for incompatibilities with the upgraded MySQL server:

> * It upgrades the system tables in the mysql schema so that you can take advantage of new privileges or capabilities that might have been added.

> * It upgrades the Performance Schema, INFORMATION_SCHEMA, and sys schema.

> * It examines user schemas.

> If mysql_upgrade finds that a table has a possible incompatibility, it performs a table check and, if problems are found, attempts a table repair.

The description seems comparable, and if anything pg_upgrade looks saner by not attempting silly things like "table repair".

>> pg_upgrade should not be needed at all! It isn't for mysql. Or if it is needed it should be automatic.

It isn't for mysql since 8.0.16, where this becomes an automatic process. Personally, I prefer having an explicit step to perform the upgrade, instead of running a new binary to start the database process and also optionally perform the upgrade at the same time.

https://status.slack.com/2022-02-22 has been updated with this:

> A configuration change inadvertently lead to a sudden increase in activity on our database infrastructure. Due to this increased activity, the affected databases failed to serve incoming requests to connect to Slack. We introduced tighter rate limits on connection requests to reduce the load on the system. This meant that some people could not access Slack at all, but also that Slack would continue working for those who were already connected.

> Once the system had stabilized, we began lifting these rate limits to enable more connections to Slack. However, we moved too quickly and the increased activity affected the system again. We reinstated the rate limits and redirected some traffic to the database replicas to relieve the demand on our primary databases.

I wonder if their bottleneck is at vitess vttablet or at mysql.

I think it is rephrasing http://mikehadlow.blogspot.com/2014/12/the-lava-layer-anti-p... from a different angle.

I think we can lock the row in the accounts table instead, e.g.

    BEGIN;
    SELECT * FROM accounts WHERE id = $account_id FOR UPDATE;
    -- select/insert/update/delete transactions
    COMMIT;

This essentially serializes all reads/writes to data that belongs to the account. Not sure if this performs better or worse than using an advisory lock.

Right, I did have to learn this the hard way back in 2014.

I thought the established wisdom is to make schema migration compatible with both the old app and the new app, whenever it is possible? E.g. you can safely add a nullable column, and it shouldn't trip up the old app nor the new app, unless you are using some crappy ORM that does "SELECT *", or you are on an older MySQL version that may take hours/days/weeks to rewrite the whole table just to add a nullable column.

https://github.com/fabianlindfors/reshape, on HN front page a couple weeks ago, has some nice tricks to help with the incompatible cases.

> The native clients offer much better performance on every operating system.

Background blur uses significantly less CPU in Google Meet running as a chrome/chromium tab, than in native Zoom Linux client. Besides, Zoom Linux client also makes some arbitrary decision that my more powerful i7-3720QM is not qualified to have background blur enabled, while my less powerful i5-8250U is allowed to do so. Background blur works just fine with both CPUs in Google Meet.

From your POV, perhaps Zoom product/engineering is doing a good enough job. From my POV, what we have is a crappy Zoom Linux client and a half-arsed Zoom web client. And if corporate wants to give everyone Mac and Windows machines only, just so its highly paid engineers and highly paid product managers can share their screen over a Zoom session, who can argue with that logic?

Have you tried running slack as a tab in chrome/chromium/firefox? I just started doing that very recently on wayland, and it seems to work fine.

Zoom product/engineering is indeed hapless when compared to Google Meet product/engineering.

On distro with rolling release, I would say pipewire has been working fine since at least January 2021.

> Is this really a problem with Zoom or one with Wayland and the Linux distros?

I would say this is really a problem with Zoom, mostly due to the lack of attention paid to their web client.

For me, when it comes to video conferencing, the "table stakes" is working duplex audio over bluetooth, and so far only pipewire has solved this, which means Linux only.

After that, two very important features to have are screen sharing and background blur. For screen sharing, both Chrome/Chromium and Firefox have great support for screencast over pipewire, and so do many wayland compositors such as KDE kwin. As such, on a "pure" wayland system with no X11 window [1], screen sharing is also a solved problem.

Background blur is more ... complicated. When Google Meet rolled out https://ai.googleblog.com/2020/10/background-features-in-goo..., it immediately worked with chrome/chromium, on either wayland or X11. However, to this day, I think it still doesn't work with firefox, for whatever reason. Meanwhile, Zoom Linux client only started to support background blur (without green screen) since 5.7.6 released in August 2021, while Zoom web client remains hopeless.

In my previous job, we use Google Meet, and all is well with wayland. In the current one, we use Zoom, and I have to setup https://github.com/fangfufu/Linux-Fake-Background-Webcam as a workaround.

[1] This is very much doable these days, unless you want to run Slack desktop, which is hit-and-miss when running with --ozone-platform=wayland (unlike other electron-based apps such as Signal desktop), and of course the Zoom Linux app, which is just crap, as described in the article. Solution: run Slack and Zoom as Firefox tabs.

At first I thought it is a well-written post-mortem with proper root cause analysis. After reading it for the second time though, it doesn't sound like the root cause has been identified? At one point, they disabled streaming across the board, and the consul cluster started to become sort of stable. Is streaming to be blamed here? Why would streaming, an enhancement over the existing blocking query, which is read-only, end up causing "elevated write latency"? Why did some voter nodes encounter the boltdb freelist issue, while some other voter nodes didn't?

And there is still no satisfying explanation for this:

> The system had worked well with streaming at this level for a day before the incident started, so it wasn’t initially clear why it’s performance had changed.

But I totally agree with you that the first thing they should look into is to rollback the 2 changes made to the traffic routing service the day before, as soon as they discovered that the consul cluster had become unhealthy.

Good catch. If Roblox only uses consul for service discovery, things should continue to work, just slowly degrade over the hours/days. There should at least be one consul agent running on each physical hosts, and this consul agent has cache and can continue to provide service discovery functionality with stale data.

Dissecting this paragraph from the post-mortem...

> When a Roblox service wants to talk to another service, it relies on Consul to have up-to-date knowledge of the location of the service it wants to talk to.

OK.

> However, if Consul is unhealthy, servers struggle to connect.

Why? The local "client-side" consul agents running on each hosts should be the authoritative source for service discovery, not the "server-side" consul agents running on the 5 voter nodes.

> Furthermore, Nomad and Vault rely on Consul, so when Consul is unhealthy, the system cannot schedule new containers or retrieve production secrets used for authentication.

Now that's one very bad setup, similar to deploying all services in a single k8s cluster.

Interesting. MongoDB actually came to mind while I was reading the other comment here:

> No one uses DynamoDB alone: they bolt it onto Postgres after realizing they have availability or scale needs beyond what a relational database can do, then they bolt on Elasticsearch to enable querying, and then they bolt on Redis to make the disjointed backend feel fast. And I'm just talking operational use cases; ignoring analytics here.

Perhaps MongoDB is prime for a comeback?

This is the PR for the freelist improvement from that alibaba article: https://github.com/etcd-io/bbolt/pull/141

Just to be clear, we are talking about this item from the post-mortem right?

> We are working closely with HashiCorp to deploy a new version of Consul that replaces BoltDB with a successor called bbolt that does not have the same issue with unbounded freelist growth.

EDIT: I see what you mean. The freelist improvement has to be enabled by setting the `FreelistType` config to "hashmap" (default is "array"). Indeed it doesn't look like consul has done that...

I think they just made the switch to the fork that does contain the freelist improvement in https://github.com/hashicorp/consul/pull/11720

Took a major incident to swallow your pride? (consul, powered by go.etcd.io/bbolt)

Meanwhile, the state-of-the-art-not-facebook MySQL deployment still suffers from schema migrations (https://github.blog/2021-12-01-github-availability-report-no...):

> As next steps, we’re continuing to investigate the specific failure scenario, and have paused schema migrations until we know more on safeguarding against this issue.

In practice, you should typically only send write traffic to 1 galera node, and only fall back to "multi-master" for a split-second when promoting another galera node as the new writer node, either due to scheduled maintenance or unexpected downtime on the old writer node. This setup works great when being fronted by a fleet of stateless proxysql nodes. The whole DBA team can sleep well, without having to worry about whether the automated failover logic would actually work or not.

Also note that galera is just a set of API (https://github.com/codership/wsrep-API), which can be implemented against other databases to bring the same replication capability.

As a gentoo user, this postinstall behavior on debian/fedora/etc was rather shocking to me when I first encountered it. With gentoo portage, either when initially installing a package or when upgrading it, nothing under /var/lib is created or modified, and the process itself is never automagically started or stopped. The postinstall behavior actually feels like a regression toward windows-style installer.

The cassandra debian package is a good example. If I let the postinstall script runs, then it will use the default config to start a single-node cassandra cluster, creating a bunch of stuffs under /var/lib/cassandra, which then need to be removed first if I want to build a proper image from the disk. Doing the removal as part of the automation feels too risky to me, as it may only take one typo to accidentally delete /var/lib/cassandra in production. Instead, I usually just put a policy-rc.d script to disable all postinstall scripts from running.

I agree with the concern regarding leaky abstractions, but ultimately the developers should still own their services and be on-call for it, e.g.

- On-call dev deploys a buggy change that messes up some transactional data. Product owner passes the complaints received by customer service. On-call dev rolls back the deployment, writes a script to fix the bad data, and then prepares a post-mortem.

- On-call dev and on-call dba are paged because the database for the service is slow. On-call dev checks the service metrics and logs, and confirms that this is not caused by a deployment nor abnormal traffic. On-call dba checks the database metrics and logs, and finds some runaway queries due to silly optimizer. On-call dba kills the runaway queries, and proposes to add an index hint. On-call dev agrees, works on it, and asks on-call dba to review the change.

- On-call sec is notified because of a new rce vulnerability. On-call sec comes up with the steps to identify and remedy the issue. All on-call devs are notified to follow the steps, to ensure that their services are not impacted.

I agree with you, but at the same time I am also a little worry that this log4j2 exploit may inadvertently push us to the other extreme. Now every time I want some essential features to be added to a library, the log4j2 exploit will be brought up as a knee-jerk reaction.

At least we already got https://github.com/mattermost/logr, otherwise there could be years before we get to have a golang logging library with async logger.