They mentioned nothing about CPU nor disk I/O, with the immediate cause being "reaching maximum number of connections", i.e. the following error from vitess vttablet:

    ResourceExhausted desc = transaction pool connection limit exceeded

My unsolicited advice: this looks like a classic case of https://bugs.mysql.com/bug.php?id=53825, which remains unfixed after 12 years.

Here's the solution space:

* Oracle added NOWAIT and SKIP LOCKED to MySQL 8.0, and called it a day. This is a bad solution because it just changes from one extreme (unbounded queue) to another (no queue).

* AliSQL added Statement Queue via hint (their older implementation extented the UPDATE syntax instead), which should work great, but do require modifying the write queries: https://partners-intl.aliyun.com/help/en/doc-detail/144127.h...

* At $prevjob we added block-level queue within InnoDB, based on an older experimental patch from AliSQL which they didn't actually bring into production. No app change required.

* That bug was reported by Facebook, but there is nothing relevant in their MySQL fork. They probably outsource the lock queuing logic to their Memcached layer.

The problem with this bug is that, without a correct analysis, the knee-jerk reactions are typically:

* adding more CPUs

* throwing in even faster disks

* bolting on a sharding layer like vitess

* blaming your DBaaS providers (well, they are to be blamed if they intend to stick with "vanilla Oracle MySQL")

Question to Github: I see you running pprof on vttablet, but have you run perf on mysqld?