throwdbaaway

The article is about migrating from RDS Postgres 9.5 to RDS Postgres 12, which is only feasible with Bucardo. Major version upgrade is less of a pain point these days with logical replication, and even Google Cloud SQL started to support logical replication since about a week ago, after a painfully long wait.

Meanwhile, you are talking about schema migration? In that case, gh-ost indeed is the best tool available. With Postgres, most types of schema migration can be done instantly with just a metadata change, or otherwise be done concurrently without locking the table. So a tool like gh-ost is not as vital, but still valuable for edge cases such as:

- altering the column type: requires table rewrite

- adding an index concurrently onto a busy server: a throttle would be nice

For GCP, both ansible modules and terraform modules are actually generated from https://github.com/GoogleCloudPlatform/magic-modules, so their "production readiness" are the same.

I understand that mitchellh himself personally created a bunch of cloud modules for terraform at the beginning, and those were likely of higher quality than whatever created by some internal developers assigned by Google/Microsoft, and might be slightly better than the AWS modules maintained by community.

Anyway, when it comes to ansible versus terraform, we shall move the discourse to states management instead. With ansible, you don't have to deal with states, but will need to clean up the cloud resources separately. With terraform, you can use the tool to clean up the cloud resources easily, but then you also have the headache of managing states. Plus, whenever you change something, there is always the nagging feeling that it will do a destroy/recreate instead of an in-place update.

Well, I'll be the first one to admit that I was naive to expect <5% variance prior to this experience. But I guess you are going to far by framing this as a common wisdom?

In the HN discussion about cockroachdb cloud report 2021 (https://news.ycombinator.com/item?id=25811532), there was only 1 comment thread that talks about "cloud weather".

In https://engineering.mongodb.com/post/reducing-variability-in..., high profile engineers still claimed that it is perfectly fine to use cloud for performance testing, and "EC2 instances are neither good nor bad".

Of course, both the cockroachdb and mongodb cases could be related, as any performance variance at the instance level could be masked when the instances form a cluster, and the workload can be served by any node within the cluster.

Yeah, that's exactly what the GCP engineer recommends, and likely why the final benchmark in the article was done using a c5n.9xlarge.

Still, there is no guarantee that after stopping the instance on Friday evening, you would get back the same physical host on Monday morning. So, while using dedicated hardware does avoid the noisy neighbor problem, the "silicon lottery" problem remains. And so far, the data that I gathered indicates that the latter is the more likely cause, i.e. a "fast" virtual machine would remain fast indefinitely, while a "slow" virtual machine would remain slow indefinitely, despite both relying on a bunch of shared resources.

I didn't expect identical performance, but a 10~20% variance is just too big. For example, if https://www.cockroachlabs.com/guides/2021-cloud-report/ got a "slow" GCP virtual machine but a "fast" azure virtual machine, the final result could totally flip.

The more problematic scenario, as mentioned in the article, is when you need to do some sort of performance tuning that can take weeks/months to complete. On the cloud, you either have to keep the virtual machine running all the time (and hope that a live migration doesn't happen behind the scene to move it to a different physical host), and do the painful stop/start until you get back the "right" virtual machine before proceeding to do the actual work.

We discovered this variance a couple of months ago. And this article from talawah.io is actually the first time I have seen anyone else mentioning about it. It still remains a mystery, because we too can't figure out what contributes to the variance using tools like stress-ng, but the variance is real when looking at MySQL commits/s metric.

> If you need more than 5% accuracy for a benchmark, you absolutely have to use dedicated hosts.

After this ordeal, I am arriving at that conclusion as well. Just the perfect excuse to build a couple of ryzen boxes.

If I understand correctly, coordinated omission handling only matters if the benchmark is done with a fixed rate RPS right? In this case, it looks like a closed model benchmark where a fixed number of client threads just go as fast as they can.

edit: Oh, perhaps wrk2 still relies on the timer even when not specifying a fixed rate RPS.

> EC2 X-factor?

> Even after taking all the steps above, I still regularly saw a 5-10% variance in performance across two seemingly identical EC2 server instances

> To work around this variance, I tried to use the same instance consistently across all benchmark runs. If I had to redo a test, I painstakingly stopped/started my server instance until I got an instance that matched the established performance of previous runs.

We notice similar performance variance when running benchmark on GCP and Azure. In the worst case, there can be a 20% variance on GCP. On Azure, the variance between identical instances is not as bad, perhaps about 10%, but there is an extra 5% variance between normal hours and off-peak hours, which further complicates things.

It can be very frustrating to stop/start hundreds of times for hours to get back an instance with the same performance characteristic. For now, I use a simple bash for-loop that checks the "CPU MHz" value from lscpu output, and that seems to be reliable enough.

> You mean Slackware users on some random forum.

Believe it or not, that's actually the official slackware forum. And whatever solution those guys come up with, it will likely become the official solution.

> Besides, the solution they came up with uses XDG autostart which has nothing to do with systemd.

The slackware solution involves a project that nobody has heard of before, just so it can imitate the "user-level service" feature provided by systemd: https://github.com/raforg/daemon

> Not to mention that it's not even doing the exact same thing as the Gentoo solution and running two more commands in addition to pipewire.

The slackware solution requires starting those 3 processes (pipewire, pipewire-media-session, pipewire-pulse) separately from 3 different .desktop files, likely because the daemon tool above can't properly reap the pipewire-pulse process (not sure whose fault is this though).

On the other hand, the gentoo solution can start all 3 processes with just 1 .desktop files, because `pkill` takes care of it. Simple and effective.

I think the key difference, in this case, is that the slackware guys are trying their best to imitate a systemd feature, while the gentoo guys seem to focus more on finding the best way to allow users to enjoy pipewire.

Well, at least they take the cake when it comes to running CMS with the largest heap size?

It is also quite ironic that they paid for Zing C4, and then came up with "P90 average RPC queue time" as the metric, ignoring Gil Tene's wisdom that you should never average percentiles.

Another problematic statement is this:

> which in turn decreases request latency and increases throughput

Gil Tene wants you to measure p99.99 or pmax, so that you will pay for Zing C4 to get lower tail latency, while sacrificing some throughput. If you end up getting both lower tail latency and more throughput, then something is already very wrong with the existing setup (i.e. the full GC).

> In the fall of 2016, we were dealing with hundreds of thousands of MySQL queries per second and thousands of sharded MySQL hosts in production.

> Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes.

I think the "today" QPS numbers are still doable with a properly tuned single-writer galera cluster running on machines with TBs of memory. Of course, with Slack workload, there would be too much historical data to fit into a single host, so I can see the reasons to shard into multiple clusters/hosts.

Still, the numbers seem a little off. Let's say back in fall 2016 there were already 200K write QPS at peak, with 200 sharded hosts accepting write. That's just 1K write QPS at peak per host on average, and let's say 20K write QPS at peak for a particularly hot shard. What could be the bottleneck? Replication lag? Data size? I don't think any of the articles from Slack has talked about this.

What Vitess provides is invaluable, especially the very solid implementation of secondary index. But sometimes I feel like it is being used/advocated as a sledgehammer ("just keep sharding") without looking at what could be done better at the lower MySQL/InnODB level, in exchange for a much more costly cloud bill.

> It allows having user services without having to hack around it with cron

Recently, I started using pipewire, which is designed to rely on a user-level service to start and stop the pipewire process.

To bypass this requirement, the slackware guys came up with an elaborate solution that wraps the pipewire command with a daemonize tool that is supposed to cleanup after itself: https://www.linuxquestions.org/questions/slackware-14/using-...

Meanwhile, the gentoo guys just did the simplest thing possible: https://github.com/gentoo/gentoo/blob/f788f38/media-video/pi...

Of course, with both of these workarounds, the semantic changes from "start this process once the user logs in" to "start this process once the desktop environment starts", but the end results are identical for almost all users.

https://news.ycombinator.com/item?id=27082522 - this guy is cut out for Extreme Programming. If something is good, why not have it all the time, right?

We mortals, are likely more suitable for moderate programming, with infrequent informal pairing when necessary.

For me, this looks like the perfect implementation of a very scalable config service, where the read RPS can get very high, but the config values are rarely changed. The consumers of the config service can be either frontend applications or backend application, thus websocket is a great choice.

How could they push it when:

1. WhatsApp didn't want to make an official port for Maemo / Meego, for whatever reason.

2. WhatsApp bans your number if it detects that you are using a non-official port.

3. WhatsApp became the essential communication platform.

The unified messaging on N900 was great, but without WhatsApp, I was that weird guy who was still stuck with SMS.

Jolla, with its android app support, fixes that problem for me. Blame Acton, not Elop.

Consider yourself lucky if "being better" is the only reason to initiate a multiyear cloud migration project. We are doing one at the moment where everything is going to be objectively worse with the new provider.

I can probably do multi-cloud with Oceania Web Services and Eurasia Cloud Platform, and still end up spending less money *and* having fewer issues, than using this provider.

But your point still stands. Multi-cloud is the way to go.

https://buttondown.email/nelhage/archive/notes-on-some-postg... - Be aware that there is a very bad performance cliff in postgres when using savepoints.

That's a good point, but sadly not how tools like terraform marketing themselves. If terraform markets itself as "we make clean up easier for that 10% of time when you need it, and to achieve that goal, we give you 3 times more headache by asking you to keep states", I am sure many more people wouldn't make the trade-off.

I like this statement:

> The first law of queues is: Every queue should have a maximum size. Queues must not grow unbounded.

Unfortunately, unbounded resource is often used as a hammer to try preventing other smaller issues, with https://github.com/grpc/grpc/issues/8603 being a perfect example.

Ansible? Compared to terraform, it should require less effort from the user, without having to worry about states. Compared to aws cloudformation or gcp cloud deployment manager, it should require less effort as well, without having to learn the different idiosyncrasies of these proprietary tools.

> It's just drivers for the cloud API in question

Exactly, and if you are on gcp, both the ansible modules and the terraform modules are even generated from one code base: https://github.com/GoogleCloudPlatform/magic-modules

Not really. In terms of functionalities, I consider ansible as a superset of terraform, without all the state management stuffs. Distributed systems are hard, and I will just let my cloud providers to be the single source of truth of all the states.

Here's a comment from the author of ansible when terraform was first released: https://news.ycombinator.com/item?id=8100036

> One of the things shown in the Ansible examples are how to do a cloud deploy in one hop, i.e. request resources and also configure the stack all the way to the end, from one button press, and can also be used to orchestrate the rolling updates of those machines, working with the cloud load balancers and so on, throughout their entire life cycle -- all using just the one tool.

That's a very good point, I wasn't aware of that.

Of course not, don't be ridiculous. My point is that whatever man-power and resources required to run the elasticsearch service can be considered as fixed cost, while the same 47.53% profit margin can continue to "scale" indefinitely, regardless of whether you have 10 customers or 10,000 customers. That's the beauty of it.

As we can see elsewhere in the comments, your biggest competitor appears to be the various kubernetes operators. With kubernetes as the common stack across multiple cloud providers as well as on-prem datacenters, the 10,000 customers can now contribute to the same operator project, to possibly get somewhere close to, or even exceed, what AWS can do with a small team of engineers.

Perhaps anticipating this trend, GKE now has an "autopilot" mode, such that a 30% profit margin is already included in the node pricing. That's one hell of a moneymaker.

I was nodding along until this:

> and THEN we think about how do we make money from it

Do you really need to think? Looking at the on-demand pricing in US East, a m5.4xlarge.elasticsearch instance costs $1.133 an hour, while a m5.4xlarge instance costs $0.768 an hour. That's 47.53% of extra money. And like you said, it only requires a small team to build and maintain the service.

It is no coincidence that all cloud providers are trying to ramp up their hosted services for open source software, even GCP, who historically only focused on their own proprietary stack. There's a lot of money to be made.

Noticed the same too. It is likely that we are both impacted by the very aggressive default of 1ms for `internal_poll_interval`: https://github.com/ceph/ceph-cm-ansible/pull/308

So now we know who is really responsible for the whole MySQL utf8mb4 fiasco -- these 2 guys sitting in a diner, conjuring up a brilliant scheme to cover 4 billions characters, which turned out to exceed the actual requirement by more than 2000x.

September 1992: 2 guys scribbling on a placemat.

January 1998: RFC 2279 defines UTF-8 to be between 1 to 6 bytes.

March 2001: A bunch of CJK characters were added to Unicode Data 3.1.0, pushing the total to 94,140, exceeding the 16-bit limit of 3 bytes UTF-8.

March 2002: MySQL added support for UTF-8, initially setting the limit to 6 bytes (https://github.com/mysql/mysql-server/commit/55e0a9c)

September 2002: MySQL decided to reduce the limit to 3 bytes, probably for storage efficiency reason (https://github.com/mysql/mysql-server/commit/43a506c, https://adamhooper.medium.com/in-mysql-never-use-utf8-use-ut...)

November 2003: RFC 3629 defines UTF-8 to be between 1 to 4 bytes.

Arguably, if the placemat was smaller and the guys stopped at 4 bytes after running out of space, perhaps MySQL would have done the right thing? Ah, who am I kidding. The same commit would likely still happen.

EDIT: Just notice this in the footnotes, and the plot thickens...

> The 4, 5, and 6 byte sequences are only there for political reasons. I would prefer to delete these.

So UTF-8 was indeed intended to be utf8mb3!

Lack of hint is definitely a controversial topic. There are situations where there is only one obvious way to execute the query, in which case a query hint would help to prevent the database engine from going berserk. However, there are also situations where different query plans should be used according to the parameters, in which case a query hint would harm.

We now also have some empirical evidence on how the query planner evolve/devolve based on this decision. InnoDB always relies heavily on query hint, and recently it has given up on trying to get the planner to support some simple pagination queries correctly, after killing too many production systems that didn't add a hint: https://dev.mysql.com/worklog/task/?id=13929. It basically declares that "we know our query planner sucks for this class of queries, here's a flag to disable that portion of the query planner"

Meanwhile, in recent releases, Postgres started to allow statistics to be created for correlated columns, so that the query planner can have the necessary information to come up with the correct plan: https://www.2ndquadrant.com/en/blog/pg-phriday-crazy-correla...

> Another surprise coming from MSSQL is PG does not cache plans at all. It actually spends time replanning every statement coming in.

This is actually the thing I hate the most about MSSQL. If I allow plans to be cached, then some queries would end up using some bad plans intended for parameters with very different data distribution. If I disallow plans to be cached, then the query planner can take 1~2 second just to come up with a plan for a rather simple query with 2 levels of nesting. A rock and a hard place.

It sounds like the author's conclusion, that he would pick MySQL over Postgres, was heavily influenced by his recent encounter with a Postgres performance cliff where some django ORM code inadvertently uses subtransactions: https://buttondown.email/nelhage/archive/22ab771c-25b4-4cd9-...

For that, I would like to counter with the 10+ years old quadratic locking bug that remains unsolved in InnoDB: https://bugs.mysql.com/bug.php?id=53825

Ah you are still on Xperia X, that explains. For me, Xperia X was great for a year or so circa 2018, before:

* everyone started to abandon the perfectly capable android 4.4 API, mostly due to the support of TLS 1.2

* payment / transport / banking / e-commerce / travel / food / coffee all started moving to mobile-first

Following that, there was a rather rough 2~3 months transition period when I upgraded to XA2 in early 2019. It has been smooth sailing since.

I once was a strong proponent of small form factor too, but I am afraid that boat has sailed, and well, XA2 Ultra actually still fits in mine :)

> I have used a Linux phone for 12 years or so. First the N900 and later the Jolla/Sailfish ones.

> None of them has been a great product.

Likewise, and I disagree. The phones that support Sailfish X or later are all great. With Xperia XA2 or Xperia 10, you can install micro-g via f-droid, and then most android apps should work fine. Yes, it is not ideal that we need android/ios apps to function properly in a society. Blame it on SV. Oh, that's us?

Anyway, back to Jolla. A good example is that if the LCD screen breaks, I can just buy a replacement, ssh into the old phone, and scp from it to the new phone (this actually happened). That's a Linux phone.

I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad.

I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.