throwdbaaway

ZFS ARC actually works the same way, i.e. the records would remain compressed in memory.

When running PostgreSQL, I would probably go with `primarycache=all` for the data dir, a largish ARC cache, and a smallish shared buffers, to take advantage of the filesystem compression.

When running MySQL, I would probably go with `primarycache=metadata` for the data dir, a smallish ARC cache, and a largish buffer pool, to still benefit (slightly) from the filesystem compression.

Not just out of the box. There is a good comment at https://news.ycombinator.com/item?id=7488498 that listed a few key features of InnoDB:

1. Change Buffering - If your benchmark result shows that MySQL is faster than PostgreSQL, it is likely due to change buffering. For random read/write workload that doesn't fit into the memory, this feature is fantastic. This is available since the very first commit of InnoDB, and of course also has been causing random corruption during crash recovery throughout the years: https://jira.mariadb.org/browse/MDEV-24449 (fixed in MariaDB, still exists in MySQL)

2. Page-level Compression - PostgreSQL still doesn't support this, but I am fine with relying on ZFS filesystem compression when running either MySQL or PostgreSQL. Arguably it should be better for the database engine to do its own compression, but see this blocker level bug that impacted FreeBSD: https://jira.mariadb.org/browse/MDEV-26537 (MariaDB only, but feel free to search bugs.mysql.com for other compression related bugs)

3. Clustered Index - The main topic in the Uber drama. I think InnoDB has the better design, since scaling write is harder for a RDBMS. I also think that InnoDB implementation causes a severe performance cliff, when it comes to optimistic update versus pessimistic update.

> Generally in MySQL we send queries massaged to a point where optimizer doesn’t have to think about anything.

https://dom.as/2015/07/30/on-order-by-optimization/ - Wisdom from the guy who managed to get MySQL to work at facebook scale.

Furthermore, the ecommerce platform in step 1, which may or may not be the same platform as the one in step 4, would likely do nothing to combat the fraud, as doing so could lower the GMV.

If I understand correctly, there were:

- issues with tail latency and cost when using gp2

- issues with generally bad performance when using st1

- issues with reliability when using gp3 (as an early adopter of aws "GA" product)

- issues with insufficient disk space when using local-attached nvme

- issues with confluent licensing cost

And tiered storage solves all of that.

The thing is, I have not seen kafka struggling with disk performance when running on gcp pd-ssd. Perhaps even pd-balanced would do the trick, as indicated by rmb938's comment. I am glad that you guys finally landed on a boring solution now, but things have been rather boring for years with another cloud provider. Perhaps there is no material impact from the high tail latency when using gp2, and you just needed a better contract negotiator? Surely the tail latency would be worse now whenever data need to be pulled from S3?

I think it is fair to expect that even with the interface actively serving traffic, the data transfer over s3 should still be faster than the data transfer over wifi. Someone should take a closer look at that S3 throttling?

It is interesting to see that they could move 76.8GB of data in 1 hour per-instance with the old architecture, while I can move 120GB of data in 1 hour locally between 2 laptops over wifi.

Here with Ryzen 5700G, it takes almost 2.5 hours to compile chromium (doing one at the moment). This is with /var/tmp/portage mounted on tmpfs.

On AWS, there are serious issues such as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1331150..., that only requires a simple sysctl workaround, yet AWS would only put the workaround into the Amazon Linux images.

Meanwhile, the same issue was actually fixed once and for all on GCP: https://news.ycombinator.com/item?id=8732356

While I also quite dislike the autoloading behavior of rails, in practice, I haven't found it to be abused too much in large rails projects. So let's say we take away autoloading, it is likely that mostly models code will need to deal with circular dependencies.

Meanwhile, in golang projects, within-package circular dependencies are happening blatantly everywhere, and somehow I don't see much complain about it.

It doesn't sound like the guys understand how ZFS CoW works? So let's say the machine crashed after InnoDB had written 4KB out of a 16KB data file page to the filesystem. Due to CoW, the old data file page would still be there, unmodified. In contrast, on XFS or EXT4, without the doublewrite buffer, the data file page would be corrupted.

I have a lot of respect for Marko Mäkelä, as he seems like the only one who tries to steer InnoDB away from an evolution dead end. For example, in https://jira.mariadb.org/browse/MDEV-24449, he fixed a corruption bug that had existed since the very first InnoDB commit. At the same time, he also tries to simplify the internals, unlike Oracle who just churns out features along with regressions.

But there are always more corruption bugs within InnoDB. I am not even sure the log checkpoint mechanism is sound, where it does a fsync of the data files first, before getting the checkpoint position from memory, which may have moved forward after the fsync. This behavior also exists since the very first InnoDB commit.

Actually, there was a significant regression. Many many years ago, Oracle decided to drop the support for the gpl-licensed readline altogether, likely because they can't ship it with MySQL Enterprise. To this day, Percona still carries a small patch to add that functionality back, which is great because I wouldn't touch any CLI without readline.

One thing to consider is that when running on-prem, you will be able to procure disks that advertise 512B sector size, and actually performing well when doing so. With that, you can use 8KB record size on ZFS, and still obtain the same 5.5x compress ratio, assuming you are currently using 4KB sector size with 64KB record size on ZFS.

With 8KB record size, there would be no read/write amplification caused by postgres. More importantly, ZFS fragmentation would be much less of an issue.

Of course, disks procurement can be a nightmare process, which is why AWS is printing money.

Indeed. It looks like they went from ZFS 0.8.x (lz4) to 2.x (zstandard) in one go. That would be too yolo for my taste, but the guys were under time pressure due to the rather sudden performance degradation. Nodes replacement would probably buy some time, to allow the upgrade to be done in 2 phases.

> Average write operation duration decreased by 50% on our fullest machines

This is probably not a fair comparison.

On the existing lz4 machines, the zfs pools are already badly fragmented, making it difficult to find regular blocks: https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSGangBloc...

On the new zstd machines, the zfs pools are still pristine, after being restored from backup.

If we want to isolate the effects of migrating from lz4 to zstd, we also need some new lz4 machines for comparison.

> On the db side, in MySQL, it's possible to disable the doublewrite buffer (which is another form of overhead, which, in theory, ZFS should avoid), but nobody really knows if that is 100% reliable or not.

It is. Actually, that's the best thing about running MySQL on ZFS. The doublewrite buffer is not just another form of overhead, the doublewrite buffer is *the bottleneck* for any workload with moderate amount of write (until it got revamped in 8.0: https://dev.mysql.com/worklog/task/?id=5655).

I see your point. Perhaps the best way is to use TOAST compression for wide field values, and then rely on ZFS filesystem compression for others? For some context, in MySQL, both the table-level compression and page-level compression are problematic, while ZFS has served me well, thus I have a lot more faith in it.

I had no problem working on a 600K lines java project. Didn't need no go-to definition.

Unfortunately that's just how postgres works? I thought the rule of thumb is still to allocate 25% of memory for postgres 's shared buffer, and 50% of memory for OS buffer cache (i.e. ARC when using ZFS)?

With cloud network-based storage like EBS or pd-ssd, I doubt you will notice any IOPS-related perf penalty when running a RDBMS on top of ZFS. Assuming a mixed read/write workload, you will likely hit the disk write throughput limit first, as ZFS just writes ... a lot more, due to how blocks are stored internally: https://github.com/openzfs/zfs/issues/6584#issuecomment-3848...

On an innodb database, I get about 3x compress ratio with ZSTD compression, and ZFS still has to write about 2x more than EXT4.

Compressed ARC has been a thing in ZFS for a long time. So let's say you allow ZFS cache to take 50% of total memory. With 4x compress ratio from ZSTD, you will be able to cache 2x the total memory. It is amazing.

1 month in after more than 10 years with nano. Can recommend it as well.

I have never found the need to use go-to definition, until I started working on some golang codebases. The code organization in these codebases is typically abysmal, with multiple files in the same package participating in N ways nested orgy.

Of course, micro is written in golang, and it seems like the author is doing just fine without relying on LSP.

Judging by https://github.com/hashicorp/vagrant/issues/7263, I would say that they are the "GNOME" of devops tooling. The quickstart is great, and you feel empowered when your use cases are supported. Beyond that, good luck.

The result of your poll is exactly my fear -- p99 seems to be at the peak of the hype cycle.

I am totally a fan of Gil's work, but I think he over-dramatized the impact of tail latency, with the rather extreme example of "If a typical user session involves 5 pages loads, averaging 40 resources per page" from his talk.

Regarding coordinated omission, I am not totally convinced that an open-model system is what happens in production, and I am actually fine with a closed-model system when running benchmark, as long as p100 is acceptable. My goal is to achieve the maximum TPS without stalling, and I don't really care that much whether p99 is 10ms or 100ms.

Other than streaming, a less obvious single threaded bottleneck would be the LCS compaction, with a rather ugly workaround of using multiple keyspaces as noted in https://issues.apache.org/jira/browse/CASSANDRA-7949#comment...

With scylla TPC approach, there will be one shard per core, and I think that shall allow more LCS compactions to run in parallel. Theoretically.

I have seen the workaround on a very large cassandra cluster. Meanwhile, I never thought to myself that "well, the 200ms p99 with g1gc sure is nice, but my users would be so happy if I can lower it to 20ms". So, if I am scylla, TPC would be my main selling point, coupled with real world examples, but somehow they want to focus on p99.

> The "reduced latencies" really reduces down to performance, where tps was a first order proxy for performance.

Thank you so much for this comment. I have come to a similar conclusion recently -- as long as the P100 latency is acceptable (e.g. all requests are served within 1 seconds), the only thing that matters is the TPS.

Back in 2016, I stumbled upon "How NOT to Measure Latency" by Gil Tene, and it really opened my eyes, especially the part about coordinated omission from benchmark tools.

Of course, after a few days, I also learnt that Gil was behind the Azul Zing JVM with the pauseless C4 garbage collector, and thus it makes sense for him to emphasize on measuring latency correctly.

At around the same time, GCP also boasted about having "consistent single-digit millisecond latency" for its BigTable offering, with Cassandra being the obvious target to attack. I was sold.

Then came Scylla, with a focus on maintaining low tail latency while running on fewer larger machines. I tested one of the early version with cassandra-stress, and the result was worse than cassandra. But I continued to follow Scylla blog posts with great interest.

Recently, I saw https://www.p99conf.io/, and something "clicked" when I read that it was sponsored by Scylla. Suddenly the hype of P99 seems to be wearing off. The video from Gil above is still correct, but I think it is applicable more to exceptional cases, e.g.

- the JVM enters full GC and do no meaningful work for seconds/minutes

- the InnoDB engine stalls for seconds/minutes due to a million different reasons

For normal cases, I haven't found the P95/P99/P99.9 metrics to be that useful. Instead, something like PostgreSQL/MySQL slow log threshold, where anything that exceeds a soft P100 target is logged, seems to be more useful.

Back to the article. If we only concern about TPS, under the "real-life" workload with Gaussian distribution, Scylla beats Cassandra by 2X. So that's it, 2X better performance on one hand, Apache vs. AGPL on the other.

(Of course that's an oversimplification. Scylla's shard-per-core architecture also allows it to avoid some silly single-threaded bottlenecks in Cassandra, but nobody wants to talk about those.)

> ZFS/btrfs has nothing to offer someone working on an S3-like service, or a globalized database, either.

Why would that be the case? Let's say I want to setup a vitess cluster (planetscale = globalized database). I can easily delegate the compression logic to zfs, which works way better than innodb table compression or transparent page compression. I can rely on zfs atomic write to safely disable innodb doublewrite, which has been the number one bottleneck to scale writes until it was enhanced recently.

Thanks for the numbers. Perhaps hype is not the right word. It is just interesting to see that some older hardware can still be used to achieve the state of the art performance, as the bottleneck may lie elsewhere.

I thought Ant is fantastically profitable, because it takes a 30% cut from the profit generated by the Yu'e Bao fund (used to be the largest in the world), with the money coming from the remaining balance of all the Alipay accounts.