tibbar

I hired someone from the thread about six years ago. So it worked for someone at least once.

Note how Robert mentions a sharp increase in back-patched commits. Many of these bugs have indeed been around, silently, for a long time.

Sure. This one already came up. [0]. In general, for a system like Postgres, many bugs affect only "extreme edge case" scenarios, but fixing them might create severe performance issues or require changing or removing other important features. So it's better to not fix them.

[0] https://www.postgresql.org/message-id/flat/c22286ad6268347c6...

By the way, Robert and the other maintainers are incredible to interact with. There are so many reports to triage and bugs to fix, but they're unflaggingly polite and quite generous with their time. I don't know of another community like it.

Old bugs are sometimes undesirable to fix because other systems/users have come to rely on their behavior.

I will add more example links in the top-level comment, but here is the thread for that bug. [0]. tl;dr timestamp vs timestamptz comparisons aren't consistently ordered in a pathological edge case, which means that index scans can return the wrong answer. There's no way to fix it without either redefining the operators in a surprising way or disabling index scans in this case.

[0] https://www.postgresql.org/message-id/flat/c22286ad6268347c6...

I did a little AI-assisted bug-hunting in Postgres this summer in between jobs, and the haul was staggering. Multiple serialization anomalies (eg [0] [1] [2]), foreign key integrity issues ([3] [4]), planner correctness bugs such as [5]. Various CVE-worthy vulnerabilities. An unfixable timezone bug [6]. The list got so long that I had to triage what I even had time to report, and settled for a dozen or two out of hundreds of candidates. Now imagine the entire world doing this.

[0] Write skew observed under serializable isolation: https://www.postgresql.org/message-id/flat/CA+COZaBo+ZpKgMvx...

[1] Serializable anomaly: duplicate primary keys: https://www.postgresql.org/message-id/flat/CA+COZaBOiiRPmEfX...

[2] SERIALIZABLE can commit write skew after SSI summarization: https://www.postgresql.org/message-id/flat/CA+COZaCtK=UQbeQw...

[3] Fix temporal foreign key validation of pre-existing rows https://www.postgresql.org/message-id/flat/CA+COZaChq_ukm2o7...

[4] Lost lock during toasting allows FK violation https://www.postgresql.org/message-id/flat/CA+COZaC-2-GQabq+...

[5] Partition pruning can incorrectly exclude a RANGE default partition https://www.postgresql.org/message-id/flat/CA+COZaDXrfTaBjLE...

[6] timestamp vs. timestamptz comparisons inconsistently ordered under DST : https://www.postgresql.org/message-id/flat/c22286ad6268347c6...

Although the simplest exact algorithm for TSP is O(n!), you can use dynamic programming to get the difficulty down to O(n^2 * 2^n), which is quite tractable at n=20.

However, sometimes you can construct exact solutions to much larger real-world data sets (tens of thousands of points) by exploiting structure in those graphs.

I used Hex at $PREVJOB and it was easily one of my favorite tools. It is maybe the one BI tool that I have felt truly attached to in my career

Ah, I was using "enterprise" in the colloquial tech/B2B sense as a company of ~1000 people or so. There are about 8,000 businesses of that size in the US, but they employ about half the private workforce. It appears to me that Cassandra adoption is very heavy within that market segment, for instance.

(Keep in mind that pushing relational databases to 10k writes/sec is not optimal! Well before you get to that point, there are much more cost-effective options.)

On to a few other nits...

> 10k writes a second is a problem that a very small number of systems have relative to all deployed web servers, the normal case is far closer to HN.

Hmm ... I would estimate that the majority of deployed servers today use a distributed database, operating out of clusters for e.g. Google, Meta, Amazon, Netflix, Microsoft, CDN operators, and so on.

And, of course, HackerNews is also using distributed databases, so it's not a great counterexample...

> Re HN using other systems for search and api, that’s more related to supporting companies the vc funded - Algolia

HN uses Google BigQuery to support a company they VC funded? Anyway, this is "no true Scotsman"-ing databases.

The issue is that data complexity is also exploding in the age of LLMs, so it still takes a lot of archaeology to get reliable (as opposed to plausible) numbers.

I think this dynamic also requires more data engineers, but I don't personally see the split in demand between those roles that you describe. Data analysts often have business or domain context that data engineers lack.

Perhaps we'll start rolling data analysis work into "operations".

Wait, data analysis/data science is still a thriving human job sector! It's just that you can do way more analysis, way faster, than you could before.

Postgres/MySQL and friends are incredibly good at handling the kind of workloads they were designed for (transactional reads and writes, and perhaps some aggregate queries if you can plan your indexes in advance.) You can scale that quite far, although if you get big enough you will eventually run out of room there (around 10k writes/sec or so.)

However, relational databases are a poor fit for writing logs, for performing analytical queries, and various other purposes that most reasonably large businesses today need. You can try to use your relational DB for them, but you will create real performance issues very fast; and even if you can get the DB to scale up for the load, you'll end up paying way more money that you would from using the right kind of DB for that workload.

The killer app for UUIDv7 is logs. High-volume, distributed, non-public data that you probably want to partition by time as well. Log volume typically grows much, much faster than transactional writes, and any reasonably big organization is going to want to keep that data out of their relational database.

As far as relational databases go, another common workload they're a bad fit for is analytical queries. If you employ any data analysts or operations staff, they can bring even the biggest deployment of Postgres/MySQL to its knees pretty fast with relatively simple queries. Read replicas buy you a little runway there, but they're just not a good or cost-effective way to power BI workloads.

> To take one example this high traffic website (HN) uses one db

First of all, HN does not use a single database! E.g. it powers search with Algolia and API queries via Big Query, and that's just for external traffic.

Second, HN is not a high (write) traffic website. It's a forum. We can actually see from those serial IDs that there are about ~50 million all time comments. HN is about 20 years old; that works out to about 5 writes per minute. Of course growth has increased over time, but it's still probably less than 1 write per second.

So, I'd agree with you that lots of interesting websites can power their primary application write traffic with a traditional relational database, but I think you'll find that those properties end up needing several other databases anyway for other purposes.

There is a good discussion of the tradeoffs here [0]. It is often fine expose UUIDv7 publicly, e.g. if the event time is not a secret and you use good entropy for the non-timestamp bits. However, sometimes you might decide that you care, and use a random identifier instead. In those cases you probably won't benefit from a second UUIDv7 identifier, since you're already paying the cost for a primary key index, but perhaps if you are also doing certain kinds of range queries and have a certain kind of storage engine you still get something out UUIDv7.

In other words, it depends. And it might not matter at all if you don't have much data, and/or it's not being generated in a distributed way, and/or you're not doing range queries or partitions in a way that can take advantage of it.

[0] https://news.ycombinator.com/item?id=45622045

This is actually a very interesting question. I assume that you are referencing extensions that add shards to Postgres, since it runs on a single server out of the box. You made me curious how those extensions/rewrites handle sequences.

It looks like Citus (one such extension that runs shards on separate workers) does something a little like what you described. Each node gets an ID, which becomes the high bits of the sequences it generates. So serial IDs are monotonic within a shard, but not across shards.

Aurora PostgreSQL Limitless uses different abstractions, but also ends up reserving different chunks of IDs for different routers to use.

Others have mentioned their difficulty in distributed systems; but even for traditional enterprise workloads, serial IDs don't scale very well. They force all writes through a single, transactional bottleneck, so you can't shard your data. UUIDv7 and friends lets you guarantee that records get a unique identifier without that coordination cost. There are also scenarios where you need to have access to an ID before the write to the database succeeds.

Of course, lot of event/log data never hits a relational database at a all.. it gets piped right to a data lake. But those records still need IDs, too.

Many years ago, I was working at a small shop that used Oracle databases. We had different tenants for different customers. However, we still wanted records across different tenants to have unique IDs, so (as I recall) all writes to all tenants phoned out to a central sequence generating function. This became a major source of latency for us. This is jut one of many ways that serial IDs become annoying if you're doing anything other than running a 1990's-style RBDMS on a single server.

Anyway, once you try to solve this problem, there are lots of schemes that give you (almost certainly) unique IDs without coordination. Whether you pick UUIDv4 or v7 etc.etc depends on other factors. But one nice feature of UUIDv7 is that it is kinda, sorta serial, which is helpful for e.g. partitioning to work well.

This is such a conspiratorial mindset. This very forum thread is packed with people providing code and confirmation of working exploits. Leading security organizations have confirmed RCE exploits with video and provided detailed writeups of how the bug works. The PHP maintainers coordinated the release. The government is involved.

Two things can be true: it's a real, exploitable CVE 10, and it's hard enough to construct that you need a good harness + verification process to keep your LLM from cheating. So, the exploit may not be easy to construct, but it doesn't actually require the artificial settings that the OP implied.

> What we demonstrated > Pre-authentication RCE on WordPress, Drupal and Joomla

[0] https://www.wiz.io/blog/php2shell-php-rce

These days people often mean the cost/performance Pareto frontier. That is, it is the best performance you can get for that cost profile.

No one likes Claudisms, especially people who write a lot. The AI cliches are nearly unbearable. However no one has quite figured out how to fix it.

That really is a fine book, it brings back good memories.

However, I don't think this article is really complaining about style. Scott Alexander has great style, the clarity/grace people would find much to love about him.

The complaint is about the length of his writing, and the content, ie focusing on endless arguments as opposed to honoring prior traditions.

Yes, a lot of pieces just aren't written for beginners, they are part of an ongoing and voluminous discourse. Writing something short just wouldn't be very interesting (for the experts), they want to engage with all of the prior art.

I think that when references to the prior work are only implicit, pieces tend to be nearly unreadable for casual readers.

However, when interesting counter arguments are stated explicitly, the long articles can still be kind of fun, and personally think this is something Scott Alexander does pretty well, for instance. He is not the main offender of unreadable long rationalist blog posts, of which there are many.

The issue is that you've got to build shared context before people can follow an unstructured dump of everything in your brain. You have to tell a story that makes sense. Also, ideally you figure out how to filter out the stuff that other people simply don't need to know. It is not easy, but it is a skill you can learn.

Wow, it is amazing to me that there is a wikipedia page about people who read Scott Alexander's blog. Still, it is (or was) a darn good blog back in the day.

It reminds me a bit of elaborate theological arguments that build up in denominational seminaries. The authors just have an obsessive compulsion to catalogue and resolve every minute detail, details that only really matter if you've already accepted many other dubious and compounding premises. IE, the articles are really for existing converts.

I don't know, I think rationalist bloggers tend to love arguing, and if you identified some assumption that they didn't address they'd just as happily write 50,000 words on that instead. :-)

I think Scott in particular just really enjoys writing. He is very articulate and obviously just brimming with clever connections and turns of phrase. And he can generate a lot of words very easily.

When I was younger, sometimes I would just get carried away by the splendor and awe of it all and soak in the 25,000 word articles (or whatever they are.)

Today I would say that either he has fallen off a little, or I've just grown a little weary of the form, but I don't really get the same rush out of his writing anymore; and I notice more of the motivated reasoning and cheap tricks.

Still, sometimes I'll come across one of his posts and think 'damn, Scott's still got it.'

Of course there is tremendous value in many traditions. However, when the world is changing quickly as it is today, it is also very important to check in frequently with reality, to face facts, as it were.

Many people also use tradition as a cover for bias, bigotry, and bullying. When one has been on the other end of that, one learns to view blind tradition with a healthy dose of skepticism.

And yet, still, there is much of value there. Ironically it takes some wisdom to learn which is worth keeping and which is not.