phamilton

Sincere question: Has anyone figured out how we're going to code review the output of an agent fleet?

`'unbounded` isn't a bad idea. With my team when someone new to Rust hits the static type constraint I usually tell them an overly simplified "is not a reference" followed up with "The data does not have any constraints (or bounds) on its lifetime that this code needs to worry about."

I don't know if `unbounded` would immediately be understood by someone new to Rust, but it captures that `T: 'static` is a negating assertion. I.e. it largely says what it can't be.

A coworker once told me: "Undefined behavior means the code could theoretically order a pizza"

Hyperbole sure, but made me chuckle and is a nice reminder to check all assumptions.

To handle 25k qps with Rails, assuming 500ms per request, you need 12k Ruby processes. That probably means you run on 200 boxes with 64 CPUs each. Maybe you get a little bit of oversubscribing of processes to CPUs due to I/O. Conservatively it's still 100 boxes.

To handle 25k qps with Rust, you can run 10 boxes. Maybe less.

Deployments are complicated (even when we wrap them in simple abstractions). A "rare" delay that impacts 1% of boxes during a deploy will become common on 100 boxes but still somewhat rare on only 10.

Running a single rust process per box allows a connection pooler to be in-process. It's more efficient, it supports postgres prepared statements, it's one less operational headache, etc.

Beyond application performance, I've also found a non-trivial drag from the complexity of scaling Rails horizontally. Databases need connection pools, which must be external. Processes vs threads ratio need to be tuned and retuned and the application evolves. Deployments take longer to fully roll out a hundred containers. It's a bunch of tiny bits of complexity that add up.

Moving to a fully concurrent runtime (rust, golang, JVM, beam, etc) suddenly makes so much of that no longer a concern.

The language is incredibly flexible and allows for "DSLs" that are just ruby libraries.

A simple example: `3.days.ago` is a very commonly used idiom in Rails projects. Under the hood, it extends the base Number class with `def days` to produce a duration and then extends duration with `def ago` to apply the duration to the current time.

Taking that concept to a bigger extreme is this mostly unnecessary library: https://github.com/sshaw/yymmdd

`yyyy-mm-dd(datestr)` will parse a date str that matches yyyy-mm-dd format. It looks like a special DSL, but it's just Ruby. `dd(datestr)` produces a `DatePart`. Then it's just operator overloading on subtraction to capture the rest of the format and return the parsed date.

That library feels unnecessary, but the entire thing is 100 lines of code. The ease of bending the language to fit a use case led to a very rich ecosystem. The challenge is consistency and predictability, especially with a large team.

http://www.supersimplestorageservice.com/ exists and calls itself S4. It's a decent gag and the immediately came to mind when I heard S2 vs S3.

I introduced my kids (13 and 11) to the sofa problem last week as we installed this cabinet organizer from IKEA: https://www.ikea.com/us/en/p/utrusta-corner-base-cab-pull-ou...

After showing them a youtube video about the problem they saw clearly how the organizer is a sofa and even made a joke about it a few days later.

Relatable math is pretty great. Also really cool is showing how academia translates to enriching our lives in benign ways.

An interesting project for niche types: https://github.com/rick-de-water/nonany

It's not perfect but does allow some flexibility.

Asserts are convenient, but every time I encounter them I ask "Can the compiler prove this can't happen?"

In the example of "min(ages) > 0", making age a NonZero type renders the assert unnecessary. Rust even has some fancy perf optimizations it can do with that information. It's a win all around.

I'd add that cloud services are dramatically cheaper when your application is able to tolerate preemption. Spot pricing in AWS is around $0.015/vcpu/hr.

That still going to be more expensive than a dedicated server, but it's much closer. If you have cyclical traffic patterns and other elastic workloads, the gap can be almost non-existent, even without considering personnel.

Thanks for sharing. It matches my experience.

After many years of a lean team serving high scale traffic (> 1 million monthly active users per engineer), most abstractions between customer and data seem to turn into performance liabilities. Any major changes to either client behavior or data model are very likely to require changes to the other side.

There's a lot to be said for just embracing the DB and putting it front and center. A favorite system we built was basically just Client -> RPC -> SQL. One client screen == one sql query.

Has anyone ever actually moved a mature application from one database to another and found that the code around calling the DB was a major painpoint?

I'm all for the unit testing argument, but in an active SaaS business I've never seen the hypothetical database change where a well architected app makes it smooth. I have certainly moved databases before, but the performance and semantics changes dwarf the call sites that need updating. Especially in Rust, where refactoring is quite straightforward due to the type system.

Yes. Running multiple multi-billion row Postgres DBs on AWS Aurora.

The number of rows isn't that consequential honestly. Btrees are very fast at navigating deep hierarchies. It's the volume and complexity of traffic that matters.

On AWS Aurora we can run 10 readers to handle our peak midday traffic (10M+ daily active users).

I wouldn't shy away from doing billion row DBs on-prem if that's what the finances detail. Postgres can handle it. But the new wave of scalable Postgres (alloydb, aurora, neondb) will make it easy.

A testament to the quality of Anandtech is that in 2011 I started a job at Micron on their SSD team and the first thing they said was to go read some articles on anandtech about how SSDs work. They covered slc vs mlc, trim, etc in better detail and in a more approachable way than anything else.

I've leaned on Anandtech ever since as a go-to source for understand technical innovation in hardware. Thanks for making everything that much easier to understand.

Postgres + Rust is one of the most exciting intersections of tech I've seen in a while.

There's external tooling like his project, but postgres extensions in Rust are exciting.

Full extensions via pgrx have been cool to see, but plrust + pg_tle is also starting to show up.

If you aren't familiar with TLE (Trusted Language Extensions), it is a postgres extension from AWS that created some privileged interfaces for procedural languages (used for user-defined functions) to do some extra stuff. Right now it's mostly auth-related hooks but my hope is that it expands in the future.

Plrust is a procedural language extension for Rust, allowing user defined functions written in Rust.

The combination of those two could open up a world of rich extensions usable in managed hosted environments like RDS.

Very erlang/otp. Joe Armstrong used to rant to anyone who would listen that we used databases too often. If data was important, multiple nodes probably need a copy of it. If multiple nodes need a copy, you probably have plenty of durability.

Even if you weren't using erlang, his influence (and in general, ericsson) permeates the telecom industry.

I've been out of the space for about a decade, so I can't say how things have continued to evolve, but the only consistent antidote to fraud has been a trusted third party.

I spent a lot of my time back then pulling in metrics from third parties like Nielsen (who do traditional TV viewership numbers) as well as companies who specialized in "viewability". We could include that data in our real-time auction and bidders would adjust what they were willing to pay based on the reputation of the ad opportunity. At least that's the theory. In practice there was so much indirection and reselling that it was hard to say what was true.

As far as the technical answer: it's a game of cat-and-mouse but we found some interesting metrics we could dig in on. One was that fps of a video hidden behind another element on the page would have a drastically different rendered fps than one in the foreground. That would be different between browsers (and browser versions) but the viewability vendors would find all those quirks, measure, and analyze to give us a normalized score. For a hefty fee, of course.

As a college student trying to figure out where I wanted to go next (industry vs academia) I had an epiphany that there was intrinsic value in doing things. Even things that were neither novel (academia), profitable (industry), nor impactful (nonprofits).

While my career has found a balance between impact and financial sustainability, I have a special place in my heart for projects that are otherwise unnecessary but happen just because it brings the creator joy.

When flash started to fade for games, many developers moved over to adtech where their skills were still valued. Flash continued for a few years to power ads as well as shims around other videos that would collect data and even trigger secondary ad auctions.

One of the final nails was the infamous Chrome 45 aka the Chromepocalypse in the video ad world.

Chrome 45, in the name of performance, defaulted to only loading flash from 3rd party domains after a "click to load". This was bad for ads for obvious reasons, but it was much much worse due to an implementation detail.

In order to get the page laid out properly, Chrome loaded the flash component and then suspended it after a single frame. From an ad perspective, that was enough to trigger the impression pixel, signifying that the ad had been shown and the advertiser should be billed. However, the ad was not shown and there was no way for the ad to detect it had been suspended. Just a nightmare. We (I led the effort at BrightRoll) had to shift a ton of ad auction behavior to special case Chrome 45 and it limited what kind of ads we could serve.

That was the inflection point away from flash for ads. While ad formats like VPAID supported JS/HTML5, they didn't start getting popular until after Chrome 45 was released.

> indexes often don't help.

An "index on expression" should perform the same regardless of the input column types. All that matters is the output of the expression. Were you just indexing the whole jsonb column or were you indexing a specific expression?

For example, an index on `foo(user_id)` vs `foo(data->'user_id')` should perform the same.

A few things:

Redis Data Tiering - Redis Enterprise and AWS Elasticache for Redis support data tiering (using SSD for 80% of the dataset and moving things in and out). On AWS, a cache.r6gd.4xlarge with 100GB of memory can handle 500GB of data.

Local Files

> I can also cache it on disk, but then I need to think about cache expiration myself.

Is the challenge that you need it shared among many machines? On a single machine you can put 20 million files in a directory hierarchy and let the fs cache keep things hot in memory as needed. Or use SQLite which will only load the pages needed for each query and also rely on the fs cache.

S3 - An interesting solution is one of the SQLite S3 VFS's. Those will query S3 fairly efficiently for specific data in a large dataset.

I mostly have used AWS Aurora there, which is significantly better than vanilla MySQL or Postgres and both are similar enough.

In Aurora, Postgres has Aurora Limitless (in preview) which looks pretty fantastic.

As far as running yourself, Postgres actually has some advantages.

Supporting both streaming replication and logical replication is nice. Streaming replication makes large DDL have much less impact on replica lag than logical replication. As an example, if building a large index takes 10 minutes then you will see a 10 minute lag with logical replication since it has to run the same index build job on the replica once finished on the primary. Whereas streaming replication will replicate as the index is built.

Postgres 16 added bidirectional logical replication, which allows very simple multi-writer configurations. Expect more improvements here in the future.

The gap really has closed pretty dramatically between MySQL and Postgres in the past 5 years or so.

Some things that are nice in Postgres and missing in MySQL:

- Create constraints as NOT VALID and later VALIDATE them. This allows you to create them without expensive locks.

- explain (analyze, buffers). I miss this so much.

- Row level security.

- TOAST simplicity for variable text fields. MySQL has so many caveats around row size and what features are allowed and when. Postgres just simplifies it all.

- Rich extension ecosystem. Whether it's full text search or vector data, extensions are pretty simple to use (even in managed environments, a wide range of extensions are available).

Is that (and more) enough for me to migrate a large MySQL to postgres? No. But I would bias towards postgres for new projects.

The goal isn't to enforce a certain query plan. The goal is to execute queries efficiently.

The planner/optimizer is good at its job. In general, better than I am. If I can inform it about a particular shape of data, all queries around that data will improve. If I tell it exactly how to execute one query, only that query improves.

Additionally, setting statistics to 10000 is really easy. Easier than tweaking a query. Creating custom statistics is more work and may or may not be worth exploring as a quick fix. But if you know the data is shaped irregularly, why wouldn't you make sure the planner knows about it?

`set local` allows tuning query planner settings scoped to a transaction, so you don't have to do it globally.

All the knobs for query planning and cost estimation can be scoped to the transaction with `set local <key>=<value>`.

That's valuable, but cranking statistics up to 10000 and/or creating custom statistics can go a very long way to helping the planner understand the dataset more fully.

I like it as an emergency measure, but I often see them used when there's a shallow understanding of operating the db.

Before using a hint or rewriting a query to force a specific plan, I try and push the team to do these things:

1. Run `vacuum analyze` and tune the auto vacuum settings. This fixes issues surprisingly often. 2. Increase statistics on the table. 3. Tweak planner settings globally or just for the query. Stuff like `set local join_collapse_limit=1` can fix a bad plan. This is pretty similar to hinting, so not a huge argument that this is better beyond not requiring an extension.