phamilton

https://www.pgmustard.com/ is a great tool. There are some free alternatives for the actual tool, but their documentation is pretty great. For example, https://www.pgmustard.com/docs/explain/heap-fetches provides clarity that isn't obvious from the official pg docs.

Beyond those resources, here are a few useful things I've learned: 1. `explain (analyze, buffers)` is useful. It will tell you about hot vs cold data. One caveat: it doesn't deduplicate the buffer hits, so 1M buffer hits could be only a few thousand unique pages. But I still find it useful especially when comparing query plans. 2. pg_buffercache. Knowing what's in the buffer allows you to optimize the long tail of queries that perform buffer reads. Sometimes rebuilding an index on an unrelated table can create space in the buffer for the data the query needs. 3. Try using dedicated covering partial indexes for high traffic queries. An index-only scan is super cheap and with the right include and where condition you can make it small and efficient.

The tips above are especially useful in Aurora, where the shared buffers are huge (there's no page cache so it's the only caching layer).

FWIW, I've had to track down licensing and proof of IP assignment for fundraising rounds too.

> Not sure how flying a plane while it's essentially doing nothing is helpful to anyone.

When we train engineers for oncall, we take a look at incidents they saw while shadowing. Just X days shadowing doesn't really cut it, they have to be meaningful.

I see the comparison to flying pretty clearly.

Chaos engineering and game days are excellent ways to get engineers more experience in a system. Does the aviation industry have something similar?

I really like the idea of canonical maps/morphisms. The idea is that there exists one and only one possible implementation of a given type transformation. For code constructed of canonical morphisms, the compiler can prove correctness.

As an example, there are infinitely many maps from u32 -> u32. But only one map from Minutes -> Seconds. If you use u32 for your times, you need to rely on tests for confidence you have the right math. But if you use Minutes and Seconds, you can rely on the compiler to prove correctness.

There's also a difference between reading K&R and being comfortable with conventional C. Reading through source code from a mature project like PostgreSQL or the Linux kernel is very different from just knowing how malloc and free work.

Being able to follow along in the PostgreSQL source code has been helpful when we see production behavior we don't expect.

It never recovered from the loss of gentoo-wiki

I sent an email and it was blocked: "Recipient address rejected: Access denied."

Digging a bit, it seems the wait-list really started to be a thing in 2019.

> It takes a ton of investment.

I'm not an expert, but for an "assembled by hand" luxury product it seems like incremental capacity is much lower risk. They don't need to stand up a factory. Couldn't they hire and train 10% more technicians for assembly? And when demand softens, scale back?

> max capacity

As determined by what? Probably the current manufacturing facilities that they have strategically chosen not to expand.

Did he have some way to keep them straight? Seems like a recipe for forgetting which one is real and which one is fake.

We have this with APM monitoring, where every SQL query in the application becomes a span in a trace. It is very useful. We can quickly see the specific parameters that make an otherwise fast query slow.

> I would strongly doubt that count() without a where clause isn’t already highly optimized

It's actually not well optimized in PostgreSQL because of MVCC. See https://wiki.postgresql.org/wiki/Slow_Counting

TL;DR:

  The fact that multiple transactions can see different 
  states of the data means that there can be no 
  straightforward way for "COUNT(*)" to summarize data across 
  the whole table. PostgreSQL must walk through all rows to 
  determine visibility.

> I'm guessing this won't be a major problem. I don't know any proficient Rust or C++ developers that are not also proficient C developers.

There's a growing trend in high level languages like Ruby/Elixir/node to use Rust + FFI to optimize specific hot code paths. Likewise, pgrx is a much more approachable path towards postgres extension development.

All of these development paths have always been available with C, but Rust has made them more accessible to a wider range of developers. I'd wager a non-trivial number of Rust developers don't feel comfortable in C.

With an AGPL license, does that make it unlikely to be included in hosted environments like RDS?

My understanding of the spirit of the license is that it should be fine as long as modifications are made available. Anyone know of any existing extensions in RDS that are AGPL?

A proprietary product, but AWS Aurora Clones are CoW.

A reason I would still bias towards postgres is the maturity of managed solutions (including decoupled compute/storage solutions like Aurora and AlloyDB).

Are managed "proper OLAP DB" solutions competitive with managed RDBMS from a price and ease of use standpoint?

That's true, however a lot of spot usage ends up being heavily diversified across instance types in order to avoid momentary supply issues and optimize cost.

Across different families, CPU performance can vary by a decent amount.

My hope is that managed data layers become more available and the problem space of "custom data structures and indexes" can be decoupled from the even harder parts of writing a database.

Projects like pgrx (postgres extensions in rust) are starting to make it much more approachable to do so.

A little lower level: build on FoundationDB. (Projects like mvsqlite are super interesting here).

In both cases, we don't yet have fully managed providers. RDS seems to be flirting with opening up the extension ecosystem with trusted languages. I haven't seen any managed foundationdb offerings though.

> The authors claim it guarantees exactly-once delivery

I find this definition has morphed from one meaningful to developers into one queue implementations like to claim. I've learned this generally means "multiple inserts will be deduped into only one message in the queue".

The only guarantee this `exactly-once` delivery provides is that I won't have two workers given the exact same job. Which is a nice guarantee, but I still have to decide on my processing behavior and am faced with the classic "at most once or at least once" dilemma around partially failed jobs. If I'm building my system to be idempotent so I can safely retry partially failed messages it doesn't do much for me.

I've explored this space pretty thoroughly, including the Dynamo approach you've described. Dynamo does not have a strict guarantee on when items get deleted:

  TTL typically deletes expired items within a few days. Depending on the size and activity level of a table, the actual delete operation of an expired item can vary. Because TTL is meant to be a background process, the nature of the capacity used to expire and delete items via TTL is variable (but free of charge). [0]

Because of that limitation, I would not use that approach. Instead I would do Scheduled Lambdas to check for items every 15 minutes in a Serverless Aurora and then add them to SQS with delays.

I've had my eye on this problem for a few years and keep thinking that a simple SaaS that does one-shot scheduled actions would probably be a worthy side project. Not enough to build a company around, but maintenance would be low and there's probably some pricing that would attract enough customers to be sustainable.

[0] https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

Yep. OOMs are the most common cause. It's definitely low frequency. On the order of one in a billion. For some systems that's once a year. For us that's once a week. If that's an important job and it just gets dropped, then you've got a problem.

With the paid features to keep it from getting dropped things still can be painful. We have a lot of different workers, all with different concurrency settings and resource limits. A memory heavy worker might need a few GB of memory and be capped at concurrency of 2 while a lightweight worker might only need 512MB and have concurrency of 20. If the big memory worker crashes, its jobs might get picked up by the lightweight worker (and possibly hours later), which will then OOM and all its 19 other in flight jobs all end up in the orphanage. And now your alerts are going off saying are saying the lightweight worker is OOMing and your team is scratching their heads because that doesn't make any sense. It just gets messy.

Sidekiq probably works great outside of containerized environment. Many swear to me they've never encountered any of these problems. And maybe we should be questioning the containerization rather than sidekiq, but ultimately our operations have been much simpler as we've moved off of sidekiq.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks.

The key retry problem is "What happens when a worker crashes?".

RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot of issues with bugs in our code that would fail to acknowledge a message and the message would get stuck until that handler cycled. They've improved this over the past year or so with consumer timeouts, but we've already moved on.

The second problem we hit with RabbitMQ was that it uses one-erlang-process-per-queue and we found that big bursts of traffic could saturate a single CPU. There are ways to use sharded queues or re-architect to use dynamically created queues but the complexity led us towards SQS.

Sidekiq solves "What happens when a worker crashes?" by just not solving it. In the free version, those jobs are just lost. In Sidekiq Pro there are features that provide some guarantees that the jobs will not be lost, but no guarantees about when they will be processed (nor where they will be processed). Simply put, some worker sometime will see the orphaned job and decide to give it another shot. It's not super common, but it is worse in containerized environments where memory limits can trigger the OOM killer and cause a worker to die immediately.

The other issue with Sidekiq has been a general lack of hard constraints around resources. A single event thread in redis means that when things go sideways it breaks everything. We've had errant jobs enqueued with 100MB of json and seen it jam things up badly when Sidekiq tries to parse that with a lua script (on the event thread). While it's obvious that 100MB is too big to shove into a queue, mistakes happen and tools that limit the blast radius add a lot of value.

We've been leaning heavily on SQS the past few years and it is indeed Simple. It blocks us from doing even marginally dumb things (max message size of 256gb). The visibility timeout approach for handling crashing workers is easy to reason about. DLQ tooling has finally improved so you can redrive through standard aws tools. There are some gaps we struggle with (e.g. firing callbacks when a set of messages are fully processed) but sometimes simple tools force you to simplify things on your end and that ends up being a good thing.

I also question the revenue claim.

Google has a lot of revenue. Pinterest, Hashicorp, Uber, Twitch, Dropbox, etc. all have a good amount of golang and collectively have a lot of revenue. It might need a few more years to tip the scale, but it's closer than suggested here.

> but no more databases

Sort of. Twice as many rails hosts means more DB connections which generally means more load/memory on the DB or more load/memory on the external connection pooler.

It's only a bit of incremental load, but it's easy to overlook how many other systems need to run to make Rails scale.

LocalSet does not require Send. https://docs.rs/tokio/latest/tokio/task/struct.LocalSet.html

EDIT: Sorry to join in the multiple responses. Clarity for others: "current_thread" by itself does not relax Send, and many libraries aren't configurable to make everything use LocalSet.

Not quite what you are describing, but sort of in the same space, are SSR heavy frameworks like Next.js. Those solve problems by having the backend do most of the heavy lifting and streaming UI updates to the browser. This makes the browser a thin client instead of a node in a distributed system.

Personally, I think it's a great direction and hope it sticks.

The writer also has traffic that can attempt to acquire the locks. Reads and writes (though our highest volume read traffic is reader only).

Contention from the standbys do not get communicated to the writer.

We got hit with a good bit of the postgres iceberg this week.

VACUUM by default runs with the TRUNCATE option. This will, if possible and valuable, truncate the heap and return those pages to the operating system. This operation takes an AccessExclusive lock, which blocks reads and writes. It is written to be very deferential and only runs if nothing else is trying to take a competing lock.

AccessExclusive is a unique lock in that it is replicated to standbys. When the writer holds the lock, reads on the standbys will be blocked. However, the writer doesn't know about them, so its conservative approach of only acquiring the lock if there's no contention is completely thwarted.

Finally, if vacuum truncate is interrupted by lock contention it must repeat a scan of the heap to ensure the blocks are still empty before proceeding with the truncation.

All of these details combined meant our writer got stuck in a "grab lock, scan, be interrupted, release lock, grab lock, restart scan, be interrupted, release lock... etc" loop. The replication of this lock to the reader seemed to have batched together, so the readers never got to run in between the loop iterations. The vacuum never actually succeeded because it kept re-scanning. We had to intervene manually to disable autovacuum and cancel the in-progress jobs.

We've hit plenty of weird postgres issues, but this was a new one for us.

Edit: of note is the somewhat rare lack of clarity in the postgres docs. "VACUUM (without FULL)" is documented as only requiring ShareUpdateExclusive, which is why autovacuum is considered safe to run. Turns out that's not true.

The problem was that it was the same one machine that always had the extra 50% traffic. The eventual fix from Amazon was to randomly select in a tie breaker so that each machine had, on average, 2.1 concurrent requests (assuming 10 machines).

We've used both round robin and least outstanding connections in AWS and found many really weird phenomena over the years.

In general, LOC is more forgiving for uneven workloads, especially for Ruby services. If a slow request hits, that machine gets less traffic than the others and gets some room to recover and burn off its queue.

Where we found weird problems was at the extremes: very low concurrency and exceptionally high concurrency.

Low concurrency meant that one host consistently had 2 connections and another consistently had 3. That's 50% more load on one host. I believe AWS has fixed this and now tie-breakers are resolved randomly, but it was kinda painful to deal with. Our solution was to scale vertically and run fewer hosts. 19 on one host vs 20 on another was less jarring.

The other extreme was 1k+ websocket connections per host. When we autoscaled, it would add a new host. That host would have 0 connections and so the next 1k connections would all go to the new host. For that system, we changed it back to round robin.