gwen-shapira

Congrats on the launch!

You may be under-selling LittleHorse a bit (here and even in the README). Few things I find cool about it:

- How easy it makes writing human-in-loop workflows (or these days, agent-in-loop). Everyone wants human-in-loop, but very few workflow frameworks make it simple.

- Using LittleHorse to introduce stateful guardrails to agentic flows. This helps with context compaction and other safety issues.

- The new (experimental?) generation of workflows from JSON. This simplifies things quite a bit.

When you run ALTER TABLE test DROP COLUMN c Postgres doesn't actually go and remove the column from every row in the table. This can lead to counter intuitive behaviors like running into the 1600 column limit with a table that appears to have only 2 columns.

I explored a bit what dropping columns actually does (mark the column as dropped in the catalog), what VACUUM FULL cleans up, and why we are still (probably) compliant with the GDPR.

This is a great question.

At this point, all transactions go through a single transaction coordinator that uses 2PC to coordinate all transactions. This makes many aspects of coordination fast, reliable and simple. It works well because OLTP systems typically have few DDLs an hour, if that (although we tested with hundreds of DDLs per second, just to be sure).

When we look at other types of transactions, OLTP systems need to handle many orders of magnitude more. Even small apps typically have hundreds of transactions an hour, while large systems require millions of TPS.

Getting from the current architecture to something that handles all transactional load of an OLTP system will be a journey. And arguably one better served with a different architecture, one not based on 2PC.

We are planning to open source it soon. Last few cleanups :)

We are still solving problems for all multi-tenant applications.

But AI companies, that tend to scale faster, also seem to benefit the most. So we wanted to make this clear.

(co-founder of Nile here)

You are right, it is hard to cover everything in one blog...

But the gist is that schemas and all the objects in them have few significant drawbacks:

1. Catalog overhead. Each schema has all the objects. PG Catalog doesn't scale to huge numbers. If you test anything that needs to work with the entire catalog, you'll run into issues in the 4-5 digits range (depends on number of tables, indexes, etc).

2. If you need to write a query that executes on all schema and aggregates results (for a report or something), you need to build your own tools.

3. Schema migrations and DDLs are increasingly hard to coordinate (I gave a whole talk about that here: https://www.youtube.com/watch?v=Nnz4VesXyUA

We basically didn't just build the distributed query and DDL layer, we also did some smart things in Postgres itself (blog TBD) that allow us to reduce the catalog overhead significantly.

I wrote a short guide to popular AI concepts, because I got a bit tired that reading a tweet requires a PhD. This is also the first part of a mini blog series about my adventures building AI Code Assistant with OpenAI, LangChain and pg_vector. Hopefully it will help you read every other AI thing that you run across...

It should read "code" instead of "pre".

I'm pushing a fix right now.

The OP mentioned that SQLServer does lock (but then, it doesn't use SSI - just SI).

The recommendation in PG docs to use a combination of SERIALIZABLE and READ ONLY transactions seems like a good one for read-heavy systems.

Possibly. I tried hard to have some memorable examples.

But I'll be honest: Every time I got review comments (or re-reviewed myself), it took a bit of time for my brain to warm up again into "transaction mode".

The big lesson may that concurrent transactions are pretty hard to reason about without external assistant like diagrams or test scenarios. I really like the system Postgres uses for transaction testing (AKA - deterministic simulation testing). Create scenario that match your business logic and then run them serially but with different ordering of statements and make sure the results are as you expect.

It is mentioned in the doc, but can be easy to mis-understand:

"If a pattern of reads and writes among concurrent serializable transactions would create a situation which could not have occurred for any serial (one-at-a-time) execution of those transactions, one of them will be rolled back with a serialization_failure error."

Note that it says nothing about the non-serializable transactions.

https://www.postgresql.org/docs/current/sql-set-transaction....

In case you are wondering why we don't have templates for your favorite language or framework - we are not yet done. We are planning to keep adding many more.

You can also request or vote, so we can prioritize with the community: https://github.com/orgs/niledatabase/discussions/211

hey :) I also like Rails acts_as_tenant. We searched far and wide for something similar for JS ecosystem and so far, didn't find anything.

We build our JS SDK by wrapping Knex (nice query builder, not quite an ORM) and injecting our logic to the connection process. You could do the same.

The issue is that there are at least 5 popular JS ORMs, so we don't get the same solve-it-once that we get in the Rails ecosystem.

Indeed! How did you guess?

We don't have per-tenant schema right now, but we have started designing multi-version experience and it doesn't seem too far from here.

Nile is different from DB per tenant in a single PG cluster in that it is much lower overhead per tenant. Since we offer a service, it will translate to lower cost.

The DB-per-tenant gets painful after the first few 100 tenants, but some SaaS have thousands of tenants, each with very low activity. Our model makes inactive tenants nearly free.

In addition, we also provide the "unified view", which lets you query all your tenants as if they were in the same DB. So you can connect as an admin and "select * from my_table" will give you the results for all tenants. Can be pretty handy.

Our goal is to make it transparent.

The architecture includes proxy layer that handles routing and maintains the connections, so we believe broken connections are avoidable.

Exactly that! Notion, Figma, Loom, Slack, Discord, Sentry have all been an inspiration toward the problem we are trying to solve.

Technically true, since we did all that with Postgres :)

Serverless isn't just hidden and priced in - it is supposed to scale automatically with your use.

AWS Lambda lets you think about "invocations" and not CPU and you trust that the CPU will show up when needed. At Nile we hope to let you focus on "queries per sec" and even "queries per second per tenant" while we scale the infrastructure as needed for you.

Incidentally, our product has a dashboard that shows you exactly these metrics :)

We don't have an official example yet, but I expect it will work. So far every JS framework that uses https://node-postgres.com works great and so no reason to think Drizzle wouldn't.

Noted :) I was checking out Xorm last night. It looks awesome!

It is a Postgres, so schema and migrations work exactly as they would elsewhere.

Tenants are your customers. It is normal for SaaS to have either a schema per tenant or one schema and tenant_id column in most tables. Nile provides you with both experiences at the same time.

Ooh, yes! I need to get a Django and other Python examples going.

With Flask it is fairly straightforward - you add the tenant configuration to the DB connection when you pick it up to serve a request. The Java and NodeJS examples do something similar if you need to see the idea.

great point!

If they travel a lot, you may want the data in one location (although it still depends, moving data is often faster than flying to a different region).

Tenant placement is especially great if their location has data residency requirements. We discovered that companies with many EU customers need this.

Nile co-founder here :)

Nice surprise to see our launch here. Happy to answer any questions...

Gwen here. Just wanted to mention how awesome Andy is. I haven't yet read his book, but as a senior developer - he has this great combination of wide domain knowledge, wide experience and very practical approach.

It is so good to talk with an engineer who thinks "What is the best solution for this product, this team, this budget, this timeline" not just an abstract "What is the best technical solution to a technical problem".

Aside from all the technical details, this mindset is my biggest take-away from our conversation.

Maybe worth pointing out the original interview (it is in a follow up tweet as well) and the blog posts that lead to it. There are so many more details.

I especially love the improvements that Vignesh and his team made to pgBouncer. Being able to modify the number of connections dynamically is super useful. And I still hope that Vignesh and team will implement the auto-tuning idea he shared at the end of the blog.

Video: https://youtu.be/DvblO-f2bqQ

The blog: https://blog.cloudflare.com/performance-isolation-in-a-multi...

The Scale presentation: https://www.socallinuxexpo.org/scale/20x/presentations/chall...

PG 11 is end-of-life, right? So you won't be getting new security updates. What's worse, libraries will stop supporting it over time, so you'll find it difficult to stay up to date in other places in your stack.

If I was running PG 11 in production, I'd be looking to upgrade sooner rather than later. Being on older-but-stable versions is good risk management, being on end-of-life version is not.

When I first became a product manager, my director, the amazing Chrix, taught me about Lead Bullets. It proved true and useful again and again. When you sell to engineers, there are no alternatives to a good product. You can't BS your way out of a product that is inferior to the competition on key dimensions.

The money quote: The customers were buying; they just weren’t buying our product. This was not a time to pivot. So I said the same thing to every one of them: “There are no silver bullets for this, only lead bullets.” They did not want to hear that, but it made things clear: we had to build a better product. There was no other way out. No window, no hole, no escape hatch, no backdoor.

I just shared an interesting fact that I learned while posting on Substack this morning. I don't know the reason, and generally I don't think of product changes in terms of good and evil.