fabianlindfors

getbb.app

I migrated off Conductor to getbb.app a while ago and can highly recommend it.

For cloud agents I just built plugins in bb for boat.dev and smolmachines.com. Works like a charm.

I moved to getbb.app. Been really happy with it for the past few months. The plugin functionality is really cool.

Although useful, I think migration linters like this one don't give enough peace of mind. I have maintained a zero-downtime schema migration tool for several years now that tries to cover all the different ways one can shoot oneself in the foot: https://github.com/fabianlindfors/reshape

It ensures migrations don't lock the database but maybe more importantly, it allows zero-downtime rollouts for your application as well by supporting both the old and new schema during the deployment, and automatically data between them. It also handles backfills and more that usually require multiple, separate deployments when using standard SQL commands.

We are! Got a few pilot customers that are using it but still early days

Sure. It's a bit hard to quantify because we need to run these on bare-metal machines and the unit cost is pretty high.

We run our test workload as well as a few other startups' that we have onboarded on one AWS ARM bare-metal machine at $1.7k a month. We don't saturate that machine fully either so I'm not really sure what the amortized cost would be. Certainly more expensive than Github Actions but not by a crazy amount, and the value we get out of it is way higher than GA.

Firecracker, with some tiny modifications to better manage memory for the deep nesting of forks

That's the point, we don't review the test code either. Our platform gives us a UI for inspecting not the test code but what actually happened during the test. Like a browser replay, the results of a database query, assertions against those, etc.

This is much more information dense than something like the tests and is a representation of what actually happened during the tests, rather than what the test itself did (which I agree sucks to review, especially AI-generated).

The framework is our own that bundles/adapts some familiar components: Jest-like asssertions, Playwright browser API, typed database client from Bun, Kubernetes client, etc. The tests are written in Typescript but the main code doesn't have to be (just runs containerized in the environment).

We've been building the platform and using it continuously since May but setting it up on a new project takes like 1-2 days of largely autonomous coding agent work. We are just two engineers on our team but have been onboarding other startups to the platform recently so there are a few different teams using it now for their own codebases. It's fully generic so works for any infra or stack.

Forking!

We run the entire stack (browser, frontend, backend, database, etc) in a Linux VM, so latency between each of the pieces is as tiny as can be. This is quite different from "standard" E2E tests I've seen where the test browsers uses something like a persistent staging environment.

The real key is that we can fork that entire Linux VM to take different paths down our testing scenarios, and can run multiple of them in parallel. Tests may look something like:

  new user signs up:
  |- creates a todo
     |- ...
     |- ...
  |- creates a list

The two nested tests then start from the exact same point, where the previous test left off, but can run in parallel. With enough hardware, the full suite will run as fast as the slowest branch of the test tree. When we switched away from our previous integration test suite to this (not E2E), our tests actually became faster because they share setup through the forking.

We try to avoid reviewing AI-generated code and built our own testing framework and platform to make that possible.

Our principle is that our tests should give us enough confidence to not have to look at the code (which ends up being true for most changes we make). The core thing that makes this possible is that we run our entire code and infra (including fakes for external dependencies) in isolated, forkable environments and write tests against that, so they are as E2E as can possibly be.

The problem then shifts from reviewing code to reviewing tests and that's why we built our own platform. We have a UI that can diff tests, so we know what changed, and a visual way to inspect what the tests actually did. A test could drive a browser like a user would, and in our UI we get a replay of that browser interaction to look at. The browser is talking to a real version of our backend, and the tests can perform assertions against the database and fakes and really anything in our system.

That’s essentially what we do. The agent writes the browser tests (and the other tests), we just review the replays of them afterwards from the test suite.

Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.

We ended up building our own test platform to reduce the human bottleneck in the verification loop.

Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.

We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.

Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.

It’s worked out pretty well for the Norwegians!

My point is that it wouldn't be someone else's "clankers". The lab would be majority owned by the populace through their democratically elected government.

I agree, the datacenters and the GPUs inside them might be the least valuable thing to come out of the boom. The research in chips, architectures, and training, and the supporting infrastructure in fabs and energy grids still seems highly valuable. Europe will still see very little of that, but at least the Chinese labs still publish some of their research.

Oh definitely, not saying they are lying about working on a big model, but as you say I'm skeptical it will be any good.

I hope they will get that boost, but I'm afraid that they haven't built credibility so far to convince governments to buy in at the level needed. Their slip away from the frontier might even have made governments more cautious on investing in AI development, which is the opposite of what we need right now.

I considered mentioning that in the footnote but my own skepticism stopped me. I'm now thinking I need to see it to believe it in terms of Mistral training a frontier model.

That's not to say I don't hope they will. It would be incredible if they could do that despite their limited funding and the wide swath of projects they are working on. I do think that lack of focus is going to be a big problem.

Yes, it probably won't be clear until we have the benefit of some hindsight.

One thing I'm inclined to believe though is that the investment being made into compute and research today may not benefit the investors, but it will be net-positive on a longer time scale for the wider populace. A comparison to the internet and the dot-com bubble seems fitting.

And from that perspective I'd also be worried about the bubble bursting before Europe even gets started, as what remains after a bubble would still largely serve the US and China.

Microsoft has been doing this for a long time: https://learn.microsoft.com/en-us/dotnet/csharp/linq/

Shameless self plug but check out: https://specific.dev (especially if you use coding agents)

No code lock-in through SDKs and built on top of AWS with great DX for both developer and coding agents

Orbstack is definitely much better but far from native speeds in my experience. From our perspective of wanting all users to have a good experience, we also can't really point folks towards Orbstack as a "solution" to make the local dev experience great.

Same to you!

Yes, exactly. Probably two different focuses between us, we are more focused on providing the full environment to build productively with coding agents, from local dev all the way to prod. The key thing for us is that the agent can write code, build infrastructure and test the entire system autonomously locally, and then deploying to production should be dead simple.

A bit of a different approach from the classic use case of docker-compose that is often orthogonal to the production infrastructure in some sense.

One thing I've used to great success though is taking an existing project or example docker-compose and simply asking the coding agent to translate it to Specific's IaC. Works a treat, especially as the coding agent can read all the code at the same time and connect it all together.

(also it looks like we were in the same batch!)

We have been trying to solve the same problem (and a bunch of other ones) with https://specific.dev as well. We’ve tried to stay away from Docker as much as we can though because of the still pretty bad experience on Mac.

Our approach is having our CLI handle port assignments (and pass any connection details/ports along as env vars) and that way being able to spin up “isolated” copies of the local dev environment. Has the added benefit of us being able to deploy the same config straight to production and switch in production database connections strings and anything else needed.

Totally agree that AI coding makes this even more important. We are working on a coding agent-first cloud and a large part of that is ensuring everything runs locally so folks can let their coding agents define the infra and test it all

Better Auth is absolutely great. Was pleased with how well it could be extended here to replace Supabase Auth with backwards compatibility

Part of this is a reimplementation of PostgREST, but as a Typescript library. The big difference is that this is designed to help you migrate away from the auto-generated and to an API that is better designed for your actual use case. That’s why this is a library so you can integrate it as part of your backend and gradually replace the PostgREST parts

The storage part is really only for Supabase-js, yeah. The idea being that folks will eventually move to a model where they integrate directly through S3 (if they want)

Yes. We are taking a stab at the entire infrastructure like Heroku did but with a focus on a coding agent-centric workflow: https://specific.dev

We have been working on this, letting any coding agent define infrastructure so we can define it effortlessly: https://specific.dev. We aren't just targeting non-developers though, we think this is useful to anyone building primarily through coding agents.

You hit the nail on the head.

> But those things are exactly things that LLMs already solve, especially for vibecoders who don't even know what the code is doing. For them it'd be so much better if the LLM just went with the standard SQL setup rather than Supabase/RLS.

This is precisely the approach we are taking for Specific (https://specific.dev). We believe you can move just as fast as you can with Supabase (whether you're a developer or not) if you just let coding agents code. What they are missing is the infrastructure parts, which we provide, like a private Postgres database that a backend can be built around (and which can't just be publicly exposed).

With that, I was referring to the definition in the article: "The spec is the main source file over time, and only the spec is edited by the human, the human never touches the code". That's how Specific works.

And I think that opens up a very interesting question about quality. If the human never touches the code, then "good code" gets replaced with "good specs" instead, and I don't think anybody knows what constitutes good specs in that context right now!