224 karma · joined October 22, 2019
jonathan@molnett.com
https://twitter.com/jonathangrahl
At Lovable we decided to spend ~1 engineering month getting our sandbox infrastructure to be 100% runnable on Mac and on Linux with predictable builds using Bazel etc.
But we actively decided to take a different route than Encore and run nested virtualization, which on macOS means vfkit -> QEMU -> Kind + Firecracker vm(s). It's been invaluable to get the same development and testing tools on both platforms tbh.
I can't say how many times the LLM-proposed solution to a jittery behavior is adding retries. At this point we have to be even more careful with controlling the implementation of things in the hot path.
I have to say though, giving Amp/Claude Code the Grafana MCP + read-only kubectl has saved me days worth of debugging. So there's definitely trade-offs!
https://hn-frontpage-2035.lovable.app/
> The last JavaScript framework: Why we stopped at React 47 (overreacted.io)
Who knows where I would be today if not for Django!
Also: Please don't evolve the UI. Its perfect as it is
Going from Manifest to OCI is a bit tricky and performance for calculating total storage based on metadata is hard to get right. But the result is that we own our full registry implementation and can take it any direction we want. Quite happy with that!
I had massive issues at my previous employer with Bazel. They did not try to make Bazel work for non-SREs, which as you can imagine didn't work very well. So it's definitely not a silver bullet!
We should probably write a blog post about our setup!
We are not using a microservice pattern at all. I am not sure where you get that from. If anything we have several "macro services".
Our final setup is quite complex as we are building a literal cloud provider, but in practice we have a Go API, a Docker registry, a Temporal Worker and a Kubernetes controller. Whats complicated is everything else around it. We run our platform on bare-metal and thus have auxiliary services like a full-blown Kubernetes cluster, Ory Hydra + Kratos, SpiceDB, Cilium, Temporal Cluster + Workers and some other small things. We need to be able to test this locally to feel safe to release to production. And in turn our production environment is almost identical to our local environments.
None of that would be possible unless we've done something similar to what we have built today. Most companies cannot run their entire stack on their laptop, more unlikely that they could run a full cloud provider.
We can start the entire platform, Kubernetes operators and all, locally on our laptops using Tilt + Bazel + Kind. This works on both Mac and Linux. This means we can validate essentially all functionality, even our Bottlerocket-based OS with Firecracker, locally without requiring a personal development cluster or such.
We have made this tool layer which means if I run `go` or `kubectl` while in our repo, it's built and provided by Bazel itself. This means that all of us are always on the same version of tools, and we never have to maintain local installations.
It's been a HUGE blessing. It has taken some effort, will take continuous effort and to be fair it has been crucial to have an ex Google SRE on the team. I would never want to work in another way in the future.
EDIT: To clarify, our repo is essentially only Golang, Bash and Rust.
I really do hope that their OSS strategy does not change due to this, as it's really friendly to people who want to learn their product and run smaller deployments. It's (intentionally or not) really hard to run at a big scale as the control plane is not open-source, which makes the model actually work.
Hope to launch it soon!
This is exactly what I would love to build if I had the time <3
Our company's biggest remaining problem is finding a reliable IdP that isn't primarily for individuals, self-hosted or based on Active Directory. There are some alternatives, but they are mainly country-specific (like Freja) or not well integrated enough with e.g. OAuth. We've considered at some point to pivot our company to focus on a fully European IdP, as we feel it's a fundamental missing puzzle piece!
It's a decent service in itself and a good alternative to many other traditional VM-based hosting companies, which Europe is absolutely full of. They are not really competing with the hyperscalers, nor even Digital Ocean in my opinion, but rather providers that likely sell pre-committed OpenStack clusters and such.
I don't believe they are fully clear of American-owned companies in their entire dependency chain, nor do I believe that it's possible to do that today. Companies like Equinix provide a damn good solution for homogenized infrastructure that is invaluable for cloud providers.
I couldn't agree less with this. At this point the whole "DevOps" industry is fueled by consultancies who make a great living from convincing business leaders that this is true. Focusing on defining clear processes for recurring events and building the fundamental building blocks that allows you to automate when it's absolutely needed should be the method, not spending more time writing Terraform.
All of this makes sense considering the extremely low price.