Lovable incident report: Recovering from being taken down by GitHub
lovable.dev
lovable.dev
Sounds like you need to run your own git server.
“ 17:00: While rewriting the migration we find other issues since S3 allows multiple concurrent writes to the same project. This is a race condition that would have been prevented by Github since it handles out of sync commits thanks to it being an actual git repository. On Github, if a pushed head already exists as a commit it won’t overwrite a newer head, but there are no controls like this in S3. The impact mostly seems to be for new projects.”
That is a relatively well known failure pattern with git on s3 and it’s a shame the s3 representatives weren’t aware or there isn’t a knowledge base to tap into. I have found with AWS you have to get beyond the first level of support to have any chance of having a meaningful conversation.
When it comes to user data, you have to be extremely careful, so I was surprised on X how quickly they were moving things off of GitHub in a relatively celebratory manner. They had a ton of attention from this incident so I guess they wanted to recover as fast as possible versus being sure they couldn’t lose user data (and to be fair it seems like it was all recoverable). Still should be careful as their service becomes more popular and important.
Not part of the report was that getting intro to people on Github significantly helped accelerate things.
Any questions about Lovable infra for those curious generally – happy to answer.
(also, we're hiring senior platform engineers in Europe to build the first European AI outlier success story:)
Also chapeau for the quick turnaround, I logged into lovable and saw the incident and you didn't let GitHub slow you down :-)