RIP Flynn.io
github.com
github.com
We’re surprised and flattered at the interest and discussion.
For those wondering why we shut down here’s a brief summary:
We created Flynn to be Heroku that you could run on your own infrastructure (cloud or otherwise). We started Flynn as a crowdfunded project with support mainly from other startups, we imagined it would be non-commercial and community-driven.
After the prototype it became clear that we would need a full time team to develop and maintain the project. When we couldn’t find anyone who wanted to fund the project long term we applied to YC and then raised a seed round which let us build a 1.0.
Unfortunately we were never able to raise more VC so we spent several years trying to build a business that would allow us to keep developing the project.
We spent the last four years with a skeleton team and while we were able to build a business doing “ops as a service” for other startups we ultimately couldn’t generate enough revenue to cover both development/features and the 24/7 on call team we needed. While we were able to support our paying customers we couldn’t deliver the features we wanted to build or support unpaid open source users.
We also take stability and support really seriously. We were extremely uncomfortable with the fact that people were running Flynn but didn’t know enough to run it safely. So we’d get panicked emails and github issues from users who couldn’t pay us but needed help. That’s a terrible situation for everyone involved.
The ecosystem has come a long way since we started (we were one of the first Docker powered projects and started about a year before kubernetes was announced). But we still feel like there’s a need for something like Flynn :(
Unfortunately it’s expensive to develop full-featured platforms and we could never get VCs interested in a Series A.
The 24/7 on call lifestyle got to be too much for our team, especially during COVID so with no prospects of things changing we decided to call it quits.
(For those who don’t know, all the partners in a YC interview ask questions of everyone simultaneously— this is complicated at the best of times and totally nuts when you’re trying to answer technical questions about distributed systems and questions about how you’ll build the business, but it’s a great mental workout!)
Think speed dating plus lots of coffee.
We're back in NZ and focusing solely on the education market which really accelerated our growth. Not sure when we'll be able to travel to the US again but would be great to catch up when that happens!
If someone came to you right now and offered an amount to keep going, no strings attached, enough to support the applications which remain hosted on your platform after those who have already lost their confidence leave, would you consider keeping it running?
What amount, in whatever currency you prefer, would make you consider it?
I've no horse in this race, but I'd like to get at least one datapoint.
This is not the first time I see a post like this, and it always makes me wonder if you could've gotten enough funds if youd gone out and said, "Look, Flynn is going to shut down if we don't get X amount minimum to support us. If this platform means anything to you, now is your chance to save it."
1. Having already shut down and the team gone separate ways and onto new projects how much would you need to get back on the horse so to speak
2. Are there still things you want to accomplish and what would it take to make them succeed
Personally I’d happily take a check to do more development. There are a lot of Flynn 2.0 features that were in various stages of completion, if someone was willing to fund development to complete them up front that’s great.
However I wouldn’t want to do sales for this/run a b2b company around this again.
So if one of the companies that was using Flynn called and said they wanted to pay our development team to keep building, that’s great. But if a VC called and said “wait! Let’s finally make this company take off” there would have to be a decent sized bonus check attached. Mostly that’s just because after years on a ramen profitable startup as a founder your finances aren’t awesome so saying “yes” means turning down higher paying offers.
The biggest problem we had financially was being able to develop features before our customers needed them. We knew what to build but didn’t have enough budget to ship them. Any offer would have to come with the guarantee that we could focus on development for day 6 months before starting sales back up.
At the end of the day there’s work we left unfinished and if someone was willing to fund that in advance I’d generally be up for it. But if someone just wanted to keep the business wheels turning that’s not super attractive at this point, so it would need to be a https://levels.fyi salary rather than a startup founder salary.
You've gifted a lot of insight.
If you don't mind me asking, what are some features you think are most desirable for you to develop, and what would be your time budget for developing them, if it is more than 6 months?
And which title from levels.fyi do you think is most applicable to the person or people who would be working on that feature?
- Database appliances (RDS) for major DBs
- turnkey security, especially for compliance (SOC2, HIPAA, ISO 27001)
- close the loop on development environments (vscode)
- CICD
That’s pretty close to our series A roadmap. Mythical man months aside, I’d say all of that could be production-ready in around 18 months with a few engineers per bullet point.
In terms of salaries it’s a little tricky for a number of reasons (people are willing to work for less for a startup and/or on open source plus we hired around the world), but for many of the developers I’d say equivalent to L4-5.
The important thing to remember about PaaS is multi-node vs single node are different worlds. There are great tools like dokku that were designed only for single node operation, then there are things like Flynn or k8 for many nodes. As soon as you’re doing many nodes you’re in distributed systems territory where engineers have to manage much more complete systems design and as a result are more expensive and in more demand. That’s not saying anything bad about simpler tools, they’re great and important, it’s just a lot harder to design around distributed systems where failure modes, consensus, partitions, etc are a lot more complicated. A lot of the challenges when we started were figuring out what the problems were and how to solve them rather than implementation time. In the last 5 years there has been a lot more written so less research would be needed to implement some of these today.
Our overall goal was to automate anything in the devops lifecycle that could be automated, so that’s a never ending process. Unfortunately it means that the more you scale the more technology you have to make, so Flynn for smaller companies with fewer users is less expensive to design and build than Flynn for huge companies with lots of traffic. (Again, distributed systems are hard) so knowing the scale of your prospective users is a big part of the cost equation.
At what you described, for 6 months, that's about 200K USD per dev, so it would cost about 600K USD to develop the database appliances "superfeature"?
Do you think this is a realistic figure? And how realistic do you think it is to get that much money together from all the users?
$180k base salary is appropriate but most of those people are expecting stock options as well (compare to Levels) so the fully loaded price* is closer to 300-500k/year x 3 people, so $1-1.5mm
We’ve already done most of the conceptual and architectural work for Postgres, MySQL, and mongo (and redis but not highly available). Much of that was inspired by the Manatee project at Joyent. So (though I haven’t looked at this I a while and the DBs may have changed in a way that becomes more challenging) this is probably a case of put money in, get value out.
However I can assure you you aren’t going to get even 600k out of the community.
We worked really hard to get the first 120k and twisted a lot of wrists to get there. (Of course if you know someone who wants to write 600k checks for open source stuff please have them call me)
The 1-1.5mm number is about what a standard YC company gets at demo day. So that’s the path I’d recommend. Add in another 500k for overhead and biz dev people and that’s a pretty solid startup. Honestly if we had just focused on that we would probably have got a series A.
One of the great things about open source is that you create 100x+ the amount of value that you capture. The dark side is that no one wants to contribute cash as users to make that happen (I’m thinking more of companies than individuals - lots of individuals will give a few dollars)
So, unfortunately, and as always, it’s easier to raise VC than sponsorships for open source.
That being said I strongly encourage anyone who’s interested to pick up where we left off. Either as a community project or a funded startup this would be a huge benefit to the ecosystem.
* of course you can find people who are cheaper and passionate about the project to do it for less, maybe much less. But if someone called me and said for what price can you guarantee this could be done, I’d say 1-1.5mm USD.
Thank you very much once again for responding with this level of detail.
When you got the first 120k, who paid that? One entity or multiple?
Our strategy was to encourage developers to talk to their bosses rather than contribute individually which we think but can’t prove worked well. Basically we tried to get on calls with CTOs and CEOs and explain that this would help the company.
Did the money come from the companies' tech budgets?
Was it a one lump sum pre-payment or periodic payments? Were there any strings like deliverables attached?
Hope this is not too much to ask, I'm really interested in the process.
I am developing something almost completely the opposite -- a website platform designed only for small-scale deployments by individuals and small communities, with purposeful mechanisms to limit/control growth, and for now deliberately without any finances, fregan.
Recently I have been coming to terms with the idea that I may get a lot further if I attract outside help, but currently I don't have much to pay with, and I've not made it easy for someone to just jump in and start contributing.
I am learning a lot from your answers.
In our case the funding came prepaid with no strings attached. Can’t speak to where in their budgets the funds were allocated from.
I think we should build an ecosystem where companies support open source projects that benefit their companies with great ROI. Unfortunately I don’t know how we get there yet.
There are so many different funding models now for different things (https://humanipo.app/ for example) that there must be a good answer. We should all work together and try harder to find what the best option is.
The numbers I'm seeing on HumanIPO are not encouraging :)
3-7 years is about the maximum I would expect an unsustainably-structured support organization to last before people burn out, so you had a good run.
I've been using Dokku [1] for a few years on a small setup, surprisingly without a single problem, taking into account it was written in "not-so-cool" bash. And I was considering Flynn as the next step if I need to scale it because Dokku doesn't have clustering support (added: looks like clustering support for Dokku is in work [4]).
After many checks, I got the impression Flynn simply wasn't there yet. Either because of low development pace, low number of supported appliances, or something else, I'm not sure. In the end, I picked up Ansible for more distributed installations.
[4] https://www.reddit.com/r/devops/comments/bgpw5w/flynn_vs_dok...
My biggest complaint would be the non-zero downside deployment.
For a company that is a single eng team, it's obvious. But progressively larger organizations, it seems like a gray zone.
Even so, K8 is a solution where running containers on a container as a service system is prohibitively expensive. The newfangled systems aren’t free in terms of operating costs and would you rather pay for extra hardware or for engineering time, knowing that hardware doesn’t take vacations or leave your company for a better job?
It is dependent on size. You could for example start with an engineering team working on a mono stack and outsource data and analytics operations. But at some point, you will decide that data and analytics have to be in-house. You start with a mono stack for each of them. But then you suddenly start splitting each of these three into sub-teams. Suddenly mono stack doesn't make all that sense.
K8s solves the problem of horizontal scaling in both teams and infrastructure. It's inevitable when (and only if) you plan on scaling up. Not that all teams/companies will/want to follow that path.
If you have a team of 50 engineers but 40 of them work on the front end of your SPA, and the backend is simple CRUD, why do you need your own container infrastructure?
I stand by that the decision should be primarily based on how much of your team’s effort is diverted to devops. K8s is a devops solution, not a way to organize code or a development framework.
None of our teams have dedicated devops engineers. Teams just have engineers that do everything: back end, front end, deployments, monitoring dashboards. Teams write run books for SRE. Dev Ops supplies eng teams tooling for deployments.
I've argued many, many times we spread our engineers too thin and we need greater specialization for better outcomes. We have enough engineers, but the problem is everyone owns their own thin but very tall vertical slice of the total system.
I've seen k8s managed by a couple of DevOps (among the rest of the infrastructure) and used in a company with 3-5 teams (and 200 microservices, but not that much code, they just drank the microservices kool aid). That was a migration from running containers on EC2 (semi automated orchestration) to running and maintaining k8s on EC2.
Best infra experience I've seen in any company I worked at (and the company wasn't that great overall).
I also ran some smaller scale K8s by myself while doing eng management work, so I disagree K8s is as hard as people make it to be
Honestly baffled at how many people are convinced you need hundreds of engineers before k8s can work well - they must be doing something very different from what we are. We did keep databases out of k8s (until recently when we added CockroachDB running inside the cluster) and only had ~6 services to move which may have kept things simpler. We're now scaling to run thousands of vcpu at peak times and a dozen different services, and it's still not all that hard to manage.
Can somebody who has had the opposite experience comment on what actually made it so difficult to implement? I imagine that if you are managing the cluster control plane yourself that will make things much more difficult - but unless you have some very specific requirement you can use a hosted k8s to reduce that.
Now if you are in Europe then Scaleway got a K8s offering not that long ago, but the rest of the world have to run that stuff local and running K8s securely on a locked down corporate network can be very complicated.
Also, I think a lot of companies have really terrible devops practices that don't work in Kubernetes. So their move to Kubernetes includes a lot of extra work that isn't really caused by the move to Kubernetes.
I'm not sure I get this. Are you saying that it obviously does or doesn't make sense?
My team is three people and k8s makes our lives easier than any alternatives I'm aware of. We used to be on Heroku, which is cool, until you need to do run anything other than a monolith or more secure than all publicly-accessible services.
The advice I give everyone is: Stay off k8s until you care about binpacking. That is, making sure you're fully utilizing the instances you pay for. When the cost of your architecture is taking up some brain cycles, start digging in.
If that's low down on your priority list, it's not worth the investment. If you're reasonable considered "a startup", invest your time/money elsewhere. PMF and getting to default alive is far more important.
If you need automated deployments, centralized logging, autoscaling, etc which many teams do, then you're going to be dealing with a bunch of complexity anyway.
There will come a time when cost of paying the overhead for a devops person (and eventually) team is worth it. At that point, k8s can be a great fit.
In my experience, that tends to be when you're at scale enough to care about costs a lot. Total spend and/or reducing COGS make it worth while. But when you look at it from the time an engineer costs, it's easier to see.
Are you gonna save 200k/year (minimum) in costs moving to k8s? Then do it. If you don't have line of sight to that, pay AWS/GCP to manage that for you, and focus on your business.
Also note, there's stages even with running k8s. Don't go all in running it all.
Start with a container, run it serverless. When k8s becomes a better fit (to reduce costs, or with other small exceptions), use EKS or GKE. Don't run your own control plane.
If you really have a need for a lot of custom stuff, then start to run your own control plane. But by this team, you probably have a team managing all this. If that cost (remembering how expensive engineers are) is shocking, you should be running a different solution.
So the only reason I might want something like Flynn is in the case of unexpected uptick in load. And unfortunately, in my first experience with Flynn, things went bad and I wasn't able to restore from backups, which scared me off more than I was scared about load balancing (read: ran out of time for the experiments and vowed "one day I'll try again. Maybe."). That said, overall flynn did seem reasonably polished, so I'm sad about the announcement.
Maybe, some day, someone will take on this middle ground between Dokku and Kubernetes. But until then, Dokku has definitely proved itself.
Isn't Dokku alive and well? I don't use it but I read about it in some related research recently, and some people on this thread report to be happy users.
It's not like you were gonna deploy Dokku at $BigCorp, no Kubernetes is a huge selling point for anything I'm gonna use in my free time.
We're still chugging along, and have recently added Cloud Native Buildpacks support[1], as well as integrations with Kubernetes[2] and Nomad[3]. Happy to hear where you found out that development on the project was halted, given that I made a release yesterday[4]...
[1]: https://dokku.com/docs/deployment/methods/cloud-native-buildpacks/
[2]: https://github.com/dokku/dokku-scheduler-kubernetes
[3]: https://github.com/dokku/dokku-scheduler-nomad
[4]: https://github.com/dokku/dokku/releases/tag/v0.23.9AWS ECS is alive and kicking!
My minimal self hates k8s but my get-stuff-done self uses k8s.
(Not that the page had changed in any meaningful way in pretty much forever)
In some way an alternative for using Kubernetes and managed services of cloud providers.
I think there is a need for something that is easier to use than Kubernetes. I’ve started the 5 minute production app for this. The idea is to use managed services and Terraform to make stateful services easy. This way you’re not locked in but have flexibility going forward.
In simple terms it was a self hosted Heroku. It got massive attention here and raised a load of cash before kubernetes etc were a thing
I just wanted to say that while we’re sad to shut down and definitely didn’t accomplish everything we wanted to we are deeply grateful for having had the chance to build something significant and get past a 1.0.
Our original crowdfunding round and supporters - almost all of whom found us on HN, Y Combinator, and our seed investors helped make that possible (plus of course all our open source contributors, alpha and beta users, early customers, and even the people who filed GitHub issues panicked because their sites had just gone down).
Most of our team worked together on Tent (protocol for decentralized social networking) before this which no VC would get behind. So we know how much it sucks to have a great idea that no one will pay for you to build.
We’re very grateful to HN for the opportunity to build something so cool.
Neither of those projects would have been possible without HN. I know for a fact that we only got meetings, term sheets, into YC, and more on the strength of the HN comments and later the GitHub stars.
Before we had a working product we could at least say, “hey, these people all say they want what we’re trying to build” and that got us in the door.
We had lots of different proof points to show investors and even customers, but the most important thing was the number of upvotes and enthusiasm of the comments.
Thank you all for being so open minded and supporting our work. We literally wouldn’t have been allowed to do it without you.
Hope to have something even better to share with you all one day.
I'm very sad to see it go of course but the writing was on the wall for a long time after the demise of all non-k8s container management systems.
This is the ultimate fate of tools like Nomad and services like ECS. k8s changed the game by providing a target API/platform that gained enough traction to force each of the large providers to support a managed implementation of it.
I miss my time working on Flynn, we were a very small team but we wrote some good software and I learnt a ton in the process.
What AWS has to be careful with here is that when people make the move to k8s they might depart AWS altogether and go to GCP/Azure as you are throwing away a lot of their lock-in in the process.
This would be less of a problem if EKS was better so they probably need to be spending more time there than ECS, that ship has sailed.
ECS Fargate makes it easy to deploy containers easily.
Note that I don't work for AWS, but do use AWS, Azure and GCP cloud services, I don't have any vested interest in ECS. I do believe that ECS has a long life in its future.
Dokku has been rock solid for almost 5 years now for my business. Would love to have multiple nodes but otherwise fantastic and easy. I had to move off Heroku as my customers require an Australian based host and heroku doesn’t support the region. Migrating was a breeze.
I wanted to love Flynn and checked back in it every year or so but the blogs stopped a long while back and the brief documentation never really fleshed out.
If you can use Heroku, I would encourage you to do so. While the price seems high, the value add you get from just its feature-set and the ecosystem is tremendous. Seriously, even if you just pay 7 bucks for an always-on dyno, you can get pretty damn far without needing to worry about a server. I've been on a few platform teams, and we've always been _miles_ away from the functionality Heroku provides.
That aside, there are a number of reasons something like Dokku (or insert your favorite self-managed platform here) may be a good idea for your organization:
- You need to own your availability: With Heroku, you're at the mercy of Heroku, Salesforce, and Amazon (roughly in that order). Sometimes there is a mandate that your org cannot use services provided by certain companies, or you need an SLA that you can control for a third party. While I don't necessarily agree with those reasons, there are plenty of cases where owning your availability is more important than saving on the cost of building and running the platform.
- You cannot use an external vendor: Sometimes you want something to run in your closet for the 40 self-hosted apps you run, and you cannot install Heroku in your closet. Maybe you work at a non-profit and you cannot easily justify the extra expense (either because of paperwork or because money is tight). Running your own platform is quite enticing in these sorts of cases.
- Running "at scale" is expensive: If you work at a consulting firm and are working on a couple dozen apps for customers, something like Heroku can start adding up, especially if you want staging environments for functionality. We have a ton of users that switch to Dokku for staging, and then end up hosting in other places for HA functionality they don't want to maintain.
- While we might support Kubernetes and Nomad, the complexity of maintaining these systems just for staging can seem silly.
- The platform does not support the functionality you need: While Heroku is _excellent_, it doesn' support everything, and it doesn't need to in order to service it's customers. When you find yourself reaching outside the box, that is when owning your platform becomes important.
i.e pretty much the same as everything else that has went the way of the dodo in this space.
Even if outside people are willing to contribute, there is usually no one with maintainer / merge powers who is willing or permitted to step up to handle contributions.
Besides, stars may not be the right metric unlike community participation [1]. Could Flynn have picked up community participation had it been donated to a Foundation like, say, the Cloud Native Foundation?
Another reason could be that because momentum is everything when faced with stiff competition, things may have slowed down for Flynn.
Some of the things that slow projects down:
1. Overwhelming feature gap.
2. Spiralling technical debt.
3. Rise in dramatic hard-to-fix bugs.
4. Inadequate staffing / inability to keep up with the changing ecosystem.
Also, looks like the lead developer (@titanous) has deleted their blog and tweets too, and so, it is likely we will never know what went on. Hopefully, they consider publishing their thoughts so that we may all learn and improve.
I wonder if Pieter Hintjens (of the ZeroMQ fame) had the right idea that a software project should always attempt to reach "completion" and stay "completed". That is, you complete one key part after the other and build on top of that solid foundation that doesn't change...? This approach may help keep things in control for a small team of core developers to tackle issues arising from taking up as complex a task as Flynn's.
[0] https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...
[1] https://medium.com/runacapital/open-source-growth-benchmarks...
As others have mentioned, the current site (README.md) says very little about what Flynn is.
You can see a version of the repo and README.md, just before it became unmaintained, here:
https://github.com/flynn/flynn/tree/2c20757de8b32a40ba06f7e5...
IMHO it would have been much better if the maintainers had added the "Flynn is Unmaintained" information to the top of the README.md rather than removing all the existing information. I've submitted a GitHub issue suggesting it: https://github.com/flynn/flynn/issues/4623
A plea to project maintainers in general: Please include enough information in your top-level README (or README.md, or README.whatever) so that someone who has never heard of your project can find get a good overview. I've also seen READMEs that only discuss the most recent changes. That's fine for readers who have already been following the project, but not for a more general audience.
I use https://fly.io now for this purpose. It's not self-hosted, but it does the job for me :)
That looks pretty cool, thanks.
Disclosure: I work for appfleet
I'm happy they marked is as unmaintained because I got the impression it wasn't all that well maintained for last few years - can't remember why anymore, but something to do with having to use master branch otherwise the package was 3 years old in ubuntu main repository...
I guess we'll be moving staging to k8s without hesitation now.
It was decent and served the purpose for our staging environment. Rip.
Feel free to hit up the `@dokku` twitter account, catch us on the gliderlabs slack (https://glider-slackin.herokuapp.com), or even provide feedback in the Discussions section of the primary Github repository (https://github.com/dokku/dokku/discussions).
(I am the dokku maintainer).
Would love to pick your brain as to what our quirks are and how we might avoid them in the future. Feel free to hit up the `@dokku` twitter account, catch us on the gliderlabs slack (https://glider-slackin.herokuapp.com), or even provide feedback in the Discussions section of the primary Github repository (https://github.com/dokku/dokku/discussions).
The installer is currently broken, I simply don't have enough time / bandwidth right now to work on it unfortunately.
That said, if you want to contribute, please let me know! I hope to get things running again soon.
https://swarmlet.dev or https://github.com/swarmlet/swarmlet
- Heroku. Sure it’s a bit expensive but it’s still super easy if you’re a dev. - OpenShift. If you’re a really big enterprise, OpenShift is a reasonable choice for PAAS. But only if you’re huge. - Kubernetes. Yes, it’s complicated. Yes, it has a steep learning curve. But it’s open source, has a huge and growing ecosystem, and it has less lock in than any other PAAS-like thing that I can think of.
The main downside of Kubernetes beyond its complexity is that you still have to build abstractions on top of it for your developers. But that world is improving regularly.
Watching and waiting for https://render.com to mature a little as it seems slightly better value.
So what's really missing is that someone would build a self-hosted RDS. A complete solution with a good backup concept (e.g. wall-g for postgresql), ability to restore from backups and in later versions maybe the ability to setup read replicas and do manual failovers for high availability (because automatic is still not easy).
So if there would be such a solution every PaaS could just say use "awesome free rds" for database storage and they have to implement a lot less functionality. And even if you don't want to use a PaaS and just install everything directly on a server, a DBaaS solution is still needed by a lot of folks.
Guess it’d have to have a different name though.
Of all the open source PaaS that mimic Heroku to a certain degree, I stick to Dokku because I can always fall back to Heroku with minimal app rewrite since I use buildpacks. This might not be an issue if you deploy with DockerFiles though.
If Flynn was still active, what would be the main reasons you'd chose it over the "cf push" experience?
Disclaimer: I have not used Flynn myself before stumbling over this thread, but I'm very interested in understanding the above given the ongoing discussions now of the future of CF.
We had the pleasure of getting on a call with the Flynn founders last week and learned so much from their experience. It was incredible to hear the first-hand account of the past 7 years building Flynn (seriously, what a ride). We took their lessons and advice to heart and hope to fill the void Flynn is leaving behind.
To the founders of Flynn and all contributors, thanks so much for building such an awesome project and paving the way. Porter is still in its early stage, and we're super excited to start sharing our progress with the community. Exciting stuff coming your way soon - stay tuned!