It takes $420k per year to run Lichess
twitter.com
twitter.com
In particular:
How they remain free: https://www.reddit.com/r/chess/comments/mpasyl/i_started_lic...
On his salary: https://www.reddit.com/r/chess/comments/mpasyl/i_started_lic...
I just signed up for a monthly donation. I complain about the ad supported internet all the time, but had never donated to LiChess. This thread is a good reminder of how far dollars to support projects like this can go.
There's a good chance that many AWS slinging devs here have not actually had to deal with the volume of traffic that lichess sees. Half a mil yearly to run the entire lichess infrastructure, including dev salary and providing stockfish analysis for every user for free takes an insanely efficient setup that most big tech companies could only dream of.
Edit: As discussed in another subthread most of the analysis happens in the user's browser, the rest is run via a fishnet by volunteers.
I doubt you can't make Alexa top 10k with a million of daily unique visitors.
One stockfish computer analysis must cost more than merely playing 1,000 unanalyzed games.
Analyzing a chess game or two is not exactly easily verifiable, but it should be somehow possible to shoehorn that into a proof of work thing.
[1] 40 a day per https://lichess.org/features
https://lichess.org/blog/YOCx7hIAACUAgsUo/stockfish-14-has-a...
It's a great project, great quality and run at a low cost but it's all that because it's run by people who sacrifice their financial situation to make their idea happen.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
I understand the temptation to do this—we all feel it. That's why that guideline is in there. Such comments reliably lead to lower-quality threads, because they don't add anything that anyone can learn from. They just assert the commenter as somehow different from / better than others.
That's not to say the comments you were criticizing aren't bad. Of course, shallow dismissals are bad. But it only makes things worse to respond with a different sort of shallow dismissal.
Developer salaries in Europe are not the astronomic salaries you see in the US in the major cities. Probably the highest paying in Europe is London in the UK, and even then its still less than 50% of what developers in SF make at probably around 75k USD per year/6-7k USD per month in London.
Though rates probably depend on income so maybe the graph only gives a rough idea (e.g. top taxpayer rates probably higher in Europe and higher rates may kick in earlier in Europe.)
* https://ie.indeed.com/jobs?q=developer%20%E2%82%AC80%2C000&l...
* https://ie.indeed.com/jobs?q=developer%20%E2%82%AC80%2C000&l...
* https://ie.indeed.com/jobs?q=developer%20%E2%82%AC80%2C000&l...
Just the first three I found with a salary >= €80k attached to the job listing
And I'm not talking about having to hire in Romania or something (where you could save 4-8X or more). You could hire the top 5% of talent in expensive cities like London and still cut your payroll costs by 2-3X
Good wine is easy enough to find though, and the outdoor scenery is pretty.
I cannot wait until end of Covid and try and spend a large fraction of our weekends outside this place.
Agreed that restaurants and food are expensive. Selection is smaller than somewhere like London, but with a population 10x lower that's to be expected. Size-wise Zurich is more like Birmingham. If you take local cuisine differences into account I don't think it's that different.
Crime, taxes and pollution are also much lower. It really depends on your priorities.
Rental yields are capped in Switzerland, and in practice Zurich rents are not at the market clearing level - renting is actually a beauty contest between applicants. Great apartments are restricted by availability instead of rent level, but even considering that, the rent levels are higher than London, excluding central London - but I lived in London for 14 years, and I'd never really consider living centrally.
I'll reserve judgement on restaurants until post-Covid permits more eating out. I've ordered deliveries from about 20 different places (mostly Zurich, some Baden) and eaten out in three different places. I have a number of preliminary conclusions (Swiss don't know how to make a good burger, Indian restaurant portions are too large and too expensive, there are far too many pizza places and the highest rated ones have been very mediocre, and the most reliably good food has had a high cheese content, whether it's fondue or cordon bleu) but I'll pass deeper judgement later.
Good wine is easy to find though, and my house has a wine cellar.
It also sounds like you're still trying to eat the cuisine you were in London, and finding the (reliably) high quality food to be the local cuisine.
A few quick recommendations:
Pizza: Napulé (the sit down in Meilen, not the takeaway by Bellevue)
Indian: Kerala (real Indian food, not sweet British-style curry)
Hearty European: Wirtschaft zur Au
Eritrean: Mesob
American: Grain Bar & Restaurant
Turkish: Gül
European breakfast/brunch: Baboo's
Ice-cream: Gelati Tellhof
Enjoy!
Some FAANGs understand this and have decided to hire more in Europe.
Europe is not all one market price wise, with big differences internally. But you can certainly get a developer for €49k. You might have trouble retaining them, as based on salary surveys on the subreddit Irish developers it's like a 20th-30th percentile salary, and this developer has a track record of leading a successful customer facing web service, but not preposterous.
Non profit orgs receive services from the state and they have a cost to fund social security and such.
Other non profits which do not have a very nice situation can receive help from the state, so why not share the bits of success of Lichess to everyone else?
That being said, it's probably a lot cheaper than rewriting it to run on less.
I studied accounting at university and no one plans to do anything but auditing or consulting. I still love accounting, gives the most straight answer of how a business is run. Case in point.
Recent years, I have a feeling that HN commenters would assume server cost should always approach $0, not matter how big, or how different a real world online service is different from their single page web app side project :)
For any server cost sharing posts, we’ll see people making comments like “I can run this under $10/month” . Ha
I don't use Lichess. If I did, I would have donated. But after knowing that they throw money down the sink like this, I would be hesitant.
While I know many people are saying don't look at the cloud costs, when I look at those numbers it makes me think of CAPEX vs OPEX trades, and the OPEX here is non-trivial, but potentially a variable offset by some labor costs. I think some of this is being driven by a dated tech stack and technical debt.
Now, I am not one to trivialize the work that may need to go into getting this efficient, but I think you are right, we should consider investing into modernization of the stack.
Maybe you could rewrite it in assembler over the weekend and run it on a Raspberry Pi, right?
I worked on Java codebases that absolutely squeezed every bit of performance from a machine and that would be very hard to do in other languages.
For example, they are paying $266/month for 16 thread 128GB 3.8TB SSD, but you can rent a better machine for 99$/month from Hetzner (AX61-NVME).
They are also spending $40k/year on "data protection services" (?!?) and 60k for "site moderation" (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate).
I'm guessing this is paying an employee/consultant to ensure they are compliant with everyone's policies. At that scale you have to care about every jurisdiction and letting the one dev guess at compliance might not be ideal.
> 60k for "site moderation" (they could instead not moderate or rely on third-party forums and messaging services they don't need to moderate).
Can you even just not moderate? There's some stuff you legally have to due to say NetzDG for German users, and compliance with everyone else.
Have you ever used the site? How does that work by not moderating? When I report someone for being racist, spamming or submit a game where I think someone cheated... who handles that, what forum or open service doesn't have some type of moderation and in this case were you want a seamless experience with your app as they do with chat, blog, forum and multiple daily tournaments.
But yes generally speaking OVH is a rather strange breast. Much prefer Hetzner. A Zen 2 64 Thread, 192 GB Memory and 3.8TB SSD DB Primary listed as $464.00 in the spreadsheet only cost about $210 on Hetzner.
$143.14 x 1.2 = $172
Assuming it applies, some of those numbers more or less work out.
Hetzner has 20TB which can be easily done in a day. Can't find their pricing, but everyone usually charges $0.01/GB, which can add up quick.
https://docs.hetzner.com/robot/general/traffic/
Also for their shared servers or servers with 10GBs nics they charge $1/TB when metered that is $0.001/GB you are off by a order of magnitude
OVH also has way better anti DDoS. https://www.ovh.com/ca/en/anti-ddos/
Not sure whether they need AWS scale and couldn't cut costs down from 420k a year to something cheaper. Naively, this seems like something MS could sponsor by providing long-term discount.
That said, I have no idea how much cheaper this would be on other cloud providers, or whether they even have the need to cut down costs. For all I know, they might be making 10x of that in donations.
As for scale they do almost 100 million games per month and keep every game available for download and online access.
I suspect donations ~= costs, which is why the salaries are quite modest to say the least.
I am not saying any of this to be critical of Lichess. There are different ways to solve these problems, and their way is clearly working. This also happens slowly over many years, so it is hard/impossible to see the end state until you are there. The app is very quick and responsive. I got my ass handed to me on my first anon game :) My feedback is more for the community here in the context of using this as a byte-sized case study.
At the end of the day we are reading and writing 1's and 0's to a network device, or a disk. Have to imagine you can run and persist chess games with a lot less resources.
If you look at the details[0], servers are really "just" 6k$ per month. A lot of that goes towards the databases, which you can't optimize much with k8s, so the optimization potential overall is already rather limited. If you then need a consultant or a developer to do the migration, you'll quickly outspend any potential savings.
I'm a fan of k8s, but this is really a showcase for hardware being (comparatively) cheap and the scale that you can go to with conventional hardware.
[0] https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSk...
It’s just a different way to utilize resources. I don't see an issue with the costs - I see an issue with this massive list of inventory that needs to be carefully managed. Managing 30 boxes is not easy when they are all funny shapes and sizes. Are they all running the same OS? Same version? How do you roll out security updates? With kube, you just bring a new node online and then kill the old one. Done.
My argument still remains - what happens when a vital box dies? How quickly can a new one be brought online? Is that automated? What about in the meantime, is service degraded or completely down? What about when the ES boxes shit the bed and you realize you actually needed a cluster of 5 nodes? Do you buy 5 new boxes that are carefully sized, or do you just scale your mega cluster from 4 to 5 and add a few new pods?
Kube is not nearly as big and scary as this community makes it out to be.
Conventional setup might've been a better choice of words. That being said, in my experience, k8s in smaller setups tends to use more small machines, rather than (very) few big ones.
> It’s just a different way to utilize resources. I don't see an issue with the costs - [...]
Assuming the developer knows k8s and the app doesn't need much integration [0]. If one of those isn't met, you'll need to invest the time to rethink you app and infrastructure. And you don't really want to setup such a setup with k8s knowledge gathered from a few medium posts, so you do really need to hire an expert, plus plan some more hardware for your controllers - there's definitely a money issue.
> [...] I see an issue with this massive list of inventory that needs to be carefully managed. > Managing 30 boxes is not easy when they are all funny shapes and sizes. Are they all running the same OS? Same version? How do you roll out security updates? With kube, you just bring a new node online and then kill the old one. Done.
Going by the price, those are most likely dedicated boxes. So it's not like spinning down an EC2 machine and starting a new one (and not going for dedicated with these hw requirements would increase costs quite a bit); instead, you'd need to reimage. So, basically, you still have to manage those boxes - in addition to also updating your containers, now.
Or you simply enter `apt install unattended-upgrades` and move on with your live. Managing 30 hosts is really not that big of an issue nowadays.
> My argument still remains - what happens when a vital box dies? How quickly can a new one be brought online? Is that automated?
For the fronted, you could only really solve that by running a second box in the same size. And if you did that, you could automate that quite easily with something like a heartbeat - a failover can be handled without k8s. And do you really need all that uptime? This is a non-profit, not a startup; being down for an hour does not end your business.
> What about when the ES boxes shit the bed and you realize you actually needed a cluster of 5 nodes? Do you buy 5 new boxes that are carefully sized, or do you just scale your mega cluster from 4 to 5 and add a few new pods?
I give you the point, for ES it would really be useful. But horizontally scaling a cluster is really not that hard.
> Kube is not nearly as big and scary as this community makes it out to be.
No, it really isn't. But it does bring in a lot of complexity - how do you handle ingress? Storage? Audits? How do you scale? Does your app run in pods, can it easily fail-over? How do you handle shared state? You also need some additional hardware to handle controlling etc. and, as mentioned above, the knowledge to configure and run a cluster for such a high-profile site.
That being said, I agree with you that, seen from a clean-state, using k8s (or maybe just k3s) here would not be a wrong decision. For all we know, they might actually be using it for ES or some other parts. But we shouldn't disregard that k8s also brings in a lot of complexity, configuration overhead, costs and, next to that, a whole paradigm shift - your storage goes from permanent to ephemeral, unless explicitly marked otherwise, and accessing your app works completely different, just to name a few things. Learning k8s takes time and, while it is definitely worth it (IMO), you can still manage an infrastructure of that size with easier tools without any problems whatsoever. It's basically sure that the overhead in both learning and operations is simply not worth it and would cost much more than any savings k8s brings - which is exactly what my original response pointed out.
[0] The app might depend on specific OS features, use a lot of storage (in which case you will also need to setup a properly HA storage provider) or have some significant state, which would need to be synced to a new instance.
They are no where near the scale where Kubernetes makes sense as anything but a tools for making deployments easier. Ignoring the deployment part, I think many are pretty clueless about how big you truly need to benefit from Kubernetes and how much ekstra overhead you'll have in terms of managing clusters.
As I outlined in my other response, it does not make sense to make the switch in this case (IMO). But if I were to design an application at this scale and the team would have sufficient knowledge in Kubernetes, I'd definitely consider deploying to a cluster - the overhead could be quickly countered by development speed and fail safety, especially when choosing something simpler like k3s.
But I agree that it's definitely not needed here, that scale is a few orders of magnitude away.