Comment by bsenftner | Hacker News Reader

Comment by bsenftner | Hacker News Reader

Parent Full thread

I'm 15 years in writing high performance Internet servers in C++, and I can confirm higher level languages provide an illusion of capability, but once you're talking high performance with high compute requirements and scaling your service, the cost efficiency of C++ is exponential better than any other language. The higher level language ecosystems are bloated beyond repair. I was able to use one 32-core physical server running a C++ http server I wrote providing a rich media web service and replace an AWS server stack that cost my client 120K per month. The client purchased one $8K 32-core server and co-located it behind a firewall at a cost of $125 per month. And the C++ server ran at 30% utilization, plenty of room for user growth. Their AWS stack of a dozen C#, Python, PHP and Node apps were peaking their capacity too. If course, my solution caused existential questioning by the non-geek CEO and the CTO, but they were in crisis and needed to radically revise how they provided their service or close.

The thing that worries me about stories like this is that there is frequently (as is the case here) no mention any sort of HA or backups. No details on what disaster recovery looks like. Those are business critical considerations that cost money, and just disappear from the discussion when people say "hey i saved all this money dropping everything down to a single server!"

Well, in my case described above, single server solutions include an automated backup sub-system, and my servers expect multiple instances of itself to running on the client network, and these multiple instances synchronize with one another via additional endpoints specific for the purpose. The whole issue of HA and backups is critical and one of the areas my approach shines.

You're still not actually answering the question of how you are HA and backing things up with only a single physical server and nothing else.

If you're backing things up to the same server, that's not enough. If the HA instances are running in the same server, that's not enough.

If there are other things besides that one physical server and it's power/network, you didn't include them in the cost, so the comparison is disingenuous there.

I do say the deployed system ends up being multiple instances of my single server, which synchronize with one another. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances. My deployed systems are not one server, they become a server mesh.

>. Those are separate physical devices each running my one server. Additionally, when a backup runs the data is stored locally as well as on a physical storage device separate from the hardware it is running. Typically clients already have a firewall/router which is used to distribute requests to the various instances.

Awesome! Really glad to hear this is the case. But those are all added costs beyond the single physical server you gave the price of.

Server cost * number of physical servers you have deployed Cost of your off-server storage Cost of your network appliance doing load balancing

It's still probably way less than the AWS bill, but it's not really fair to compare the total price of infrastructure in one environment vs. just a portion of the other.

I am in the same boat (writing native servers). I will also "disappear" if you start asking me about HA/backups/etc. A particular solutions are very much case specific and can depend on business conducting rules just as much as on pure tech factors. Properly answering your question requires way too much writing and hardly a subject of a single post. I have HA solutions for the products I built but this post is the extent I am willing to talk about the subject ;)

I'm not asking about the specifics, and I don't really care about them.

But the fact of the matter is quite simply is that any single physical server solution will never be satisfactory for backups or HA purposes.

You can't store your backups in the same place as your data and call it good - what do you do when you have multiple disks fail and your RAID can't be rebuilt? This happens. What do you do when operator error accidentally destroys the array? This happens. What do you do when there's a datacenter fire and the server burns up? This happens. None of these things should be a business ending event, but if you only have a single server handling quintuple duty, that's what it has a real chance of being.

If you need HA, a single server isn't good enough even if you've got multiple VMs running the service. What do you do when the utility power is out and the generator fails to kick on properly or they run out of fuel? Both of those happen pretty frequently. What do you do when there's a network outage at the DC? This happens. When someone fucks up BGP somewhere and now the prefix you used is being routed to god knows where? This happens. When you have any sort of physical server failure that brings your single box down? This happens. Any of those situations will take you offline and render your HA meaningless.

I'm not saying that any of these things are impossible to do when colocating your hardware - but they're not free, and they're not mentioned even at a high level in this story. And since we're not talking about running benchmarks on price/performance and instead talking about a service that a business needs to keep available to their customers to make money, these are important aspects to talk about. Everywhere I've worked, keeping our services available and being able to recover from hardware failure are far more important priorities than being able to optimize performance. And doing those things properly takes more than one physical server.

I can't speak for FpUser, but you misunderstand the idea of creating a single server: one does not just run one of them, they know about and expect multiple copies of themselves to be running at different IP addresses, and they synchronize with one another, as well as maintain individual backups that additional background processes validate between different instances.

Each and every one of your disaster scenarios is handled by the architecture. Each an every one of your disaster scenarios has happened and we've lived through them, as well as after the fact reviewed and optimized how we handled the events. As you are, we're professionals.

Making my Node.JS app run on cheap providers raised performance and reduced cost several times over, such as OVH.

This sounds like an amazing war story that I just want to hear more of. Is there any more? What's your c++ stack like?

Currently using Restbed (https://github.com/Corvusoft/restbed) as the server core, wxWidgets as a server side gui, with Boost, Curl, SQLite and Standard Lib. It's not that complex, beyond using lambdas in a few places. It has extremely high performance, and can run on an Intel Compute Stick, but I tend to use an Intel Nuc at minimum, with clients typically using whatever they have, gaining over redundancy and an ability to pair down. The memory management in a C++ application is just another resource one manages with whatever level of algorithm support you feel comfortable. There are ref-counting systems and complete garbage collectors available one can integrate into their business logic, unlike in a high level language that "transparently" manages memory outside application control. Have you ever thought about how much processing a typical 3D video game performs every frame? What if that caliber of optimized algorithm logic were handling a rich media, non-3D game server hosted business application? It would have pretty amazing performance and scale very economically. That's what I do. Before doing this, I wrote 3D video games and their production environments.

I love hearing optimization stories. can you tell us more about algorithm optimizations you've done in both business apps and video games?

AWS is expensive. It costs 20 times more for CPU.

Reply on news.ycombinator.com