Hacker News Reader

Ignoring the fact that this sounds like an ad for Hetzner, I'm not sure this is good _generic_ advice. It may be good for _some_, but the vast majority of single SaaS founders have access to platforms now (mostly via big IaaS providers) that allow them to build, develop, and deploy without ever worrying about RAID, Apache servers or Postgresql restarts.

> Plus they'll sometimes force reboot or force migrate ( =pause) your instances.

Extremely rare, but probably happens at a similar rate as your "single box" dedicated provider losing an HDD or having a datacenter blip.

My point here is that what you described sounds like the kinds of things SaaS developers needed to worry about ~10 years ago. The platforms of today aren't perfect, but they abstract away 90% of that and allow you to focus on business logic, which is exactly what a single SaaS developer should be doing.

The topic is reliability, not ease of use. What was described is not a big deal to pull off.

If Hetzner fits your use case (now and future case) then it's a great way to go.

As someone else who has been happily running a small business on dedicated hardware with a managed hosting service for years, I share some of the GP's scepticism about modern cloud hosting.

Modern platforms should remove most of the complexity of operating routine infrastructure and allow you to focus on your business logic, but it doesn't always work out that way.

Just the fact that your VMs have a significant chance of being forcibly shut down with little notice is a significant downside, for example. As a solo operator, you now have to arrange all the automatic scaling and failover configuration on your cloud host as well (possibly at considerable extra cost for capacity you might not be using 99% of the time) and you have all the 24/7/365 monitoring problems that OP was asking about.

Cloud services are also notorious for obfuscating their pricing so it's hard to work out the TCO. In my experience, arguments that cloud hosting works out much cheaper overall tend to be based on rather optimistic assumptions. It might be true if you lease some VMs at carefully chosen sizes and then set everything up yourself including scaling things down again any time you don't need them. However, once you start using the automatic services that actually do something for you beyond supplying a machine on demand, the prices might jump 3x or more (sometimes much more, like orders of magnitude) compared to ordering the equivalent basic resources and setting the same functionality up manually.

Then there are the security and compatibility updates. The basic cloud services tend to be provided as-is and it's up to you to ensure everything gets updated when it needs to be. Or again, you might be able to get a more automated service that does some of this for you, but it will come with a pricing premium.

Meanwhile, a solo operator using a more traditional managed service probably doesn't have to worry much about any of this, because those services will often be happy to take care of things like setting up your redundant database servers or monitoring the security mailing lists and applying emergency patches very quickly so you don't have to. That's the level of individual service and advice they tend to offer to distinguish themselves from the generic cloud hosting services. Obviously you do pay extra for that management service compared to just a basic hosting arrangement, but whether you pay more than you'd have paid trying to do all of it yourself on AWS or Azure or even DO is another question.

In theory, I agree with you that cloud providers should be more comfortable and more resilient. It's just that my practical experience has been the opposite.

BTW, I don't get commission, payments or anything from Hetzner. I'm just super enthusiastic about them because their affordable pricing is making me rich.

I agree that it's a tradeoff, but the question was about small companies. And there I'd say 5min of occasional downtime are absolutely fine if it saves you $100k annually. And for a single founder, those 100k in profit will be kind of a big deal ;)

I agree with most of the points here, but it's interesting that you mention:

> In my experience, good dedicated servers practically never crash

One of my toy servers (ecc ram/xeon cpu - but bought "second hand" via hetzner's auction) disappeared the other day. I thought maybe a disk had failed - but I couldn't bring it up in their network booted rescue mode - and requested a "hands on" power cycle - and after a few minutes the server was up again:

> Dear Client.

> A fault in your neighbor servers PSU tripped the fuse of the small rack segment which your server is located in too. We have fixed the issue and now your OS is back online.

Now, I think that box had a 700-900 days up-time before - I didn't really have to do anything (or pay) to get it back up.

But it was kind of surprising.

I guess all I'm saying is that I do like cheap, dedicated servers from hetzner - but if you need to guarantee five nines uptime, the architecture part is important.

I guess all I'm saying is that I do like cheap, dedicated servers from hetzner - but if you need to guarantee five nines uptime, the architecture part is important.

Five-nines is less than 10 minutes of downtime per year. I doubt anyone is really guaranteeing that without 24/7 active monitoring and maintaining extensive automated failover systems, which is already several full-time jobs. No solo operator is credibly providing that level of service.

Agreed. I doubt most people/services should build to 'guarantee' even .9999.

.9990 or .9995 is much cheaper, much easier, and probably closer to what your end user's network connectivity is anyway. (Yes, they're multiplicative, but if your user is connecting from a single-path residential connection and a $50 router, your 5th nine isn't needed to demolish their local 5-10 hours of downtime per year.)

That argument works pretty well in b2c, but in b2b your customers often insist on high uptimes, even if they benefit little from them.

Though the promised uptime might not matter that much in practice, since the penalties for a couple of hours of downtime are often affordable.

That sounds like an enterprise feature with "call us" pricing to me. :-)

I’m running five-nines with my setup and I’m the only operator. Monitoring and automatic failover is not difficult but I think it requires a solid architecture from the ground up. When I first started in 2011 I was running DRBD in VMs and zebra to unicast my presence. Future upgrades were incremental steps to more resilience to where I am today with a fully redundant architecture in 2 data centers. In fact the only thing that made me miss my uptime target one year was failed generator maintenance by my provider.

I’m running five-nines with my setup and I’m the only operator. Monitoring and automatic failover is not difficult but I think it requires a solid architecture from the ground up.

OK, I concede that it is not completely inconceivable to do that, but unless the service you're operating is relatively light in its demands on the tech stack, I think it's a very impressive achievement to maintain infrastructure that can consistently and reliably deliver that performance on your own if you're also the person doing the development work and your infrastructure costs aren't getting silly.

We have a simple, fully redundant architecture at one of my businesses as well, and I suppose we probably do achieve five-9s most years, but I wouldn't be willing to guarantee that to customers with serious money on the line if we missed it. We're still only D disk failures at similar times away from degraded performance while we spin up new machines from scratch, or N network failures away from degraded performance until we can bring up more capacity where it's still available.

I remember significant outages of S3, Gmail, and Heroku last year. So to me, it looks like they also don't reach five nines uptime.. .

Also, I'm not convinced that it's needed for a normal product. When my work Gmail was offline, I just had lunch early and then later it worked again.

A single founder offering higher availability than Gmail sounds like a masochist to me.

> If you go dedicated, it's much easier and you only need to keep that one box running.

Can/should you really run everything on just 1 box, with rather huge projects? Why not gain redundancy/uptime/peace of mind by having multiple (redundant) dedicated boxes?

Everything else being equal, having more boxes leads to more failures. If each server fails on average every 10 years, then with ten servers you expect on average one failure every year.

True. But a failure of a redundant server (say, 1 out of 3 application servers) would then not force you to cancel your night/weekend/vacation.

Even day to day operations/maintenance becomes that much painful. Think of all those security patches, zero down time deployments, failovers, log aggregation, monitoring setup so on and so forth.

Two servers with manual failover is probably the sweetspot; especially if you exercise the failover often. Confirming after every release is best, but once a month is probably fine. Then your incident response can be verify the lead server is dead, or ensure it's dead and switch to the alternate.

But one server is way more convenient, until it isn't.

To avoid configuring every service for high-availability on two servers, you can do surprisingly well with a replicated VM and DRBD or equivalent.

In the event of unplanned failover, it looks to the VM like an unplanned abrupt reboot took place. In reality it reboots, usually very fast, on the other host.

All services running inside can recover in the usual way (journalled filesystems, databases, programs restart), and don't need any high-availability configuration or replicas configured.

You do need to ensure I/O is committed durably across the network, including I/O barriers. This is a combination of VM host, filesystem and DRBD config.

(It is actually possible to do this with the VM not even seeming to reboot, so network connections and processes are unaffected by the fail-triggered instant migration. This is done by running VMs in synchronised tandem and is a rather more advanced technique. I've never used it.)

If I were doing the small/medium SaaS thing, I would vastly prefer to scale vertically rather than horizontally.

Maintaining a single machine is always going to be much easier than a cluster with k8s. Not to mention you can often toss most of your data set in RAM.

Not having to worry about sharding, affinity issues, DNS/addressing/networking, extra security is a godsend. Everything is easier on one machine.

Having a redundant machine for failover and release staging might be a good idea. But you'll need to figure out how to replicate your database and possibly your in memory cache layer (redis/memcached/etc.) and test it all. Not to mention database migrations can get tricky. Really, most people can probably get away with the typical maintenance window and notification, and shut everything down for 4 hours on a Saturday night or whatever. I mean... major banks and utilities do this. You'll be fine.

I had a dedicated server at Hetzner that would completely lock up every few weeks to months randomly, so they aren't always reliable. I ran it for about a year as I always had other things to deal with, but the abrupt loss of my services was kind of embarrassing and I should have reported it sooner than I did.

To be fair to Hetzner, eventually when I reported it they immediately took the initiative to replace it, no questions, and gave me options for when I'd like that to happen. Never had any problems since.

> Plus, honestly, would your customers really mind if you're offline for 5 minutes?

Idk what kind of software you work on but in my slice of the B2B SaaS world, 5 minutes of downtime during business hours would generate 100s of support tickets with very angry users letting us know they couldn’t do their job.

> 5 minutes of downtime during business hours would generate 100s of support tickets

Perhaps part of the answer is that if you're a solo founder who is staying solo, try not to create a business where you have these kinds of dynamics.

OR, if you are creating one that does, scale up past being solo ASAP.

Fair point and agree 100%.

I would argue there are some services that need 5 nines (or more) of availability.

Your support ticketing system and company web presence are two I can think of. I provide the ticketing service myself but I outsource my web presence (the landing page not the app) to a third party service. My thought is the web site should never go down. If the ticketing system is down it means things are beyond hosed :)

If you are a solo SAAS operator and have enough customers to generate hundreds of support tickets in 5 minutes, maybe hiring more help several years ago would have been a good idea?

I doubt you're trying to solve the same problems as the GP at that scale, or probably even the same types of problems.

That’s fair. I forgot all this was in context of solo founder. Back when I was solo we definitely didn’t have that many users.

Out of curiosity how many customers do you have as a solo founder?

I definitely had too many users before we hired more developers and support staff.

Maybe 400 businesses each with 10+ employees and 10,000+ customers (who also had access to the platform). It was too much and I regret not hiring sooner.

Another thread mentioned here that hiring someone you trust enough to own problems when you’re unavailable is daunting. I concur with that.

Maybe 400 businesses each with 10+ employees and 10,000+ customers (who also had access to the platform). It was too much and I regret not hiring sooner.

FWIW, that sounds like an interesting story/case study, if you're willing to share some time. I'm not surprised by your conclusion, but I'm quite impressed if you managed to scale even tolerably well to that level before bringing in some extra help.