sksjvsla

I sincerely appreciate it — and I would never downvote a reply like this. It's clear you’ve been around the block, and I respect the experience and nuance you're bringing to the discussion.

On the topic of Hetzner and account risks, I completely agree: this is not just a technical issue, and that's why we built a multi-cloud setup spanning Hetzner and OVH in Europe. The architecture was designed from the start to absorb a full platform-level outage or even a unilateral account closure. Recovery and failover have been tested specifically with these scenarios in mind — it's not a "we'll get to it later" plan, it's baked into the ops process now.

I’ve also engaged Hetzner directly about the reported shutdown incidents — here’s one of the public discussions where I raised this: https://www.reddit.com/r/hetzner/comments/1lgs2ds/comment/mz...

What I got in a private follow-up from Hetzner support helped clarify a lot about those cases. Without disclosing anything sensitive, I’ll just say the response gave me more confidence that they are aware of these issues and are actively working to handle abuse complaints more responsibly. Of course, it doesn't mean the risk is zero — no provider offers that — but it did reduce my level of concern.

Regarding Cloudflare, I actually agree with your point: vendor contract structure and incentives matter. But that’s also why I find the AWS argument interesting. While it’s true that AWS is incentivized to keep accounts alive to keep billing, they also operate at a scale where mistakes (and opaque actions) still happen — especially with automated abuse handling. Cloudflare, for its part, has consistently been one of the most resilient providers in terms of DNS, global routing, and mitigation — at least in my experience and according to recent data. Neither platform is perfect, and both require backup plans when they become uncooperative or misaligned with your needs.

The broader point you make — about counterparty risk, legal ambiguity, and the illusions of control in tech stacks — is one I think deserves more attention in technical circles. You're absolutely right that these risks aren't visible in logs or Grafana dashboards, and can't always be solved by code. It's exactly why we're investing in process-level failovers, not just infrastructure ones.

Again, thank you for sharing your insights here. I don’t think we’re on opposite sides — I think we’re simply looking at the same risks through slightly different lenses of experience and mitigation.

As I wrote elsewhere in this thread:

Being able to stay compliant and protect revenue is worth far more than quibbling over which cloud costs a little less or much a monthly salary for an employee is in various countries.

The real ratio to look at is cloud spend vs. the revenue.

For me, switching from AWS to European providers wasn’t just about saving on cloud bills (though that was a nice bonus). It was about reducing risk and enabling revenue. Relying on U.S. hyperscalers in Europe is becoming too risky — what happens if Safe Harbor doesn’t get renewed? Or if Schrems III (or whatever comes next) finally forces regulators to act?

If you want to win big enterprise and governmental deals, Then you got to do whatever it takes and being compliant and in charge is a huge part of that.

> Disaster recovery, monitoring, etc

The ISO 27001 has me audited for just that (disaster recovery and monitoring) so that settles it, no?

Also worth noting that these are the two things you don't really get from the hyperscalers. If you want to count on more than their uptime guarantees, you have to roll some DR yourself and while you might think that this is easy, it is not easier than doing it with Terraform and Ansible on other clouds.

I have had my DR and monitoring audited in its AWS and EU version. One was no easier or harder than the other.

But the EU setup gave me a clear answer to clients on CLOUD act, Shrems II, GDPR, Safe Harbor, which is a competitive advantage.

I wanted to comment on this but mistakenly put the answer here. Sorry.

https://news.ycombinator.com/item?id=44335920#44346481

If you want me to assess what I would be needing the next 5-10 years, I'd make a very different thread here on HN.

The defining conditions is my current setup and business requirement. It works well and we've resisted pretending that we know where we will be in 5 years.

I am reminded of the 2023 story of the surprisingly simple infra of Stack Overflow[1] and the 2025 story of that Stack overflow is almost dead[2]

Given that the setup works now, one can't add that it is only working "for now". I see no client demand in the foreseeable future leading me to think that this has been fundamentally architected incorrectly.

[1] https://x.com/sahnlam/status/1629713954225405952

[2] https://blog.pragmaticengineer.com/stack-overflow-is-almost-...

Exactly. Well said.

A thing we learned in this process is that there's many levels of abstraction which you can think of rollback and locking down SSH and so on and so forth.

If your abstraction level is AWS and the big hyperscalers, it would be to use Kubernetes, but peeling layers of complexity off that, you could also do it with Docker Compose or even Linux programs that are really battle tested for decades.

Most ISO certified companies are not at hyperscale so here is a fun one: Instead of Grafana Agent from 2020, you could most likely get away better with rsyslog from 2004.

And if you want your EKS cluster to give you insights you have configure CloudWatch yourself so does what hands-off is there comparing that setup to Ubuntu+Grafana Agent?

I have not experienced this in spite of rumours online. As I mention in these two comments, given these we decided to design our way around it by assuming that they would both go down at some point of time (but not at the same time).

1. https://news.ycombinator.com/item?id=44335920#44339234

2. https://news.ycombinator.com/item?id=44335920#44337619

As I wrote elsewhere in this thread:

Being able to stay compliant and protect revenue is worth far more than quibbling over which cloud costs a little less.

The real ratio to look at is cloud spend vs. the revenue.

For me, switching from AWS to European providers wasn’t just about saving on cloud bills (though that was a nice bonus). It was about reducing risk and enabling revenue. Relying on U.S. hyperscalers in Europe is becoming too risky — what happens if Safe Harbor doesn’t get renewed? Or if Schrems III (or whatever comes next) finally forces regulators to act?

I see where you’re coming from — no doubt, services like SQS and S3 make it easier to build reliable, distributed systems without reinventing the wheel. But for me, the decision to shift to European cloud providers wasn’t about wanting to build my own primitives or take on unnecessary complexity. It was about mitigating regulatory risk and protecting revenue.

When you rely heavily on U.S. hyperscalers in Europe, you’re exposed to potential disruptions — what if data transfer agreements break down or new rulings force major changes? The value of cloud spend, in my view, isn’t just in engineering convenience, but in how it helps sustain the business and unlock growth. That’s why I prioritized compliance and risk reduction — even if that means stepping a little outside the comfort of the big providers’ managed services.

If you were to divide the AWS customer base into a 10% bucket and a 90% bucket, a 90% bucket would not be the ones needing the infinite scale of AWS.

Yes, web apps all need logging, performance dashboard, redundancy, DB backups and such.

This could be a stack that could be parametrised with sound defaults just requiring some terraform provider credentials as well as a path to an executable web app and a choice of database engine.

ISO readiness built-in and abstracted at the OS level rather than programming language level.

If anyone wants to "assetize" what I built, reach out at jk@datapult.dk. I bring a battle-tested setup that has been ISO certified by independent auditors.

You bring clients directly or indirectly with marketing/growth hacking mindset.

You’re describing exactly the kind of vendor lock-in treadmill I was trying to avoid. What you see as “rapid iteration” looks a lot like redoing the same work every few months because of shifting cloud-native limitations.

Also, the idea that using VPS or non-hyperscaler clouds means “poorly integrated and insecure mess” feels like AWS marketing talking. Good ops doesn’t mean gatekeepers — it means understanding your system so you don’t need to swap out components every quarter because the last choice didn’t scale as promised.

I’d rather spend time building something stable that aligns with my compliance and revenue goals, than chasing the latest AWS feature set. And by the way, someone still has to keep all that AWS software up to date — you’ve just outsourced it and locked yourself into their way of doing it.

People keep comparing cloud costs to employee costs, but I think that’s the wrong metric. The real ratio to look at is cloud spend vs. the revenue you can unlock.

For me, switching from AWS to European providers wasn’t just about saving on cloud bills (though that was a nice bonus). It was about reducing risk and enabling revenue. Relying on U.S. hyperscalers in Europe is becoming too risky — what happens if Safe Harbor doesn’t get renewed? Or if Schrems III (or whatever comes next) finally forces regulators to act?

Being able to stay compliant and protect revenue is worth far more than quibbling over which cloud costs a little less.

It is more trivial than it seems. How did people manage a Postgres instance prior to RDS? Of the entire feature list, what parts of RDS do you use?

1. Dumping a backup every so often?

2. Exporting its performance via Prometheus, and displaying in a dashboard?

3. Machine disk usage via Prometheus?

4. An Ansible playbook for recovery? Maybe kicking that into effect with an alert triggered from bullet 2 and 3.

5. Restoring the database that you backed up into your staging env, so you get a recurring, frequent check of its integrity.

This would be around 100 to 500 lines of code of which an LLM can do for you.

What am I missing?

The network allows relevant ports from the respective IPs and so does the UFW so the servers can communicate between in each other in a restricted way. Needless to say communication is encrypted with certificates.

Our logging server will switch primary DB in case the original primary DB server is down. Since we are counting on downtime, the monitoring server is by default not hosted in the same places as the primary DB but in the same place as the secondary DB.

We assume that each clod will go down but not at the same time.

We did not consider that because we are a lean, small startup. We barely have a premise to put anything on.

We talked to a few premium hosting vendors in Denmark and to build our own redundancy beyond what they guarantee, it actually became more expensive than AWS.

The cloud provider can be compliant with your app being so.. Most apps will not pass an ISO audit unless designed to do so.

Nice observation.

I took for granted that Hetzner and OVHcloud would be prone to failures due to their bad rep, not my own experience, so I wanted to be able to direct traffic to one if the other was down.

Doing load balancing ourselves in either of the two clouds gave rise to some chicken and egg situations now that we were assuming that one of them could be down (again not my lived experience).

Doing this externally was deliberate and picking something with a better rep than Hetzner and OVHcloud was obvious in that case.

Yeah, the SSH port isn't publicly exposed

Well said. It reminds me of a story I heard in a podcast once.

A database administrator for a drug cartel became an informant for the police.

His cartel boss called him in on a weekend due to a server errors. He said in the podcast "I knew I've been found out because a database running Linux never crashes"

Makes you wonder what everyone is telling themselves about the need for RDS..

Fred Brooks, the author of The Mythical Man-Month said:

> “Software is ten times easier to write than it was ten years ago, and ten times as hard to write as it will be ten years from now.”

Ansible, Hetzner, Prometheus and object storage will give you RDS if you prompt an LLM, or at least give you the parts of RDS that you need for your use case for a fraction of the cost.

Read through this list here, it should give you a good sense of what logging and monitoring is sufficient for ISO and valuable to us:

https://news.ycombinator.com/item?id=44335920#44337659

If you have any more questions, just reach out at jk@datapult.dk

Sounds like an interesting use case.

Totally valid concerns — I don’t disagree that DIY hosting comes with real risks that managed platforms abstract away (but AWS could close your account too).

We didn’t go into this blind though — we spent a lot of time testing scenarios (including Hetzner/OVH support delays) and designing mitigation strategies.

Some of what we do:

• Our infra is spread across multiple providers (Hetzner, OVH)) + Cloudflare for traffic management. If Hetzner blackholes us, we can redirect within minutes. • DB backups are encrypted and replicated nightly to various regions/providers (incl. one outside the primary vendors), with tested restore playbooks.

The key point: no platform is free of counterparty risk — whether that’s AWS pulling a region for legal reasons, or Hetzner taking a server offline. Our approach tries to make the blast radius smaller and the recovery faster, while also achieving compliance and cutting costs substantially (~90% as noted).

DIY is definitely not for everyone — it is more work, but for our particular constraints (cost, sovereignty, compliance) we found it a net win. Happy to share more details if helpful!

Oh, an imagine being kicked out of AWS and you used Aurora.. My certified multi-cloud setup with standard components should not make you cringe.

Before migration (AWS): We had about 0.1 FTE on infra — most of the time went into deployment pipelines and occasional fine-tuning (the usual AWS dance). After migration (Hetzner + OVHCloud + DIY stack): After stabilizing it is still 0.1 FTE (but I was 0.5 FTE for 3-4 months), but now it rests with one person. We didn’t hire a dedicated ops person.

I am curious why you think AWS services are more hands-off than a series of VPSs configured with Ansible and Terraform? Especially if you are under ISO 27001 and need to document upgrades anyway.

I looked at Kamal, Dokku, and CapRover — all great tools if you want to abstract away server management. But for a HIPAA/ISO 27001 certifiable app, I need a higher level of control and auditability across the entire stack.

With Ansible, I can version everything — from server hardening to DB backups — and ensure idempotent, transparent provisioning. I don’t have to reverse-engineer how a PaaS layer configures things under the hood, or worry about opaque defaults that might not meet compliance requirements.

There's nothing wrong with these tools, but once you're in the mood for the ISO certification, and once you start doing these things yourself, they actually seem like a step backwards or add very little value.

I also prefer running my own DB backups rather than relying on magic snapshots — it's easier to integrate with encrypted offsite storage and disaster recovery policies that align with ISO requirements. This lets me lock down the environment exactly as needed, with no surprise moving parts.

Tools like Kamal/Dokku/CapRover shine for fast, developer-friendly deploys, but for regulated workloads, I’ll take boring, explicit, and auditable any day.

Agreed, LLMs helped us with this.

This comment is great. Upgrade processes should be part of your internal processes if you want to get ISO 27001 certification that is not just checking the boxes but actually something you use for more control of your development and release cycle.

AWS RDS does not upgrade major or minor versions of Postgres or, as you mentioned, MySQL. In that case, they might patch update it. But these patch updates are easy to do yourself and does not take long to be reminded of in your ISMS and then subsequently carry out.

The purpose of this post is not to justify cloud hyperscalers versus European servers. It is actually a post on how to manage a highly regulated, compliant, and certified server setup yourself outside AWS because so many people just have their ISO certification on AWS infrastructure and once they got that they are never able to leave AWS again.

If you have no client demand and no real need to work on updating your infrastructure yourself, then you can go ahead and not go for an ISO 27001 certification and let AWS RDS update as it pleases. But if you operate a complex beast in a regulated industry such as employment law, finance, and such, then you get some more fun challenges and higher need for control.

You can't do single server setup for your workloads if you are ISO 27001 compliant and, further, you must have a separate server for logging and monitoring.

No matter load, there is a need for complexity for this certification.

Not all employees log in daily. For a scheduling app, most people check a few times a week, but not every day.

Daily active users (DAU) = around 10,000 to 20,000

Peak concurrency (users on at the exact same time) = generally between 1,500 to 2,000 at busy times (like when new schedules drop or at shift start/end times)

Average concurrent users at any random time = maybe 50 to 150

Why cloud costs can add up even for us:

Extensive use of real-time features and complex labour rules mean the app needs to handle a lot of data processing and ultimately sync into salary systems.

An example:

Being assigned to a shift has different implications for every user. It may trigger a nuisance bonus, and such a bonus could further only be triggered in certain cases, such as when you had the shifts assigned compared to when it start time.

Lastly, there is the optimizing of a schedule why is computationally expensive.

Here you go. Write me at jk@datapult.dk, if you need more.

A.5.25 Security in development and support processes:

Safe rolling deploy, rollback mechanisms, NGINX health checks, code versioning, Prometheus alerting for deployment issues

A.6.1.2 Segregation of duties:

Separate roles for database, monitoring, web apps; distinct system users

A.8.1.1 Inventory of assets:

Inventory management through Ansible inventory.ini and groups

A.8.2.3 Handling of assets:

Backup management with OVH S3 storage; retention policy for backups

A.8.16 Monitoring activities (audit logging, monitoring):

auditd installed with specific rule sets; Prometheus + Grafana Agent + Loki for system/application/audit log monitoring

A.9.2.1 User registration and de-registration:

ansible_user, restricted SSH access (no root login, pubkey auth), AllowUsers, DenyUsers enforced

A.9.2.3 Management of privileged access rights:

Controlled sudo, audit rules track use of sudo/su; no direct root access

A.9.4.2 Secure log-on procedures:

SSH hardening (no password login, no root, key-based access)

A.9.4.3 Password management system:

Uses Ansible Vault and variables;

A.10.1.1 Cryptographic controls policy:

SSL/TLS certificate generation with Cloudflare DNS-01 challenge, enforced TLS on Loki, Prometheus

A.12.1.1 Security requirements analysis and specification:

Tasks assert required variables and configurations before proceeding

A.12.4.1 Event logging:

auditd, Prometheus metrics, Grafana Agent shipping logs to Loki

A.12.4.2 Protection of log information:

Logs shipped securely via TLS to Loki, audit logs with controlled permissions

A.12.4.3 Administrator and operator logs:

auditd rules monitor privileged command usage, config changes, login records

A.12.4.4 Clock synchronization:

chrony installed and enforced on all hosts

A.12.6.1 Technical vulnerability management:

Lynis, Wazuh, vulnerability scans for Prometheus metrics

A.13.1.1 Network controls:

UFW with strict defaults, Cloudflare whitelisting, inter-server TCP port controls

A.13.1.2 Security of network services:

SSH hardening, NGINX SSL, Prometheus/Alertmanager access control

A.13.2.1 Information transfer policies and procedures:

Secure database backups to OVH S3 (HTTPS/S3 API)

A.14.2.1 Secure development policy:

Playbooks enforce strict hardening as part of deploy processes

A.15.1.1 Information security policy for supplier relationships:

OVH S3, Cloudflare services usage with access key/secret controls; external endpoint defined

A.16.1.4 Assessment of and decision on information security events:

Prometheus alert rules (e.g., high CPU, low disk, instance down, SSL expiry, failed backups)

A.16.1.5 Response to information security incidents:

Alertmanager routes critical/security alerts to email/webhook; plans for security incident log webhook

A.17.1.2 Implementing information security continuity:

Automated DB backups, Prometheus backup job monitoring, retention enforcement

A.18.1.3 Protection of records:

Loki retention policy, S3 bucket storage with rotation; audit logs secured on disk

> Techincal

I'm trying to share as much technical across this thread as for your two examples:

System upgrades:

Keep in mind that as per the ISO specification, system upgrades should be applied but in a controlled manner. This lends itself perfectly to the following case that is manually triggered.

Since we take steps to make applications stateless, and Ansible scripts are immutable:

We spin up a new machine with the latest packages and once ready it join the Cloudflare load balancer. The old machines are drained and deprovisioned.

we spin up a new machine We have a playbook that iterates through our machines and does it per machine before proceeding. Since we have redundancy on components, this creates no downtime. The redundancy in the web application is easy to achieve using the load balancer in Cloudflare. For the Postgres database, it does require that we switch the read-only replica to become the main database.

DB failover:

The database is only written and read from by our web applications. We have a second VM on a different cloud that has a streaming replication of the Postgres database. It is a hot standby that can be promoted. You can use something like PG Bouncer or HAProxy to route traffic from your apps. But our web framework allows for changing the database at runtime.

> Business

Before migration (AWS): We had about 0.1 FTE on infra — most of the time went into deployment pipelines and occasional fine-tuning (the usual AWS dance). After migration (Hetzner + OVHCloud + DIY stack): After stabilizing it is still 0.1 FTE (but I was 0.5 FTE for 3-4 months), but now it rests with one person. We didn’t hire a dedicated ops person. On scaling — if we grew 5-10×: * For stateless services, we’re confident we’d stay DIY — Hetzner + OVHCloud + automation scales beautifully. * For stateful services, especially the Postgres database, I think we'd investigate servicing clients out of their own DBs in a multi-tenant setup, and if too cumbersome (we would need tenant-specific disaster recovery playbooks), we'd go back to a managed solution quickly.

I can't speak for cloud FTE toll vs a series of VPS servers in the big boys league ($ million in monthly consumption) and in the tiny league but at our league it turns out that it is the same FTE requirement.

Anyone want to see my scripts, hit me up at jk@datapult.dk. I'm not sure it'd be great security posture to hand it out on a public forum.