Ask HN: Best practices for multi-datacenter infrastructure? | Hacker News Reader

Ask HN: Best practices for multi-datacenter infrastructure?

2 points__warlord__5 comments

I'm deploying a set of applications on top of Kubernetes like GitLab and Argo suite (Workflows, CD, Rollouts), Harbor, Prometheus, and more and I need to have an active-active solution distributed across two data centers for all (if possible) of these tools.

One approach I'm investigating having Kafka queueing the commits (and other requests) and one consumer on each data center writing the commit into GitLab to execute CI/CD pipelines and deploy applications based on that commit. (CI/CD will only execute once)

This way I don't have to worry about replicating each application data from one data center to another and worry about where data is being written and so on.

I want to ask HN, are there best practices or documentation on how to do this kind of architecture that you can point me at?

Or is there a better way to work on this solution?

I did a quick check, but I'm not an expert.

Make one the main and the backup in readonly.

https://docs.gitlab.com/ee/administration/read_only_gitlab.h...

You can stop the readonly from doing CI.

Do a healthcheck on the main repo and let a gateway in front it.

If the main is down, undo the readonly access.

I'm not 100% sure what to do when the main goes backup though.

That aside, I'm not sure if you want this. What duration/year was a downtime on the datacenter blocking deployments/coding?

> I'm not 100% sure what to do when the main goes backup though.

I'm wondering this as well, it becomes very cumbersome to bring both instances at the same data consistency without a third party (like a queue)

I don't think a queue is possible concerning commits.

This could be part of the solution though. But automating everything seems not worth the effort.

https://docs.gitlab.com/ee/administration/geo/disaster_recov...

Modifying each application to add active-active replication sounds like a nightmare. I would seriously question where these requirements are coming from.

Is not modifying for active-active __per se__ but we want to have as much uptime as possible.

Async replication is another option but that needs to be enabled per application and on each database cluster we use. That's why I'm thinking to have active-active using a queue (to simplify replication and being able to have a "journal" that we can do replays from.

Reply on news.ycombinator.com