I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.
We're a company making WISP/ISP software here in India and our cloud (hosted) offering is also hosted on servers at our office itself[1]. Our infra manages hundreds of thousands of internet subscribers in India.
We're a small tech company with 25-30 people or so, with linux kernel experts who have experience of running software on questionable hardware so that service providers can take internet to new places in India, that experience helps a lot in keeping all our services up with minimum downtime.
Our office runs 24x7 with support. If anything goes down our business will be in jeopardy so we have protocol to be back up in minimum amount of time.
How many hours we spent on it? Less than a couple hours on setting up the self hosted gitlab instance. More time on getting people to get used to it and setting up the runner for CI, but I would have to spend that time on other paid solutions too.
As for time costing $0, definitely not, but people in India don't cost nearly as much as they do in the USA. Also because of experience with self hosting it is less of a deal for us to spend time on this.
[1] we have multiple bandwidth backup, electricity backup, etc needed to self host and we've always stuck with self hosting.
We have SaaS offering for provisioning, subscriber management (AAA, Radius), billing, OSS, CRM, customer portals (including mobile apps) and more!
Please let me know if you'd like to know more, also I would really love to know what software you use to manage your users, etc. and what WISP software is popular in the UK. Thanks!
As for this:
> Who will be woken up at night when there's an outage?
Unless they are working with teams around the world, does it matter if their server is down the whole night, as long as they can get it back up first thing in the morning?
> GitHub has the right to suspend access to all or any part of the Website or your Account at any time, with or without cause, with or without notice, effective immediately.
https://help.github.com/articles/github-corporate-terms-of-s...
So, we already had a 3-DC virtualisation platform.
We also chose Gitlab, with a primary in our primary DC (where dev/QA infrastructure resided), with daily full backups rsynced (as created by Gitlab) to the standby gitlab instance (and backups from the stadnby rsynced to the primary) and repos sync hourly. Plus, all volumes on the SAN-attached storage array were snapshotted and backed up to virtual tape in both sites.
The 3rd DC was too small for the same footprint, but had a smaller SAN array and we would be able to host the gitlab instance there in ~1 hour if both the primary and standby failed.
Total cost: * 2 VMs with 8GiB ram. All other costs were negligible. * 2x50GiB SAN volumes (at ~ $1/GB for capex+5-year-opex) * Some virtual tape costs (negligible to our other backup costs) * Time: * Approx 1 hour to set gitlab up * Approx 1 hour to setup backup replication and test restores * Approx 1 hour spent on Gitlab upgrades per year * Benefits: Integration with other on-premise infrastructure we would prefer not to expose to the 'cloud' via e.g. VPC gateways etc. * Unlimited (except by ram/storage assigned to VMs) private repos * Unlimited (except by ram/storage assigned to VMs) * Ability to deploy applications regardless of any other external outage (network, cloud etc.): priceless
I am not aware of any failures in the first 3 years of operation.
> What did the server cost? Negligible, we already had 12 physical hosts with 256GiB ram each in the primary and secondary site
> How many hours did you spend on it? 3 more hours than we would have if we had used a hosted solution, if the hosted solution would have cost 0 hours to integrate with our on-premise infrastructure in non-internet-exposed DMZs.
> What's your SLA?
Our standard SLA was 98.5% availability of all internal services measured monthly with 1h MTTR.
> Who will be woken up at night when there's an outage?
Same team that gets woken up for the other ~200 VMs that are more critical than the git repo
> What's your backup and recovery procedure?
Assuming we couldn't recover the primary: * Ensure the primary gitlab instance isn't taking writes/updates (e.g. disable the virtual NIC) * Start a restore from the lastest backup on the standby * Check if there were any commits to the git repos since the last backup, of so, push them to the recovered gitlab instance from the clones * Check that the repos are accessible and working * Flip the DNS from git.company to point to git.secondary-dc.company instead of git.primary-dc.company
We built everything to be able to failed over from the primary to secondary site and had tested all failovers (application servers, multiple databases, authentication servers, monitoring, infrastructure management applications etc.).
> but it's a bit naive to believe it's better than a hosted solution right away.
Depends on your use case and existing infrastructure/skills/operational posture etc.