Hacker News Reader

Comment by jonaf | Hacker News Reader

Your question seems to indicate that you have a runtime dependency on github. If true, this is a problem no matter where your git repo resides. You need to architect a solution wherein you do not have a runtime dependency, especially if it is a single point of failure, or accept that your reliability will never be better than the sum of your services' reli abilities (there was an article on the "calculus of availability" or something like that recently).

What I personally do in AWS is bake my artifacts or other git-sourced data into AMI's. If you want a middle ground, you can instead push your artifacts to an s3 target -- s3 has better reliability / track record than github for this purpose.

View on HN

Thing is, runtime dependencies on Github is something that Github is promoting and selling now (https://github.com/marketplace/). Maybe this incident puts that value proposition in a more visceral context.

Just because they sell it doesn't mean people shouldn't know better.

How is the GitHub marketplace a runtime dependency? Code review, project management, build bots or whatever else should NOT affect the availability of your website.

Sure, if you measure uptime of your business and dev team by whether you're serving pages. I wouldn't.

If Github being down means you're not fixing your site today, I'd call it a runtime dependency.

I kind of see your point but it's the wrong phrase. It's not a runtime dependency in a technical sense.

see now you're just splitting hairs and getting into philosophy. if your website will be stale tomorrow because you can't update it today, then that is a runtime dependency imo.

Why it is down tomorrow?

(Hint: sans hug of death, because a runtime dependency brought it down.)

Incorrect. You need build-time dependencies to change an application, and runtime dependencies to run it.

Deploying/scaling is somewhere in the middle.

Yeah, I should have clarified I was talking with my operations hat on. From a development perspective, you might call it a "deploy-time" or "scale up" dependency.

naive tip: give your app that is relying on github a SKIPGIT environment variable to ignore the bits around git operations, and just make local changes as an interim hack.

> Your question seems to indicate that you have a runtime dependency on github.

Really he is speaking to the problem with SAAS. Sure it's cheaper to rely on someone else to do the heavy lifting for you and they can do this because they 'make it up in volume'. But the other side of that coin is no one really knows how to do that anymore. If you wanted to roll your own it's going to be very hard for your company to do the basics because you've become dependent upon the cloud, and someone else paying employees to do the basics for you... The basics have never been hard.

But a positiv counter argument would be that github/gitlab also does the sharing part of the jobs and you (hopefully) pay them to handle all the security aspect of that public sharing.

Opening a port to your local instance of git implies more thinking and security consideration. Of course you could host a mirror on a cloud instance, but then it's saas anyway and you have even more work to do before ever starting coding.

So yeah github/gitlab does the heavy lifting for you, but for small organisations that might be cheaper overall than to pay somebody mastering all required stack to implement and maintain your own instance(s). For big company no question that an in house git team is probably more efficient.

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

Git is inherently distributed. There's really no reason that Github has to stand between development and production.

I think the point is git is becoming the new svn for many people.

"Becoming"? Can you point to any large project (at any point in the history of Git) that doesn't use some centralized service as a single point of distribution/source of truth?

The beauty of Git's design, though, is that you've still got everything to use it distributed even if you, under normal circumstances, just use it centralized like a fancy SVN.

There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.

But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.

Agreed, that shouldn't be too hard if you're really in a emergency and can't wait. What I see though is some deep integration with their API with automatic tagging, releases, etc... That's hard to work around but still not impossible if you're willing to live with some chaos in your process until things are back.

Not if you're using more than git from github, for example downloading releases as zip files for deployment.

There are probably a lot of other tempting features to put on your deployment critical path but I'm not in an environment that uses github so I've forgotten.

Yeah definitely. I was just thinking about the basic use case where you deploy from it and that's it.

Webhooks are a big one: rather than polling a git server for changes it is very easy for CI/CD systems to rely on GitHub's HTTP webhooks in their critical path to kick things off on push/PR/branch/merge. Especially because GitHub does a great job of populating its webhooks with tons of useful information about the event that's tough to replicate with just git post-hooks.

Git supports multiple remote repositories. Have github be your main repo, keep a mirror locally or on another service (updated using hooks), and switch to it if github fails.

> Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub?

It's not a runtime dependency, it's a stupidity to rely on random repositories from the internets when building and deploying (remember the left-pad farca that halted work for half of the web programmers). You should always have your local copy of necessary sources or use repositories that can trivially be swapped with some other mirror.

We used to have the infrastructure for those things (in the form of binary and source package repositories), but it was mainly used by sysadmins, and programmers usually have an alergy for such infrastructure.

Speaking from an operations perspective, yes, it's a runtime dependency. From a development perspective, you might think of it as a "deploy-time" dependency. Think of it in contrast to compile errors. If your program has errors that your compiler can catch, then it will refuse to compile the program and you must fix the error before executing the program. A runtime dependency is one that may change after your program is compiled -- these are harder to test, especially when the program may fail depending on its environment. In the case of dependencies on services such as Github at deploy time, it's a "runtime dependency" for the operations programs. Since the operations programs are the ones with the runtime dependency, it's also in the operations team's purview (and responsibility) to revise their program(s) and eliminate the runtime dependency such that testing (and reproducibility of) deployment is deterministic and reliable.

> you can instead push your artifacts to an s3 target

This is our tactic. I tried to do baked AMIs at one point, but the 10-15 minute turnaround in registering them meant that we couldn't use them for staging or testing (too long to iterate changes).

Previously we were capistrano-deploying with git from bitbucket - every server had to individually git pull from servers on the other side of the continent, over the public internet. Susceptible to all sorts of problems.

You can speed up AMI baking by using chroot + a mounted EBS volume on a running instance. It should drastically reduce the time it takes to produce a new image. Also be sure that you're building EBS-backed AMI's, as S3-backed AMI's will require a lot of time for copy.

Reply on news.ycombinator.com