GitHub is down
status.github.com
status.github.com
Make sure your infrastructure is ready to weather such storms - have local backups and fallbacks, make sure your code is available even if github (or the Internet) goes down.
The more I hear 'cloud', the more I try to depend only on tools that work offline.
If you are connecting different services (message queues, databases, storage, transcoding), and spinning up and down instances based on current demand and spot prices, then cloud hosting is totally different. On the other hand, if you are just renting a couple of dedicated cloud instances, then yeah - they're just (virtualized) servers.
(Because yeah, if I host a Dropbox clone on my Raspberry Pi sitting on the wardrobe, I have my files "in the cloud" now.)
We need a new word for automated, dynamic allocation/deallocation/management of remote instances for various services. Something that would differentiate it from buying a VPS.
I guess it's frustrating that cloud means quite different things depending on the context, but I guess that's true of a lot of language.
It's the cloud-based offline tool storage platform that gives you access to your offline tools...from the cloud!!
My side project https://statusgator.com shows currently a dozen different services all posting warnings or down notices on their status pages. It's mostly CI services and the like, all which break in some way when GitHub is down.
It boggles my mind. If you're not adding a new dependency to the project, your CI server should never hit GitHub when doing builds. Who sets those things up without local cache?
The main reason that we switch to that was exactly to avoid deployment issues. Security is also a bonus (in case somebody modifies existing packages).
Probably there are tools like this for all type of packages.
INB4: my comment shows an example of someone going meta.
Along the lines of "two is one, one is none", anything that you don’t own or where you don’t have a SLA has to be considered not existing for a business.
People are getting too spoiled by perceived reliability. Internet is not electricity yet. And especially third-party services on the Internet.
If you are in a situation that you're using a DVCS but relying on centralised everything else, and saying that you're unable to work because your Issue tracker/CI server is down, have you really gained anything over the alternatives?
How do you propose to stop that with your proposal?
Perhaps the only case breaking both at once would be deliberate action (DDoS etc.).
Thanks for the snark. I know git is a distributed system. My question was about everything else being down (your issue tracker, CI server, wiki/documentation), not about git itself.
Or at least down to "fuck, the power grid went out" levels.
Generally try not to put external services in a critical path.
Fossil tracks issues (tickets), documentation, and wiki pages at the same level as it tracks code, so when you clone a Fossil repo, everything gets downloaded to your local computer. You can then respond to tickets and perform repo maintenance offline, syncing everything to the server when you're back up.
Edit: A few weeks ago I got a job a one-hour train ride away. I can commit, close tickets, and write docs on my way in, sync everything to the internet when I get to work, then do the same on my way back - even though I don't have a net connection during my commute. It works great. (Ok, not as good as not having to travel at all, but great nonetheless!)
This just keeps happening. It's not GitHub's fault that they keep getting DDoSed or have a network outage every now and again. But it's not a problem with no solution.
Github is trying their best as far as I can see, they manage to fix the majority of issues within three hours and even during those three hours its not the end of the world because you can still code.
Issue tracking and wikis are not a version control problem. Sure, their data can be version controlled. And documentation has been version controlled since Sccs, why would it require special care?
It doesn't, really, but it's in the same place as the code and everything else (this is what I meant as being "on the same level"). You don't need to clone the code and then clone the tickets, for instance. When you run "fossil timeline" on your local machine, you'll see entries for tickets being closed and wiki pages being updated interspersed among the actual code commits. This is useful to me as they're as much a part of the project as the code itself. Does this answer your question?
An basic example of an org file for tracking issues:
#+title: Project issue tracker
#+description: descr...
#+todo: OPEN(o!) | CLOSED (c@)
This is the issue tracker for Project. In order to add a
new issue, add it to the top of the entries as a plain org
entry, then hit =C-c C-t o= to mark it as open. In order
to close an issue, on it's headline, hit =C-c C-t C=,
this will prompt for a closing comment, see the buffer that
pops up for instructions.
Common tags: docs, features, bugs, security, ideas
* CLOSED Write README. :docs:bookkeeping:
- State "CLOSED" from "OPEN" [2016-03-22 Tue 02:16] \\
Written.
- State "OPEN" from [2016-03-18 Fri 22:33]
* OPEN Implement new features :features:
- State "OPEN" from [2016-01-27 Çrş 23:35]The differences that remain are purely in the interface. For example, although they're synced the same way, VCSes tend to display issues differently than source; it's easier to filter out issue changes from the timeline, and harder to accidentally delete other people’s comments. It also makes it possible for people who aren't actual members of your project to raise and comment on the issues, which wouldn't usually be possible (unless your VCS can manage permissions for each file).
One particular use case of yours that Fossil does support is the ability to have a "wiki/" subfolder and have the web interface automatically format and display the pages inside, so you can link to them more easily if you're using the HTTP server for hosting. I think I use this feature more than the built-in wiki because it’s possible to version the docs along with the source, ensuring they never go out of sync.
Being fair, fossil performance on giant scale projects might not compare quite as well against git or proprietary solutions like Perforce, but for the large majority of projects I expect it is more than adequate.
You can't do patches. You can't rebase. You can't rename files.
Tell me all about this good software and imagination you speak of?
Patches: You can select any 2 commits and create a patch between them ( http://fossil-scm.org/xfer/vpatch?from=cbda43e71bda52f0&to=e... ). If you want to apply the patch, you can just use patch. Whether patch itself should be integrated directly into fossil is I guess something people can argue over. I don't use patches often, but haven't had issues with them.
Rebase: This is what Hipp has to say - http://permalink.gmane.org/gmane.comp.version-control.fossil...
These things may not be exactly how you prefer them, but you're misinterpreting my point about good software and imagination. Almost no software in existence is perfect for everyone in every scenario, but that doesn't also mean that no software is good.
Please do not devolve the conversation by making outlandish statements that have no bearing on the conversation.
>If you need something that works in exactly the way you're used to and that's how you evaluated fossil
That's the exact opposite point I made though. Fossil constrains me in how I work and how I accomplish certain tasks -- with no benefit. Neither Git or Mercurial do that. What Fossil can do, both Git and Mercurial can do better. So why would I choose Fossil?
If you don't get any benefit out of a tool, then don't use that tool. I get value out of it and am not meaningfully impacted by the issues you have with it in practice.
I'm not a salesman and it's free software, so do what makes sense to you.
As an expert on myself, I must disagree.
>If you don't get any benefit out of a tool, then don't use that tool.
I don't use it. But I think you need to remember what I replied to, as that's an ill-placed sentiment in this thread.
You can create bundles and submit them for approval. Very similar.
> You can't rebase.
You have alternatives. The work philosophy is different, though.
> You can't rename files.
False.
Bundles do not have anywhere near the same flexibility and nuanced capacity that patches do.
>The work philosophy is different, though.
That's an apology if I've ever seen one. Again, not the same.
>False.
Rename a file. Then try to merge. Tell me what happens.
Interesting. That's basically what the people who suggested bundles (a Hg feature) said about patches.
I've used both as appropriate. Handy that Fossil has both.
Off-topic, but I'd reccomend that you install HTTPS Everywhere: https://www.eff.org/https-everywhere - the fact that you are linking the HTTP version means you don't have this very basic extension that really improves your security.
Granted, the documentation needs work. But all issues and docs are exposed as git repos.
Github makes it easy to host git repositories, but so do a bunch of other services and it's not that hard to host your own either way or to set up trees of repository or multiple mirrors or what have you. The reason people use a central hub is all the meta-information and exchange about the code and the contributor communities.
The repository on its own has very little value.
And lest you think it's different for the kernel or whatever, a mailing list or an IRC channel are still centralised SPOF[0], when the ML provider goes down so does the project.
[0] though possibly more resilient and easier to switch out of ones
Depends for who. When I'm a user of a project, I don't care at all about anything other than the repository. The source code is the only thing that matters.
Anyway, you can't really avoid centralization in most of the projects. Barring some extreme social experiments, a project always has identity, a "canonical version". That's your SPOF right there. Of course we have tons of experience with mitigating potential issues - from caching and mirroring to not doing stupid shit like telling your CI server to download dependencies from GitHub every time it builds your project. Temporary GitHub outages seem to be a problem mostly for people doing the latter kind of stupid shit.
What would be interesting to see though is some kind of torrent network for Git repos. The kind of where I ask for a particular commit of a particular project, and I get a way to download it from some random computer that has it (with all the magic crypto code signing for confirming the code was not maliciously changed, etc.). Node.js developers have local mirrors of third of GitHub anyway, so that would pretty much solve "GitHub is down" issues.
Though it would not solve the problem of people too lazy to have a local cache of their dependencies...
The repository doesn't have any more value for that use case, tarballs would be just as good if not better.
If you don't want special git semantics around it, you'd have to be clever about how you store it so you don't have conflicts within the metadata. I.e. a naive design just adding a markdown file per issue or for all issues will require manual merging all the time. Still, it seems doable.
There were a number of tentative distributed bug trackers a few years ago, they sucked and fizzled out. IIRC Fossil is an attempt at an entire distributed project management system, I don't know how it fares.
> I.e. a naive design just adding a markdown file per issue or for all issues will require manual merging all the time. Still, it seems doable.
That only works for your own personal project where you're the only user and contributor. Bug reporting by editing a markdown file (or even something actually usable like an org-mode file or an sqlite or BDB file) isn't going to scale very high and is way more effort than most bug reporters (even technical ones) will be willing to put in.
And even if your users were willing to subject themselves to that, they still need a way to send back those contributions somehow.
Or dat http://dat-data.com/
Or Camlistore? https://camlistore.org/
Unless you're working across multiple projects, it doesn't matter to you that a GitHub outage takes out everyone else's projects at the same time.
Can you beat GitHub's uptime with a self-hosted solution? Sure. But you'll waste more time setting it up & managing it than you'd lose from GitHub downtimes.
But there still are many open problems: - incentivised file storage - fair name spaces - search and indexing and more.
Also:
sudo npm install --global gittorrent
Cool! It automatically installs a local copy of half of GitHub! ;).
Someone could serve you a maliciously-modified repo, but Git itself would then reject those objects, since they won't lead to the SHA1 that Git knows that it's looking for -- you would keep trying other peers until someone gave you bits that match the correct SHA1.
I'm not so sure about that. I've only used it once, an only to test its functionality, I didn't do any in-depth testing. Well I wouln't use it in production, but in a non-critical, small-scale project it could be worth a try...
Fossil SCM seems to fit that bill. As another user said, they can address issues in a local, non-connected manner, and then merge the issues back to main.
The last piece would be to use IPNS so you can have the same key for a changing package. I just hope versioning support gets added in soon (it's in the spec, just not done yet with go-ipfs).
I think this should not be done in the same system as where people store their source repos. Mixing the two is convenient, but can stifle innovation. Better to have flexible and well documented abstractions and good separation of concerns.
Could you elaborate?
I think I've heard the argument before, but I don't really see how. These outages show up again and again, partly because people cannot access the project issues.
Also keep in mind that protocols are much more important than the actual code implementing them (see git for a good example!)
If I sync a git repo from random location, I have the files to make that package. However none of the documents that aren't explicitly in the git repo are provided. With Git[hub/lab/...], this is metadata outside of the repo. Ideally, I want and need that as well. If there's an issue, I can research it quickly and see if it applies to me... Or if I'm a maintainer, I can make a solution offline and then merge it when online.
I understand that it may not be the perfect tool to include everything... but it does seem to fit the bill. Fossil does seem to do this correctly. I'll have to look into it.
The problem is all the shit that is not in the VCS and not directly based on it: issues, code reviews, discussions, … and the community around them if you have external contributors.
"Gittorrent" attacks a problem which by and large is already solved, it might further improvement on it but it's improving something which is not the bottleneck.
Thanks for the downvote, my hearts goes to you. :/
1. There is literally no amount of money which will get you 100% uptime. You may be able to buy guarantees of 100% uptime, however...
2. ... which will not change the fact that things still go down, and that if something goes down in a way which violates your signed contract, the most you can hope for is some kind of monetary compensation and, maybe, an apologetic-sounding employee to scream at until you feel a bit better. If you expect to buy someone's care, you're woefully naïve.
My point is that the paying customers don't necessarily get a better service. All they get is limited compensation for enduring the same bad service all of the rest of us get.
Git clients probably supply some anonymous but consistent identifying information when doing a standard pull... per-client rate limiting, anybody?
It's a nice idea[l] at least; practically speaking there's enough bandwidth to keep the current disaster ticking over, so... :(