Incident with Actions, API Requests, Codespaces and Pages
githubstatus.com
githubstatus.com
Apr 17: 17:17 - 17:42, 19:28 - 19:53
Apr 18, 09:28 - 09:51, 14:35 - 15:29
Apr 19, 18:28 - 19:27
Apr 24, 12:57 - 13:27
Apr 24, 21:14 - Apr 25, 03:18
Apr 25, 23:26 - Apr 26, 00:29
Apr 27, 08:59 - 09:56
Apr 28, 12:26 - 12:45
May 2, 14:58 - 15:49
May 4, 15:55 - 16:23
May 9, 08:07 - 10:04, 11:32 - 21:14
May 9, 22:39 - May 10, 00:06
May 10, 13:00 - ?
Edit: This comment was in response to the original title, which has since been changed.
3-days in May so far with outages.
GP point is, that's anything but "daily".
GP also depicts, it's not at the same time either.
How many billions of dollars did this collectively cost? That's lots of wasted engineer time at best, and the inability to build a hot fix for your own outage at worst.
There SLA is for 99.9% ("three nines").
Which equates to:
Daily: 1m 26s
Weekly: 10m 4.8s
Monthly: 43m 28s
Quarterly: 2h 10m 24s
Yearly: 8h 41m 38s
Today they already spent their whole quarterly budget. With the incidents April and May they have already exceeded their whole yearly budget.
Saying "the people own" is just a way of saying the state owns something. The state owns all the things under Communism. Every business is a state business.
The state owns nothing under Communism as Communism assumes no state, social classes, private property and money. I do not think Communism can be achieved though, maybe in prehistoric society of docile gatherers of which we are not.
But that's no reason to give up and make greed mandatory.
Under original doctrine there is no State because "the people" will have evolved enough to do without it.
Well, yes, since “capitalism” was literally coined to describe a particular then-existing system in which the instrumentality of the state not only exists but also, most particularly through a specific structure of property rights, directly serves the interests of the capital-owning class to the relative disadvantage of other segments of society, and has consistently since then referred to that system (or ones sharing some of its salient features) both for those who have advocated moving farther from it and for those advocating its adoption or a return to it in places which have moved on, in whole or in part, from it.
Other than it's gibberish? Capitalism, a system where money is used to purchase property by individuals, is a society ... without issued currency or individual property rights, you say?
Look around you, are capitalist countries stateless, and are stateless countries Capitalistic?
As you say, the difference is whose interests dictate these ends: those of ordinary citizens or those of the monopolies and international finance.
https://www.institutmontaigne.org/en/expressions/influence-w...
6.7% of the country runs 92% of the boardrooms.
... If you're not a woman, gay, trans, black, poor, ...
So, yes, in the meantime, the state takes control over some amount of enterprise and manages it for its own goals rather than to generate profits for a small number of owners. Is that a bad thing? In a world where for-profit healthcare in the US is famously more expensive and less effective than nationalized healthcare in any other OECD country, where the (unionized!) USPS is one of this country's most-popular and most-reliable services even despite years of meddling from capitalists in government pushing to gut it, in a world where famously most of our R&D is publicly funded, is it really so hard to imagine that it might be an effective way to manage production?
> doesn't have the same connotation of "ruling over others without their consent".
Usually taking things by force requires violating consent. That is a very basic tenant of Commmunism. The bourgeoisie are forcibly removed from power, with arms, usually killed, and their property taken.
> whose contradictions are propped up by a force-monopoly in the form of "a state"
And how is the dictatorship of the proletariat preserved against the minority who disagrees? Hint, it involves rifles and usually firing squads.
> society can be transitioned over time into one where a state no longer needs to exist
Yes, that's gone so well up to date. Dictators with extreme powers over entire states are well known for quietly surrendering their power and moving to a form of utopic anarchy.
Even Marx couldn't answer how the utopic Communist post-state society actually continued it's existence without a state. "The vanguard" who at best are stateless secret police?
> in the meantime, the state takes control over some amount of enterprise and manages it for its own goals
Correct
> Is that a bad thing?
DMV, VA, Census Bureau, the IRS.
> USPS is one of this country's most-popular and most-reliable services
and is private but government directed
> In a world where for-profit healthcare in the US is famously more expensive
and has no waiting lists compared to 14.5 weeks in the UK at the moment and 20.9 weeks in Canada
> and less effective than nationalized healthcare
The US is in the top 10 for survival rates for most diseases/conditions (eg cancer) and we beat out the UK and Canada significantly. We have a lower life expectancy because we are too fat.
(This is not a suggestion to try anyway, nor a suggestion to eliminate humans in order to further any kind of Marxist "ideal")
At 3/4 Capitalist, your nation can prosper dramatically. At 3/4 Communist, you will know nothing but suffering and deprivation.
Countries that have gotten closer to a free market orientation have prospered. Singapore, Hong Kong, the United States; all of the success stories of post USSR Europe are stories of market economics winning (including the dramatic example of the Baltics). All of Scandinavia operates via market based economic systems. Germany is a market based economy, not a Socialist economy, and they're the economic dynamo of Europe; the same goes for Britain, and even France (which certainly has a large welfare state, and is a mixed economy but not Socialist). There isn't a single prosperous Socialist nation in the world today. There are dozens of prosperous market based economic systems by contrast (the top several dozen most prosperous economies are dominated by market systems).
Going the other direction, the closer you get to Socialism, the worse things get. And there are obviously limits to the market direction, anarchy does not work (although I'd argue that's not the market direction; market based economies in particular require the rule of law and a government to enforce it).
The only country that can be argued to stand apart from this, even if temporarily, is China. However they're still drifting on the liberalizations of Deng and the benefits it spurred, that is grinding down under Xi. Their situation has worsened as they have headed back toward Socialism (and its inevitable increased authoritarianism) and away from market liberalization.
My preference is for a social democracy, of the variety sometimes found in Europe.
Soviet Capitalism never existed. The Russians barely made any effort at enshrining a just rule of law, protecting individual rights, including protecting property rights.
One of the first things Putin did when he took power is to begin destroying individual rights (so as to bolster the power of the state), which market based economic systems rest upon.
Not sure why some commenters here saw the name Marx and immediately jumped to the conclusion it was an endorsement of Marxism.
You can see similar behavior (hiding issues, lying about them) in any scenario in which punishment is tied to the failure to meet the metrics. When the metric becomes the goal, people will game the metric in order to avoid punishment and/or reap the rewards.
The only kind of measurement that doesn't cause this behavior is the neutral measurement - measurement which is guaranteed not to change the decisions directly, but only serve as information which is later used in high-level optimization of systemic processes. In context of software companies, it would mean that status reports should only be reviewed in combination to other factors, such as tool used, features implemented, deadlines tightened, etc., and only to further understand how to optimize the process to avoid human error.
It's risky to implement that, because you have to trust your employees enough to say "you won't be fired because of your metrics, ever", but I believe it's less worse than having people falsely report good metrics.
Besides, why should anyone trust a leftist in power? We already know your lot will immediately outlaw and jail the opposition and pretend their uni-party system is "democratic". So I don't think you actually care about owners and power, you're just mad that you aren't in power perpetually.
Productivity (of something...) is ostensibly the intent of a capitalist venture, but the owner and ultimately prioritize anything they prefer. This is a problem for any hope of rational practice.
While that is true, the whole argument against metrics as goals is that they decrease productivity, which goes against a common interest of both capitalist and communist societies.
Productivity as a goal simply makes sense in the real world, regardless of whether there's an owner who is chasing profit, or a worker collective who has to meet their planned quota for the planned economy. The state of facts is such that binding metrics to rewards is harmful to their overall reliability, regardless of which economic system it's happening in.
- Regularly rotate personnel
- Reward review by a random board
A review based on a blend of quantitative metrics (basics/table stakes) and qualitative metrics (everything else) prevents quantitative-metrics-gaming from dominating.
The system has a ton of flaws, is inefficient, and produces suboptimal outcomes a fair amount of the time, but damned if it doesn't scale.
Goodhart's law!
The book "The Tyranny of Metrics" by Jerry Muller talks about this phenomenon in depth, giving many historical examples of metrics making systems worse instead of better. Very good read, I'd recommend it to anyone.
Me too! Fortunately, Google was smart enough to turn it up from "law about metrics".
> The book "The Tyranny of Metrics" by Jerry Muller talks about this phenomenon in depth, giving many historical examples of metrics making systems worse instead of better. Very good read, I'd recommend it to anyone.
Thanks! I'll have a look.
I know you probably heard that one, but as long as there is a boss or a general, it's not communism.
Also, democracy is vulnerable to this as well. No politician wants to appear incompetent, leading to lying about results and taking unnecessary risks.
I believe you may be confusing communism with anarchy.
Even the “communist” government still around like China just have capitalist economies in sheep’s clothing.
In the case of tech companies reporting record profits amidst massive layoffs, it's worth exploring whether certain aspects of Marxist theory could shed light on the systemic issues within our current economic system, rather than dismissing them outright based on past examples of "communism"
This sounds like a SRE's absolute nightmare.
I'm not sure I follow your logic here. People will ask SREs "are we up yet?" more because the status page automatically show downtime rather than manual, or what are you saying?
Good times, Amazon. Good times.
I.e., a material misrepresentation about the company's health.
I agree, but I'm going to repeat the same thing I said yesterday:
It really doesn't matter if your uptime is 80% as long as that 20% of downtime is happening when nobody is working.
An 80% uptime architecture is hopelessly simple to maintain (and restore and so on).
Complexity increases exponentially the more 9's you add, and those complexities need more and more people to paper over, eventually you end up with a false dichotomy. If you have 99.9% uptime and it's down when you need it with no clear resolution possible then you're not winning by having availability when you never needed it.
The people arguing against self-hosting are so committed; I can't help but feel people know that it's a false economy but they're financially incentivised or they like having pretty green dots on a special website or something.
Rachel by the Bay is US based (I am EU based) but even she has something to say on the subject: "My Nines are not your Nines": https://rachelbythebay.com/w/2019/07/15/giant/
First, like many people, I work in a company that spans multiple time zones. At least 16 hours of the day are in someone on the eng team's working hours. Plenty of places _do_ practically speaking need to use version control 24/7 when you factor in both global teams and out of hours incidents, which at a large enough scale become routine.
Second, when something fails, it doesn't _usually_ come back up of its own accord. It comes back up because someone fixes it. So you're either pulling people in out of hours to fix problems in these tools, or you're fixing them during work hours, which means that you have an outage when people are working! And if you work in an even slightly global team, oops, it might be both – it's in working hours for the people affected, but out of working hours for the people that maintain and can fix this system!
Third, because operating these systems tends not to be someone's full time job, even trivial issues like "the disk is full from log entries because it got mistakenly set to be in debug mode" can wind up taking a surprisingly long time to diagnose and fix.
What am I missing that makes it practical for a system to have 20% downtime, but only when people are not working?
I mean, human error is 95% of all outages, intentional changes to the running system and how it operates. You definitely can choose when you do those. Hardware or system failures outside of that are exceedingly rare, though I do submit they happen; simple systems are easier to stand up again if they fall over in the event that does happen.
I mean; Stopping a database for a backup is a prime example of something that's significantly harder to do when you're running it 24/7 365.
Similarly, performing a host migration, or running a potentially dangerous major version upgrade.
What we do normally is limit the cost of rolling back, which is nice.
But, instead of "iterate quickly and if it causes an outage we roll back" it can easily be: one guy stays a little bit late and upgrades the server and if it doesn't work then they roll back. -- impacting a much more limited set of people.
Also: if you work globally, you can run upgrades on your edges.
Also also: 20% unavailability is just an extreme case. I have 99.997% uptime with a single host machine that lives on a shelf in my room. I'm not saying this is how you should run systems, but it's pretty normal for even single nodes to have insanely high availability out of the box. Bonus: if it does have an outage, because it's so simple a restore takes 23 minutes.
I know this because I run restores on another machine pretty often, and if I run a backup I usually run blue/green between these boxes.
If your company is in Europe, and your gitlab instance goes down for 4hrs every day between 2.30am and 6.30am -- does anyone care? 0 people working those hours means 0 hours of lost dev time.
If you have 5 minutes of availability lost at 9.30 every morning it's much worse (at least in GameDev, when everyone is pulling the latest generated binaries). 900 people unable to work for 5 minutes is 75 hours of lost dev time; quite a lot more than 0 hours, even if the actual availability of the system was significantly higher.
If your audience is yourself, and people aren't supposed to be working, availability of those services is not necessary.
Hell. the Google SRE workbook goes into exactly this, so I know I'm not crazy.
I’d like to get off Mr. Microsoft’s Wild Ride and go back to the much simpler and more reliable GitHub of yesteryear.
By their published stats, surely. In actuality the number of times I go to do a git operation and it fails because GitHub is being flakey is pretty high, and most of the time when I check the status page, it's clean.
edit:// self-awarewolf: Another reply from me pointing this out. haha
If your knowledge is minimally valuable, you should self-host your git and collaboration tools. If it's not, then whatever.
I basically have js disabled on GitHub except for certain pages.
Current pet peeve: clicking the little search field doesn't put my cursor in that field, but instead pops a HUGE command-bar-type UI that covers everything. I hate, hate, hate "Surprise! Ta-da!" UI like that. And the more of that crap they add, the slower the site's gonna get, to load and to use.
Really, you would get more uptime if you self-hosted using GitLab or Gitea, etc. even some open-source projects like wireguard, RedoxOS, ReactOS, GNU Ring, etc and many others are doing this, with no issues.
Centralizing to GitHub [1] really isn't a good idea and it is showing that it is beyond unreliable for years.
GitHub is going great, and it has never been better. /s
But yeah, if you have no plan or organization, too many cooks is detrimental.
- A large number of users will increase the likelihood that one of them will encounter a temporary issue; even if problems are intermittent, a single person using the system simply won't notice them, but lots of people will.
- More users tends to mean using more features, more system resources, etc, which increases the likelihood of an issue from either buggy/complex features, or resource exhaustion.
- Updates for a team typically involve being more careful and performing updates on a regular schedule, in order to minimize downtime.
- If a problem does occur and you're not around, the team is stuck until you are available. If you go on vacation or are hit by a bus, they're stuck longer.
- Having more users tends to require things like disaster recovery contingencies, security, etc. Somebody has to do this extra work, which is a cost in time and labor.
Where self hosted shines is in keeping complexity down and changes minimal. GitHub is a giant system, making it more likely for problems to interrupt service. They constantly ship changes, increasing the likelihood of interrupted service. A self hosted option can use the same simple system and version for much longer, only taking security patches until the version is EOL and needs to be upgraded.
Problems arise with the operations. Nobody is confident with doing the updates since they happen so rarely. If system is critical, do you dare to do updates during business hours? Is there proper test system to practice updates? Is anybody prepared to handle recovery scenarios? Who is testing backups? How do you transfer the knowledge when people who originally built the system leave?
The SBC runs updates weekly, rebooting as needed as per zypper ps. It takes about 2 minutes for a reboot cycle, and reboots about 50% of the time. During this, the service is hard down.
The NAS selfupdates roughly monthly, and it takes about an hour. The service is stopped during this time.
This gives me a "perfect hardware" uptime of about 99.8.
I've had a single hardware failure in the last year, but it had the service down for 3 days waiting on shipping for a part, which gives an empirical hardware of 99.17.
I've not had any major maintenance on this setup in the last few years, though I expect to replace the NAS's drives in stages over the next few weeks. However, for a best-case, I'll give it 100.
My internet has not gone out in 2.5 years, so I am going to give it an optimistic 100.
My power, however, has been out for a total of 30 hours, of which, 28 hours outlasted my UPS. Based on syslog from NUT calling shutdown, until the first timestamp on reboot, therefore I have a Grid+UPS power availability of 99.68
If we compose all these, we have a total uptime of 98.65 of total services, which is far less reliable for basic operations (git cli interactions) than any of Git(Lab|Hub), Bitbucket, SourceHut.
This only covers all services down cases, my single-system setup doesn't often partially fail, and isn't engineered to hide maintenance (and doing so would be far out of scope and budget). Large services tend to fail partially, as they are composed of many individual systems with varied duties, and the system is often designed to fallback to basic function in the event of failure.
Not to even mention the question of capacity, which my 4c 8t of amd64 comes nowhere close to the CDN-ish power of the major SaaS's in this space.
edit: I know this is a bit circle jerky, but damn I am frustrated
Maybe it's possible to run github actions locally in an emergency?