Hacker News Reader

Comment by poooogles | Hacker News Reader

>Does this mean that Docker -- a major infrastructure company -- does not have any on-call engineers available to fix this?

It appear to be that way. Reminds me when all of the reddit admins were stuck on a plane on the way back from a wedding [1].

Remember kids, improve your bus factor.

http://highscalability.com/blog/2013/8/26/reddit-lessons-lea...

View on HN

> all of the reddit admins were stuck on a plane on the way back from a wedding

Clearly, we need to operate in cells, so that no one knows everybody and will have everybody over for weddings and other parties.

Haha. Well its not uncommon to avoid putting an entire team on one plane. Pretty risky for a company to do that!

Yeah... the entire point of an on-call rotation is to specify who is available for incident response... if everybody is on the same plane at once then by definition nobody is on-call.

And in the extremely unlikely event of a plane crash, the company goes belly up.

There was a comment a bit below that suggested that those who paid for commercial support got it 24/7, but if that's true, id imagine the fix that commercial support would have given to paying customers would have fixed it for everyone else too...

Disclaimer: I work at Docker.

I believe commercial releases are downloaded from a separate infrastructure (to be confirmed).

Either way, the availability of Docker packages, free or commercial, is critical infrastructure and we should treat it as such. IMO our primary infrastructure team should have been involved, and someone should be on call for this. We'll do a post-mortem, find the root cause, and take corrective action as needed.

Apologies for the inconvenience.

You may want to lock that GitHub thread soon, it's getting argumentative and not very helpful.

> You may want to lock that GitHub thread soon, it's getting argumentative and not very helpful.

In open-source we call that "thursday" :)

What ever happened to community over code?

Part of a healthy community is accepting that people disagree a lot, have different values, and communicate their ideas in very different ways.

Where we draw the line is if people are being intimidated, bullied, insulted, or anything that even remotely resembles harassment.

Although I personally feel that some of the comments in that thread are pretty unfair and poorly informed, they don't seem to violate the social contract.

It's also possible that the latest release wasn't pushed to the commercial repo.

The commercial releases are different to the open source ones (they have different patches, and they do different releases), and have a different release schedule.

And don't put everyone on the same bus, I guess.

Actually since the aim is to avoid being hit by a bus, inside a bus seems like a pretty good place to put everyone.

Assume N is the total number of buses. Where N=1 your solution works. For N>1 the engineers can still be hit by a bus. This also ignores all other vehicles such as garbage trucks, semis and stealth bombers, all of which can take out your bus.

You want to avoid sticking everyone on the same (air)bus too.

https://en.wikipedia.org/wiki/Pacific_Southwest_Airlines_Fli...

Also called (by me) as "Baldrick's Bullet": https://www.youtube.com/watch?v=pKRxX3s3JlM

Captain Blackadder: Baldrick, what are you doing out there?

Private Baldrick: I'm carving something on a bullet, sir.

Captain Blackadder: What are you craving?

Private Baldrick: I'm carving "Baldrick", sir.

Captain Blackadder: Why? Private Baldrick: It's part of a cunning plan, sir.

Captain Blackadder: Of course it is.

Private Baldrick: You know how they say that somewhere there's a bullet with your name on it?

Captain Blackadder: Yes?

Private Baldrick: Well I thought that if I owned the bullet with my name on it, I'll never get hit by it. Cause I'll never shoot myself...

Captain Blackadder: Oh, shame!

Private Baldrick: And the chances of there being two bullets with my name on it are very small indeed.

Captain Blackadder: Yes, it's not the only thing that is "very small indeed". Your brain for example- is brain's so minute, Baldrick, that if a hungry cannibal cracked your head open, there wouldn't be enough to cover a small water biscuit.

The problem was that, even though Reddit had multiple engineers, they were all on the same plane, and thus unavailable at the same time.

There are two tiers to Docker's support, for certain, but as pointed out on the Github issue by a Docker team member here (https://github.com/docker/docker/issues/23203#issuecomment-2...), there's a definite urgency sensed by the team.

I don't think this would surprise anyone that has used Docker Hub in their CD pipeline. So many reliability issues.

We moved off it last year. When we went to cancel our subscription the other month downgrades/cancellations were broken on the site as a known issue; had to open a support ticket. Most of the UI issues were still present along with some new ones.

Reply on news.ycombinator.com