1,633 karma · joined June 7, 2013
(Other comments have explained the bug so I won't repeat them)
I'm an ex-Googler, and I know how PRISM worked. This did not happen. All the statements made in https://googleblog.blogspot.com/2013/06/what.html are true at the time they were written. The sentiment in the title of that post accurately reflects how everybody involved felt about it.
(I can't talk about whether they're still true because I've not been there in nearly two years, I wouldn't know)
WoMs are a training exercise, intended to build familiarity with systems and how to respond when oncall. A typical WoM format is a few SREs sat in a room, with a designated victim who is pretending to be oncall. The person running the WoM will open with an exchange a bit like this (massively simpified):
"You receive a page with this alert in it showing suddenly elevated rpc errors (link/paste)" "I'm going to look at this console to see if there was just a rollout" "Okay, you see a rollout happened about two minutes before the spike in rpc errors" "I'll roll that back in one location" "rpc errors go back to normal in that location" ...etc
(Depending on the team and quality of simulation available, some of this may be replaced with actual historical monitoring data or simulated broken systems)
The "chaos monkey" tool, as I understand it, is intended to maintain a minimum level of failure in order to make sure that failure cases are exercised. I've never been on a team which needed one of those: at sufficient scale and development velocity, the baseline rate of naturally occurring failures is already high enough. We do have some tools like that, but they're more commonly used by the dev teams during testing (where the test environment won't be big enough to naturally experience all the failures that happen in production).
That's not what our 20% time is for, and 20% is way too small a number for that purpose. "20% time" (the way we use the term) is for personal/career growth/scratching itches.
Time spent on building systems that make our service better is my primary job. Manual remediation ("toil") is something to be tracked as a dangerous antipattern that must not be allowed to take over.
Toil and oncall response should be less than 20% of my time, together. At least half my time should go into engineering projects. If the level of toil is in excess of 50% of team activity then I would expect only percussive intervention to get the team out of this situation.
> core subjects, like container management (this happens...everywhere nowadays) or cluster management
Curiously, these are subjects which most Google SREs won't know much about. One team deals with all that stuff as a service so the rest of us can get on with something else.
What would I pick out as our core skill sets? Ignoring technology-specific details that won't apply anywhere else: troubleshooting a system that you don't understand (reverse-engineering it as you go), and non-abstract large system design.
Yes, there are many things I just can't share, and the only data point you can get in that area is my own opinion. How much value you place on that is up to you; I'm giving you the only thing I can. If that is of no value to you, you're free to discount it. I freely acknowledge that I can't prove you wrong. The only alternative I have is to say nothing at all, which is what I usually do. If you would prefer to have no input from people like me at all, by all means say so.
> It's obvious that discrimination has taken place that has affected a fellow engineer and human being.
I would like to make it clear that I am responding only to the comment I responded to, which raised a very specific question that I could answer. I do not feel that I have any basis to comment on the original article; please do not associate what I am saying with that.
You never go into an encounter with a new person or team being unsure of whether they're going to be difficult. You never have to avoid dealing with "that guy". You get to trust everybody that you meet.
It doesn't just improve overall quality, it makes it a better place to work. Good engineers are happier in this environment, and there is pretty near universal agreement from people who have experienced the results that this is a thing worth preserving.
There are plenty of things about the hiring process which get enthusiastic internal debate, criticism, and data-driven analysis. This is not one of them. This is a thing which we really like.
(Full disclosure: I have gone through the interview process twice, failed the first time, passed the second.)
What exactly is your objection to this post?
I do some work on this personally. I'm just not allowed to discuss the details at present.
(The reasons why I'm not allowed to discuss this are due to tedious bureaucracy, not anything interesting. I have tried to get that changed, but it would require more effort than I am willing to expend on a problem that will go away in time.)
But I would like to point out that of the two of us, only the one who doesn't know is suggesting that we have "terrible hiring figures" or a "terrible interview process".
We expect and accept a high false-negative rate. Our interview process is optimised for zero false-positives at the cost of many false-negatives. This is a deliberate choice. So yes, I would expect to see a significant rate of rejections of people who are clearly qualified.
The sort of people that we want to hire are likely to come back for another try anyway, and the long-term effect of this process seems to be doing what it was supposed to.
m.hotmail.com
watson.microsoft.com
Assorted *.msn.com domains
apps.skype.com
msftncsi.com
"Add spying domains to hosts file" is dishonest, at best. This appears to be a determined effort to break random services for the user which happen to be run by Microsoft. Hotmail, Skype and the NCSI detection are particularly inexcusable things to block under the guise of "destroy spying".
That's not the same thing as saying it was "a 0.005% tax rate". The issue is over what fraction of the non-US profits are taxable in Ireland, not over the tax rate.
I haven't seen an explanation of where that 0.005% number comes from, but I would be unsurprised to find similar forms of confusion in it.
When the law has become so restrictive that people would rather not engage in commerce at all, then it is broken. This isn't helping anybody.
Performance. gRPC is basically the most recent version of stubby, and at the kind of scale we use stubby, it achieves shockingly good rpc performance - call latency is orders of magnitude better than any form of http-rpc. This transforms the way you build applications, because you stop caring about the costs of rpcs, and start wanting to split your application into pieces separated by rpc boundaries so that you can run lots of copies of each piece.
I cannot sufficiently explain how critical this is to the way we build applications that scale.
It doesn't stop people from firing bad coworkers, and it appears to have no negative effects on employment.
While it might serve that purpose, it's primarily to make sure that middle-management takes reasonable steps to let people correct and thinks the decision through before acting. Nobody wants to work in a place where people get surprisingly or randomly fired, or where a manager is firing all the people they don't like. Having processes (mostly) prevents that sort of thing from happening.
I can't talk about the details, but you can assume that "static analysis" of the form being talked about here is something we've already done, and it's not enough to handle cases this complicated.
Let's go with "yes", as the most accurate answer. As soon as I or whoever is oncall has figured out what change was responsible, we can usually revert it quickly and easily. Usually, if I'm oncall and I have reason to even suspect a recent change might be the cause, I'll revert it and see if the problem goes away.
The difficulty becomes more apparent when you realise the sheer number of infrastructure changes being made every hour, some of which will be fixes to other outages, and some of which will be things you can't revert because they are of the form "that location has fallen offline; probably lost networking" or "we are now at peak time and there are more users online". So if your question is "can we just roll the whole world back one day" - no, too much has changed in that time.
> Perhaps your architecture wouldn't "compile" if the network traffic will go the wrong place, or if a rate limit is above the capacity something is expected to handle, or if the change would impact too many servers at once.
So in the first instance, I tend to like this sort of idea. However: we are already substantially ahead of the sort of things that you're thinking of.
Full static simulation of a system as complicated as all the components involved here is... well, I can sort of see how it could be done, but it would be a herculean effort; I don't think it would ever be good enough to catch cases like this the first time they happen. There are systems where this sort of thing can be done, but all the ones I can think of are much smaller in scope.
Tax rates on netflix vary, in both rate and whether they are collected at all.
Plenty of people make money from this: your ISP, the utility company that pulls cables under the road, the construction company that dug up the road to put them in, the owner of the site where your ISP's network equipment is located, and many of the websites that you interact with.
I believe that the source of the confusion here comes from trying to slice the world into "VC monetization" and "human value", as if these were different or opposed things. I have a different way to look at this space which reveals useful insights:
We see lots of technologies go past that people seem to be excited about, but which then fail in the market. It is currently popular to imply that this means "the market" is some alien thing which is not aligned with what people want. A more realistic view is that people have multiple levels of interest. We can order some of them from lowest to highest:
- willing to read an article - willing to write a comment on the article - willing to blog about the technology - willing to open their wallet - willing to pay the full cost of making it
What we see is that a lot of technologies can only reach levels 2 through 4: people are interested, but not interested enough to cover the cost of making it. By any reasonable standard, that means we shouldn't make the thing: its value to people is less than the value of the raw materials that went into it. "Failed in the market" is a way of summarising this decision, but it gets a lot of negative press because it hides all the details so people don't understand the value comparison being made here.
The neatest mnemonic to think about this is "money is the unit of caring: you can measure how much people care about a thing happening by measuring how much money they are willing to spend on it".
"VC monetization" fits neatly into this picture: VC want to know more or less immediately if people are going to reach interest level 4 or 5 on this scale. They do not want to burn time and money on things which can only reach level 3: those things never had a future. You cannot tell the difference without asking people to open their wallets.
It may involve criminal fraud, but it is also possible that it does not.
It's not my service, and I'm talking about the general case rather than this specific one, but that seems to me like it would be pretty complicated - you'd need some sort of review procedure to determine what material can and cannot be distributed, you'd need to somehow do this while preserving user privacy, and you'd need some engineering work to make all this possible.
A project of that scope could take weeks or months to complete, depending on the amount of data involved.