followed by
> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.
is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?
https://about.gitlab.com/jobs/production-engineer/
-------------------
Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be conducting the interview, find her/his job title on our team page.
Qualified applicants receive a short questionnaire and coding exercise from our Global Recruiters
The review process for this role can take a little longer than usual but if in doubt, check in with the Global recruiter at any point.
Selected candidates will be invited to schedule a 45min screening call with our Global Recruiters
Next, candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead
Candidates will then be invited to schedule a 45 minute technical interview with a Production Engineer
Candidates will be invited to schedule a third interview with our VP of Engineering
Finally, candidates will have a 50 minute interview with our CEO
Successful candidates will subsequently be made an offer via email
But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen.
So maybe the process really isn't that great.
This is an issue during hiring when companies have minimal Ops background. They (usually) attempt to interview the candidate as they would a normal dev position. This has 2 serious problems. 1) Real Ops candidates are going to seem lacklustre compared to web devs when giving web dev coding tasks and inteview questions. 2) You probably aren't going to ask them anything actually relevant to their ability to do Ops well because you just don't know.
This can usually be solved by hiring an experienced SRE manager. However those people are expensive and hard to find.
The next best thing is to have a gun SRE in your personal network that you can convince to join your team and help you build it out.
If you can't achieve that for whatever reason then you really just have to hire based on experience and suck up the cost of hiring an ex-Google/Facebook/Netflix/Twitter SRE that you can be reasonably confident will be good.
That is still hard though because those sorts of people know there is a ton of shit shoveling to be done if you are the first real SRE on board.. you have to clean up the mess left in the wake of "devs doing ops" which is not Devops btw. Devops is what is done by SREs. "devs doing ops" is how this sort of thing happens.
/rant
> Real Ops candidates are going to seem lacklustre compared to web devs when giving web dev coding tasks and inteview questions.
This drives me insane. I think 50% of my comment history this year is about how not to botch an interview.
> You probably aren't going to ask them anything actually relevant to their ability to do Ops well because you just don't know.
Yes. You won't be able to quiz that person. You can take a thoughtful look at experience and talk about that, though.
> hire based on experience
Yup. One of the spook factors at any company is when I see it only hiring down, i.e., lower than the aggregate skill level of its current team. If the VP of engineering knows more about each concentration area than the people he's hiring, there's a storm a-brewin'.
The reason for my rejection? I had an ongoing side project and development/consulting gig which had been paying the bills for a few years which they thought I 'misrepresented'. I guess they assumed I was still working there (I was and would continue to do consulting and development on my own time), but the biggest LOL of the whole shitty process was their reasoning: "We are very risk-averse in our hiring." No shit.
I guess they weren't risk-averse enough in their operations though, so I'm glad I didn't get through. It sounds like it would have been an uphill battle all the way to make changes to keep things sane.
And having to crowdsource your move from cloud to your own hardware? Listen to your people, they already had good ideas, all of which were documented in your open issue tracker.
I think they were risk averse which is the problem except they were judging against the wrong candidate profile. The risk adverseness meant they never hired anyone who diverged from their incorrect profile.
Yes, go right ahead and filter out some (disclaimer before the rant: some, not all) of the best talent. The kind of potential employee that gets rejected due to perceived personality problems is exactly the kind of person who would tell management to shove a stick up their ass for demanding a 2 week deadline for a project requiring 3 months to execute properly.
Maybe if GitLab had hired the best talent, instead of the best "behavioral/cultural fit", at least one of their 5 backup systems would have been functional. Many people who are perfectionists in their craft, who would never have allowed this kind of failure to take place under their watch, come with abrasive personalities. If you only hire those who are submissive during the interviewing process, you will get exactly what you chose - people with no backbone to push back against unreasonable business expectations.
Case in point: would you want to hire me based on this comment of mine? Hell no! You're going to steer clear of me and give me an instant fail during a "behavioral interview", because you can't look past my belligerence to understand that there is value in having employees who obsess over the little things like having systems that do what the fuck they're supposed to do, rather than being able to give a conformant first impression full of social prowess. "Whoa, he used the word 'fuck' to hammer home his point; definitely avoid hiring this guy!"
tldr; Sometimes, people who are "talented" or "skilled" get to that point by being obsessive freaks who sit at home in the dark all night hacking away at stuff, with no social lives. The result can be someone who knows what they are doing because they invest all their personal free time into the domain, but consequently has absolutely no social skills to put on display.
shorter tldr; Businesses focus on the liability of a person without considering the potential.
Moreover, many with abrasive personalities are the ones who demand and take for granted excessive overtime the moment they are in senior or lead position. "I am too tired and it is late at night lets do it tomorrow" is not an option to abrasive personality.
As a rule, obsessive freak with no social skills you describe wont be able to effectively reorganize the team nor create process nor (if in extreme version) work within process someone else created. He is more likely to end up in endless quarrels about petty differences in coding style.
Ability to create, fight for and enforce sane process is not the same as being abrasive.
He is however absolutely fine when you have isolated position.
That said, depending on their interview process they may be selecting out candidates who even diplomatically disagree about the existing ops processes which results in less than stellar hires.
> Did the person who set up that S3 job simply write a
> script or something and just go "yep, it's done" and
> walk away?
My small business uses a global network of cheap servers to provide low latency to customers, so we have encountered significant lengthy failures of disk or network once every few months, but hey, we can be damn sure our backups and deploy scripts work because we're forced to restore them all too often.
Is it even possible to pay for the hosted GitLab.com instance?
The only for pay versions are self hosted EE and independent cloud hosting.