Hacker News Reader

Comment by tptacek | Hacker News Reader

I have. In fact: it became sort of a norm. Want to talk more about it?

I'm sure there's a definition of "weak candidate" that perfectly captures "uncoachable". But my definition, and I think it's the common-sense definition, is that a "weak candidate" is one who gets as much negative feedback as positive (Joel Spolsky famously advocates for "if you have to ask, the answer is NO HIRE"). Most of the people we hired at Matasano for several years fell into that category, and most of them wound up outpacing the strong candidates (none of them did badly).

View on HN

Are there any other companies that you know of that do what you did at Matasano (offer post-hire specialist-type training to non-specialists/generalists is how I understood your posts about it)? The moneyball/trainingball idea seems like such an obvious way to get the people a company needs that one would think it would be somewhat common by now.

For all the alleged shortages, can it really be that all these years later companies are still unwilling to gamble on training because they feel like people will leave too quickly?

Companies have too incomplete a notion of the value of training and a positive environment to retention and productivity. People really do think skill is innate and people 'are who they are' more or less.

It's probably not true in 80% of cases, is true in 20% of cases, and the 10% of cases where people got burned by a poor candidate who couldn't improve are the ones they remember and say "never again;" likewise the 10% of cases where they are amazed by a highly skilled candidate and think all of them should be that way. They're missing the majority effect that impacts the productivity of 80% of their workforce.

This is why it's crucial to have an innate sense of human variation and psychology, a la W. Edwards Deming. So, in that sense, it's not the employees that need the training, but the managers.

I know lots of the big name consulting companies in London at least used to do it. A friend of mine with a physics degree from Oxford got job offers for all kinds of things on the assumption that if you can get a physics degree from Oxford, you can probably learn other stuff as well. He ended up doing some sort of forensic accounting.

A friend of mine in Australia became a management consultant with a PhD in physiotherapy, they essentially were saying a PhD in anything is a good base.

From my perspective, I see that several other skill-based professions assume novice hires, and emphasize a some sort of on-the-job training as part of career progression.

In software dev, as much as we have stack overflow and such where we can figure out trivia of the tools we have to work with, I guess we'll come to the realization that some kind of formalized training post-hire will be useful, especially as development becomes more specialized or domain specific.

I know you've talked about your case as a counterexample, but I think it's because it's not coding ability you were finding hard to get, but security and pen-testing knowledge and experience.

To my mind, that's domain knowledge, and domain knowledge can definitely be taught, and much faster than how to write good code in good time.

I think sending some pen-testing books to a bored and under-exercised VB programmer may well get good results; but put the same guy in front of a problem that is almost completely abstract, and if he's able to understand the solution you present to him and create code (in any language he likes; we don't care about specific language experience either) that implements the solution, then he'd get hired at my company too. Because we don't test for domain knowledge; we don't expect it. We test for ability to communicate, understand and turn concrete implementation ideas into code.

It's a programming job. Matasano wasn't a netpen firm; every project involved us digesting and processing some giant blob of other people's code. Everyone on the team had to write tooling and, more importantly, quickly understand other people's code well enough to outguess the authors.

I don't accept the argument that we were hiring for an easier job than "programmer"; I think we were looking for a subset of the population that ordinary software development teams look for.

The bored and under-exercised VB programmer, by the way, helped found the NCC Cryptography practice, and moved on to one of the greatest crypto software development jobs in the industry. It's not my place to say exactly which job, but if you make a mental list of the 5 coolest crypto software development jobs that might exist, it's probably one of those five.

The problem is everyone else runs hiring processes that assume the bored VB programmers are just VB programmers. They are wrong. Joel Spolsky was also wrong about this (despite the other nice things he said about VB).

> I think we were looking for a subset of the population that ordinary software development teams look for.

I think that's actually why your experience might be less representative, but I'm open to being corrected.

Security is generally considered harder than regular development, at least to the extent that it would typically be hard to get a security job without security experience.

From what I understand, your model was to take programmers with no security experience and train them on security. The effectiveness of this doesn't surprise me at all, because it's fundamentally a different problem than taking someone who's bad at programming and teaching them to program better.

I have absolutely no problem with hiring someone who's a good VB programmer and teaching them to do web programming. What people generally dispute is whether you can hire someone who's a bad programmer in general and "train" them to be a good programmer.

> I have absolutely no problem with hiring someone who's a good VB programmer and teaching them to do web programming. What people generally dispute is whether you can hire someone who's a bad programmer in general and "train" them to be a good programmer.

How do you know if a programmer who is in a field you are not familiar with is good or not?

You can ask someone to show you some code they've written and teach you about it. Concepts transcend language and field: if you understand what you're doing and why you're doing it when writing a VB program, I have high confidence that with a bit of time you'll also learn to understand how and why to write other kinds of programs effectively as well.

Why does it have to be this complicated? Explain some of their existing code? Why not just give them a representative programming problem that your team has already solved, and ask them to code up their ideal solution? That's called "work sample hiring", and it's the gold standard for candidate evaluation.

It boggles my mind that everyone doesn't do this.

Frankly, your statements are extremely conflicting. Pure work sample hiring is in direct contradiction with hiring trainable people.

I do not expect a VB programmer to be able to code up a single-page React app during an interview, even though building a React app would be a representative problem. Do you?

So which one is it? Hire exclusively through work samples, or hire trainable people? I mean this seriously: how do you reconcile these two? You're a strong proponent of work sample hiring but also of training—do you expect people to come in and do a security audit for their interview, even when one of your hiring goals is to hire people with no security experience?

I would personally much rather hire someone who is a strong overall programmer (but might very well not be able to code a JavaScript app in an interview) over someone who knows a little JavaScript (so they can do a simple problem) but has relatively shallow abilities.

> That's called "work sample hiring", and it's the gold standard for candidate evaluation.

Having a discussion about someone's existing work is, in fact, another gold standard of hiring and is quite similar to having someone do the work on the job. It's not quite as good as having someone execute work, but it's the next best thing—especially when you're trying to hire people who don't have direct domain experience.

Hire through work samples. I think you're going to be surprised at who does well with them.

I think you're neglecting to remind folks that Matasano would give people a crypto textbook to study from first.

So if a VB programmer were applying for a React job, you'd say, "here's a React book; we're going to have you code a React component during your interview. Let us know when you're ready to schedule."

That's true, but how much credit do I want to give myself for extremely obvious hiring process flourishes? Yes: give candidates study material!

So how is your process really any better than Google's? They will also tell you exactly what to study and give you reference links + time to study it.

Google does not hire with a work-sample test. tptacek's claimed process is therefore easy to distinguish from Google's. Maybe better, maybe worse.

Google does allocate non-trivial time on training, though, both on the candidate's time before the interview (as you note), and on paid time after hiring.

What a low effort reply. Honestly, I expected better from you. This is pure argument from authority without any attempt to confront the fact that there is in fact a very real contradiction between hiring through work samples and investing in training.

For the record, I have done work sample hiring. It's definitely my preferred strategy, but it's also far from a panacea.

But it's also equally true that if you give someone a work sample to build a React app and their entire experience is in backend programming, they will do not very well. Would you really be surprised if someone fails to produce good work in an environment, language, and domain completely foreign to them?

Taken to the extreme, some companies traditionally hired engineers who had absolutely no programming experience and trained them to program on the job. Would your advice to them also be to "hire through work samples (of programming)?" How would candidates who, by definition, need training to even do the work be able to produce a work sample?

Your whole "I know best, so I don't have to provide logic or evidence" schtick is really getting old.

There's no contradiction. Set the bar where you need it to be, but make the bar about ability and aptitude, not about people talking their way through an interview.

If your bar needs to be super hard, set the bar there. If you need people to understand in detail when and how to use shoudComponentUpdate and how to write a pure functional component, do that. I'm not arguing that React shops should hire people who suck at React.

I think you think I'm saying something I'm not saying.

> I'm not arguing that React shops should hire people who suck at React.

I guess I misinterpreted that you were a proponent of training people.

Yes, if you want to exclusively hire people who are already good at React then of course work samples are an effective technique. I personally think the majority of good hires are not already familiar with my particular stack.

> Set the bar where you need it to be, but make the bar about ability and aptitude, not about people talking their way through an interview.

I think you might have misinterpreted my position as well. When I ask people to talk their way through their existing projects, it means they literally bring some code with them and we spend the interview working through it and them teaching me about the decisions they made in building it. You should try it some time.

Also, it's easy enough for a VB programmer to get started with React, he/she can quickly teach himself/herself some js and then follow a tutorial online, and after some point build a toy application with React. And that can be the work sample. It won't be a production ready app, but it will be fine for a junior React guy.

This perspective depends on mentorship quality being constant across all of these programmers.

Then mentorship and teaching need to be taught also.

In the startup world? I'm not holding my breath.

Yep, you'll have to do more work than just breathing. :-P

Oh, I agree about VB; once upon a time, a long time ago, I too was a VB programmer (before I discovered Turbo Pascal and then became aware that I found the distinction between Let and Set disturbing; I ended up loving Pascal so much I spent 5 years at Borland on the Delphi compiler).

While at Borland, another engineer (Allen Bauer) shared his personal theory, that some engineers were great at reading code, more were great at writing code, and not so many were great at both. Sounds like your job required people that were very good at reading code. It intuitively feels very testable.

Reply on news.ycombinator.com