Hacker News Reader

> Here's an entirely novel concept for evaluating if somebody can write code that solves actual problems. We could ask people to, IDK, write code that solves actual problems. They're there for a day. A good coder can solve some pretty interesting problems in a day.

You mean solve an actual business problem that the interviewing company has? The ramp-up time to be productive in any real business that I've been associated with is significantly greater than the time available in a single interview (usually an hour or so). This is why toy problems are used.

Another issue is that it's not really fair to ask an interviewee to solve your company's problems for free. In fact, without some sort of contract, the interviewee technically owns her own work, so even if she manages to solve a real problem, the company really shouldn't use it.

(BTW, 5K interviews at one hour per interview is 625 eight-hour workdays. Assuming 50 work weeks of five days each in a year, that's 2.5 work years devoted to nothing but interviewing in your career. Yes?)

Not an actual problem the company has, but a pared down version of a problem the company had.

Easier said than done. You probably end up with something that's essentially a toy problem anyway.

Thankfully not 2.5 work years. See above for explanation. Probably around 0.5 work years, smeared over almost a decade.

And no, not an actual business problem, a constructed problem. But in the same domain. (See other comment above)

> I've seen a few interviews in my time. On the order of 5k of them)

That's actually pretty cool. May I ask how?

> Because "knows DP" is pretty much the only bit of information you get

Yes if the candidate meets with only one interviewer or if every interviewer asks only DP. I hope most companies don't interview that way - that would truly be a broken interview process. Obviously you ask about other stuff in the other interviews.

> The interviewer picked them up somewhere, is now familiar with them,

Lol, isn't that one reason every interview question is asked? I've never asked a question whose answer I wasn't familiar with. I know how you meant that though.

There are plenty of companies that don't do whiteboard problems or ask DP. Here's a few: https://github.com/poteto/hiring-without-whiteboards

> We could ask people to, IDK, write code that solves actual problems

It's pretty hard to get someone solving actual problems in a single day. Do you have some examples of how this might be done?

On a lot of companies’ interview loops, if one interviewer says “no hire,” it’s an overall “no hire.” If that person is asking DP, and you don’t know it, then you will fail the whole interview.

> It's pretty hard to get someone solving actual problems in a single day. Do you have some examples of how this might be done?

Solve a small problem - of which millions can be found in issue trackers.

Here is an example I just read on Github:

Somebody reports that reading a JSON file results in a value that clearly is an integer being misidentified as "Not an integer". The source code is a parser written in C and the code is quite readable.

Give the candidate an hour to try to figure out what is happening (obviously, the language and context must be a match for the job).

Important - I think: It would be good if the problem is unsolved and the interviewer him-/herself does not know the answer. Also: Don't send them into a room alone, watch them. I know that adds stress, but learn how to lower it, make it clear you too don't know the answer. If the candidate really can't cope with someone watching, well, okay, let them try alone, I just think if the atmosphere is right this is valuable. There is no right way to approach solving such a problem, but without and not meant for judgment, I think this is just interesting to see how different people approach problems. Of course, if the interviewer has string opinions about how it should be done that is a bad method and I myself would not like to be interviewee in that case.

> It would be good if the problem is unsolved and the interviewer him-/herself does not know the answer.

I think one-use questions like this must be are bad. It makes it harder to compare candidates, which is what you really want in an interview.

you can also think about them another way, namely: How long it took somebody else in your team to solve the same problem

Making allowances for interview conditions, namely high stress levels, etc ...

Not perfect but an idea worth exploring.

I speak as a former smug interviewer that while didn't ask DP questions, did have a set of favourite questions

> How long it took somebody else in your team to solve the same problem

I don't think that is useful. Variance is too great, even for a constant single person.

I think that anything that attempts to cast this issue as some objectifiable measurement problem is on the wrong track. Measurements work - on a statistical scale. Even then its hit or miss depending on how good the chosen values to be measured are. Just like public health issues. As soon as you try those approaches that work just fine on a large scale on individual problems you are toast. The results of those individual decisions can be measured using statistical methods, but don't try them on individuals.

> How long it took somebody else in your team to solve the same problem

I don't think that is a good measure. I know many developers who want to solve everything fast because fast is productive in their eyes. The problem is they end up having a bug or three and spend some back and forth time with QA. So, now they didn't actually solve the problem fast and they wasted QAs time. I prefer to get a more holistic understanding of the code before fixing the issue and not get any bugs back from QA. In retro, when the QA tells the team how many bugs there were, I am proud when I'm not a part of that count.

I disagree. If we're considering the debug/issue fix approach (which I quite like, because it displays code literacy and clarity of explanation in a context like that one would find in a typical workday), if I use the same task/tasks for each candidate, I'm implicitly (and tremendously) biasing myself toward a universe shaped by the tasks I chose. What I'm really looking for is a more universally distinguishably trait of lucidity and perspicuity (coupled with sufficient technical skill, which, again, bias and variance from your tests' structure is always an issue in accurately scoring), in any case.

I think this would be great. It would give me, the interviewee, a chance to feel out how it is to work with the person interviewing me. Will they crap on every solution I propose? Or will we have some back and forth and find a solution that suits us both?

Also, you don't have the added stress of writing all the code from scratch, having to whiteboard recall boilerplate that a good IDE handles for you.

See another comment above for explanation how I got to that count. (In essence: reviewing lots of interviews)

> Do you have some examples of how this might be done?

I do :)

We did it at a previous company. We brought in several candidates at the same time, declared them a team, and gave them a half-finished video game (since it was a games company). With the goal of turning it into a better game by the end of the day. We also embedded one employee with that team for the whole day, and other reviewers would drift in and out, observe how people worked together.

The key components there: * Some groundwork is already done, you pick up an existing project. (Also tests a really important skill - reading and understanding code) * You work in a team - more output, and tests for your ability to work in a team. * You have a source of experience that can answer all your questions around tooling and libraries, and you can lean on them to get some of the hairier stuff done. (Also tests if you're willing to let go of the desire to show off to instead make things happen for the team)

And yes, we'd hire the whole "team" of around three people if they were all good, and we'd pass on everybody if they were all bad, and everything in between.

It worked pretty well, gave really good results, and both interviewers and interviewees enjoyed the process. It's also something you can't really learn by rote in advance, because you do what you'd be doing every day anyways.

I won't lie, it sounds like an interesting process and I wouldn't mind participating once if I'm qualified. I wonder if you could answer some more questions about it. Sorry, I know it's asking a lot, I understand if you don't answer:

1. What if you want to hire a primarily Python-writing programmer to write Ruby or C#? They're not going to be super-productive on day 1 - maybe they'll struggle to run gem install or whatever. Even though they'd have made a perfectly fine employee given a week's worth of ramp-up, they may end up flunking the interview (Maybe it's a less of a problem in games because everything is C++? I don't know). How do you calibrate these performance variances due to lack of familiarity with the specific tech?

2. How do you ensure that the odd toxic candidate doesn't ruin the team dynamic? It's hardly fair to reject everyone if they'd have done well without the one asshole.

3. It seems odd to expect people to collaborate in a situation where (for all they know) they are actually competing for the same position(s). What if you don't actually have enough open positions to hire everyone?

4. How do you track contributions by individual candidates? Commits? Observations by the embedded employee?

5. Do you tell candidates upfront they're being graded for co-operation as well as productivity? (and is that how you're actually grading?)

1) Games are somewhat easier, because C++ is pretty common. It'd also work for larger companies, because you can group candidates by languages. I don't have a good answer for companies that can't do that.

2) That's why there's an observer with the team the whole time. If there were a truly toxic person, they'd have the power to pull them. I didn't see it happen (likely because everybody is on their best behavior during interviews), but I did see a few large egos - and they were pretty much always shunted into a corner of the project.

3) You obviously shouldn't do that if you can't afford to hire everybody. Which means it's likely not the right process for smaller companies.

4) Both by the embedded observer, and by looking at commit history. But it rarely comes down to that - what you want is "smart, and gets stuff done", and that fairly clearly separates out without counting commits. Or maybe we just got lucky.

5) All we told candidates is that we'd expect them to have a working game by end of day. Grading was less by cooperation, and more by how much of a positive contribution people were. We e.g. had candidates that up-front said "look, I work best by thinking quietly, and I'm an expert on X - would you mind if we carved out a subproject around that, and we reintegrate from time to time". And we did hire them - because a person who knows what their style is, and how to collaborate with a team with a different style, is a huge win.

I was in a team with a very talented competitive programmer guy. He came up with solutions for some pretty tough problems I couldn't wrap my head around.

Fantastic programmer right?

The other 99% except hardcore algorithmic optimizations, like structuring code, creating simple code, naming variables, splitting code into classes/functions and refactoring? No clue about any of that. All he produced was "read-only" and had many glaring problems.

It wasn't only during competitions either, he used the same style for assignments. Turns out he got shafted by the assistants and then he copied answers from another student (yes he got caught).

At my first company, there was this one file in the solution that we called DaVinci.cpp because it was over 20000 lines long and written by one developer who had long since retired. The thing is, the code was complex and worked so no one ever refactored it.

> The other 99% except hardcore algorithmic optimizations, like structuring code, creating simple code, naming variables, splitting code into classes/functions and refactoring? No clue about any of that.

So...you're saying a student wrote code like...a student? Color me shocked.

You can coach talented programmers to write neat code, design classes, and name their variables right. It's part of turning a junior engineer into a senior engineer. It's far harder to teach someone how to solve programming problems.

> Here's an entirely novel concept for evaluating if somebody can write code that solves actual problems. We could ask people to, IDK, write code that solves actual problems. They're there for a day. A good coder can solve some pretty interesting problems in a day.

My last interview was structured like this. Remote. Able to work on it outside of work hours (so no missed office days). Tracked a shared repo. Was paid a retainer up front for n hours worked on the task at an agreed-upon hourly rate.

Needless to say it was a great experience.

I've also had interviews in the past where the interviewer didn't have the correct answer to a question they asked me, and just got confused when I tried to explain my answer more fully. In the end they said I was incorrect, due to their incorrect answer (that they must have taken down from another source). I didn't get that job. Probably for the best.

There are also some people who just shouldn't be interviewing others. They just don't have the emotional intelligence to manage the situation.

> I've seen a few interviews in my time. On the order of 5k of them)

Sorry, your "5k" doesn't pass the smell test, unless you can give some convincing reason why 20 years of one interview per day would be believable.

So, I'm calling BS. You just seem bitter about interviews that you don't like and are trying to con people into giving your opinion more weight than that of others.

Besides, if other companies are doing interviewing "wrong" as you think, then just do it "right" in yours and have a competitive edge by hiring all those brilliant coders that other companies are too stupid recognizing. That would be a reason for you to be joyful instead of bitter, no?

If you interview at a traditional tech company, you go through 5-8 interviews in a day. If somebody were to review, say, around 4 of these packets every week, they see about 20-30 interviews every week. That's ~1000-1500 a year. The 5K mark comes faster than you think.

And if you work at a company that size, changing the interview process is a bit more work than "oh, we just won't do that".

Not everybody works at a small startup.