A proposal: how to hire junior engineers in the age of AI

It'll be interesting to see if we get better / worse at figuring out what effective LLM interaction should look like, since that seems prerequisite to this kind of evaluation. Right now there's so little data on whether e.g. casual conversational interaction works better than structured prompting, and so, so, so much superstition - I don't know if I'd really trust judges to have good instincts about what's effective rather than what resembles their own use patterns.

The one thing that will hold out forever or will at least be the last thing to fall is critical thinking. So I think you simply interview for that.

At my small company we interview all candidates, regardless of experience level, using the same set of questions during the primary round. They’re a combination of open ended scenario type questions and pure debate style questions. We try to design them such that when you’re answering them, it’s similar to when you’re on the job collaborating on a technical solution, or triaging an incident.

We provide limited information at first to force the candidate to immediately ask for more info, to see what kind of questions they ask and what kind of information they care about. When they start to provide the solution, we like to throw in alterations or wrenches to see how they pivot. It’s very much “turn based”.

If you’re taking it seriously, it’s very obvious when candidates are either solving it for the first time, are recalling true experience, or are simply reciting information like they studied it. That’s why many of our questions work for many levels of engineering.

Additionally, because there is so much back and forth, you get to also “vibe-check” the candidate engineering wise. In our experience most candidates have enjoyed the process, so we feel like we uncover a lot about personalities here as well.

Unfortunately, the heel to this approach is that you pretty much have to conduct these in person. Because an LLM hypothetically would do well in this style of interview. We’re hybrid one day a week so it works for us, but it doesn’t exactly scale.

I would like to preemptively acknowledge that this is not the center of the article, but:

> Did you know IQ tests are effectively illegal for hiring in the US because they disproportionately screen out minorities? I learn things when I research these posts.

The provided link discusses how this is itself a common misunderstanding and they're not illegal if they have to do with the job, but you have to justify that they do have to do with the job.

> Using fully automated scoring would also be against the law in some places: New York City requires annual bias audits of automated hiring tools, and you don’t want that heat.

The elision of the gap between "[it's illegal to do this]" and "[it's illegal to do this without checking for bias]" is interesting.

Why hire juniors? There seem to be plenty of unemployed mid- and senior-level people and the market is so poor that some of them (e.g. me) will work for cheap.

Here is the problem with almost all of these attempts: there is no need to be a “pop quiz” where you have the answer, and the candidate is made to guess what you feel is right (even if you're wrong) and is walking in blind. This literally never happens in the real world - and if it does it means you’re getting fired. It’s just sadism you are pushing to make the in group people feel smarty pants and superior.

Tell the candidate up front what you want to talk about. Do it on a subject you dont know either so you are on even footing. Let them do research before the “quiz”. Since tokens are pretty cheap, ask them to build a working application and then discuss it with them.

This really isn't hard if you get off your imaginary high horse, and show candidates some basic respect.