They had this massive programmer test. It was something like 40 pages long, hardcopy, double-sided. With answers needing to be written on the test generally in essay format; no computer allowed (and so no real coding involved; it was all about concepts and not about syntax). It covered everything; math, physics, logic, sound, calculus, hardware, etc. The thing was so long that we estimated it’d take a really strong applicant about four hours to complete the whole thing if they somehow were experts on all the topics simultaneously.
We informed the applicants about the test’s length upfront; that it was designed to be a 4 hour test, and that they were only going to get to spend one hour and that we absolutely didn’t expect them to know everything (or even most things) on it. That the point was for them to pick the questions they were most interested in and to only work on those ones.
And then we never formally scored the test.
Instead, we basically used it as a conversation-starter for the interview which immediately followed the test, using it as a springboard to talk about their understanding and skills and why they had selected to do one section over another, or about particular questions which they’d provided interesting (or problematic) answers to.
The “here are a whole lot of questions, answer whichever ones you want, you can’t answer them all and we’re not scoring you on how many you get right” approach always felt kind of ‘fair’ to me in a way that pure interviews or coding questions never did, for me.
(caveat: I wasn’t hired using that test; I had been hired as a junior long before, based on my portfolio and a short interview, and the folks who did get hired via taking the test may not have liked it as much as I did. But I loved that it didn’t hang everything on a single gimmick code question the way I’d seen from a whole lot of other companies)
That's always the catch. The founders and initial employees of a company never had to go through the crappy hiring process that the company adds later as it gets bigger. And then we claim that the tests make tech a "meritocracy" and "objective", but those at the top are exempt from the tests.
I recently tried interviewing for a growing small company, who nullified the initial good impression with a multiple choice test about certain technologies they were using, administered over the web by some test company.
The best questions were "which of these four functions or classes do you use to do $common task with $framework", testing documentation speed reading skills rather than actual coding skills; other questions were ambiguous, debatable or completely wrong.
Sounds like testgorrilla. I swear they have a poor markov-chain based AI generating the questions.
I just thought it was a really neat and kind idea to let the applicant choose what question or questions they wanted to discuss and in what order, instead of forcing them to answer a single pre-defined high-pressure coding question and hang the whole interview on that.
The least I would do is politely decline such an interview. I would be sorely tempted to give them the choice between letting me use a keyboard or hearing from an ADA lawyer.
I assure you that I’m not a monster and I would really appreciate it if you’d assume positive intent in such replies.
I don't think you're a bad person for it, and ok, good to know you're willing to make accomodations, but my reaction to being asked to do this would be sharply negative. The keyboard is inherent to the profession.
Assuming you read the essay it is impossible to not judge the applicant on their penmanship. A difficult skill with absolutely no bearing on the qualification for the job.
I urge anyone reading this to not offer a 40 page handwritten essay to software developers in the first place. It's an awful idea.
There was obviously no “sadistic 40 page handwritten essay”; individuals engaged with whatever individual questions they wanted to, and would normally write their answers into the bits of blank space left in between the questions. I’d estimate that a normal applicant who answered ten to twenty questions would normally write substantially less than what would constitute a single page of text in total.
Again, I’m not a monster. I’d really appreciate it if you’d assume positive intent and ask for clarification rather than jumping to the worst possible interpretation you can invent and then stridently declaring that you’d take legal action on the basis of the fictional “sadistic” thing you’ve chosen to believe I was making people do.
In most of the world a "no thanks I pass" will suffice
Basically, don't test for something which isn't a job criterion and excludes people based on ability. An absurd example would be asking programmers to lift a 20kg sack of flour onto their shoulder and carry it across a room: this is a legal and normal thing to ask if the job involves lifting things, do it for a desk job and you'll get sued eventually.
Whiteboarding is an actual part of the job at many companies, I just let them know that words will probably be illegible and muddle through it.
No software job calls for handwritten pages of text, it clearly discriminates on ability and it shouldn't be a part of hiring. Simple as.
Then the first hit on google was a paper from 2018 about how to hire based on handwriting.
Yes. You're at worst the average, at best part of the majority.
Almost everyone sucks at interviews and most job hiring processes are incredibly degrading. Feedback is terrible (includes no feedback). Hiring managers are all looking for different things, either subtly or wildly. Onus to get good at interviews is entirely on the individual, not on the company to make the process less of a hellscape. Hiring managers are largely biased, untrained, as are most interview processes. Many aspects of interview processes aren't even grounded in reality.
But almost nobody is talking about this as it's just something you "get done" and "bite through" once every N years. Plenty of perfectly capable people fall to the wayside and are told they are the problem. Or worse, you get a bunch of (LinkedIn) thought leaders trying to gaslight everyone into thinking corporate holds the truth and the remainder of us are peasants who simply don't understand.
It's never been as easy to interview with multiple companies as it is now. It's a numbers game, and I would (and have) had interviews purely for practice, or our of curiosity about some companies tech stack. If you make it all about the numbers, suddenly you feel less bad about blowing an interview, or filtering bad ones out early in the process.
> not on the company to make the process less of a hellscape.
I mean, they do suffer in the long term. Companies after all, are the products of their hiring processes.
When I conduct interviews I simply have conversations to explore subjects as if I were talking about an issue with a coworker. I tell them directly that my goal as an interviewer is to give them an opportunity to showcase something they know so I can say good things about them. I give the interviewee a choice of subjects and I let them change the subject. If the interviewee doesn't know something, I explain it. We work together, as partners, to find their strongest area -- and then we pursue it until we reach a limit.
I also tell them that I will keep exploring the subject in question until we run into things they don't know, or that I don't know. I reassure them that I expect to reach a point in our conversation where we just say "I don't know" and that it's totally OK and expected.
After the interview, I assess the communication. What did they know? What did they ask? And perhaps most important: What did they learn? Were they defensive about the boundaries of their knowledge, and how did they deal with that?
I find I usually have a very detailed understanding about the interviewee's areas of competence as well as their communication abilities. A strong communicator and learner is far more valuable than someone who can recite a few technical details.
I think quiz or challenge style interviews are easier. The interviewee doesn't need to be skilled and doesn't need to risk exposing their own ignorance on a subject - often nontechnical recruiters administer these. They can be used to maintain an illusion of superiority on part of the interviewer and company. It's perhaps a similar defensive dynamic as which keeps peer teams from openly interacting and problem-solving, when teams challenge one another with problems rather than opening up about what they know and what they want to know.
You could call it a free-form interactive essay interview. Pick any subject and converse on it however you like. If it's not going well, pick any other subject.
And there is a reason most _interviewee_ guides tell you to always be honest and say "I don't know". That's to kickstart the discussion/next question as soon as possible rather than unnecessarily get stuck in the "bullshitting" phase that helps no one. It's rather common to encourage this, not a rare thing.
It appears your apparent confusion is due to this assumption. It's the opposite of the process I described above.
I meant exactly what I said. The candidate can pick any topic they like. I trust the candidate to pick a topic to showcase their own skills -- and if they don't, that's another useful datapoint for the hiring process.
There's a huge difference between using a predefined process vs letting the candidate drive. It guarantees we won't waste time talking about subjects the candidate isn't familiar with, and that we will focus on their strengths. Figuring out a candidate's strengths is the entire point of interviewing!
It's also relevant to the "fairness" issue mentioned above, for the same reason.
Can it be said that refusing to hire women, for example, has almost no downside? In my opinion, this is a kind of ethical issue, because the "false negatives" are not just random. Audition-style coding quiz interviews tend to systematically discriminate against people who don't interview often, who haven't recently studied CS in college, who don't have time to "train" for interviews, or who tend to have situational anxiety in job interviews (which is fairly common, and not an indicator of on-the-job anxiety). A study sponsored by Microsoft itself showed that these interviews test for anxiety rather than skill. https://news.ncsu.edu/2020/07/tech-job-interviews-anxiety/
Remember that the author said, "A few weeks later I’m about to go in for my interview and I’m so nervous I can barely think straight." These screening tests were all-or-nothing, so if he flubbed the question, he was eliminated. You might respond, "Well, he made it through the onsite interviews." Yes, that's precisely why using coding tests as screening is so pernicious, because a short screening test is not a good indicator of what someone can do given more time and under better working conditions.
This gets repeated over and over, makes sense rationally on a superficial level, but falls apart once you think deeper into it without substantiating it with evidence.
We don't work on a binary false positive / true positive scale. We don't know the efficiency of each variable going into the equation with which we filter false positives. Layering proxy on proxy doesn't guarantee an increase in the ratio of true positives to false positives. All these variables are costly and subject to diminishing returns. At the end, the workplace is still arguably a far bigger deciding factor in the outcome than who you hire.
If you're a big corp like MS getting at least a few dozen candidates after initial automated filters, that's one thing. If you're a relative no-name with a handful of candidates and a fat management layer pulling these shenanigans, it's about time we get real and call them out.
I really don’t think this is true. Are there any successful programming companies who just hire whoever for development work? All the top-tier ones have settled on being extremely selective. Of course, that could be coincidence or bias… but this practice is fairly expensive, so there would be nontrivial pressure against it. Furthermore, I think a lot of people can attest to working with both very good and bad developers on the same team - so even inside a workplace you see a range of individual productivity.
"Write a function that, when given an array of length n + 1 containing integers 1 through n, find the duplicate integer in an array.
You can use this equation: sum(1 to n) = n(n + 1) / 2"
Basically, if your solution is made trivial by someone knowing a generic formula or equation, just give it to them.
Not giving them the formula doesn't give you any additional information.
We can debate whether or not any of the above is useful information (I think it is), but not giving them the formula definitely gives you more information than just giving it to them.
They are not, and no company cares about fairness
That they can get away with such a bad measurement though just shows that there’s a large gap between the costs per and profits per developer.
Sure interviews are unfair (there are many problems like the one described in TFA, as well as people who just practice the skill), but are there better options?
I’ve worked with people who needed time to think, but at the end of that time came up with better solutions than the people who were quicker.
High pressure leetcode tests select for a specific type of thinker who may or may not be good at solving the problems you’re actually hiring them for.
> but are there better options
I’ve had success pair programming together on a problem you both don’t know the immediate answer to. I’ve also had success with allowing candidates to choose take home assignments.
I’ve also had success doing what every other other industry does: trust their resume, ask them to talk me through projects, and talk to references. Then fire them quickly if it turns out they are one of those dreaded fake programmers that everyone is worried about.
This is not different to what is being described in TFA -- you do discuss with the interview in order to reach the solution, and as many have said in these comments already, if you just lash out the solution without revealing your thinking process, many interviewers get disappointed and may throw you another problem.
They are not simply asking you for your leetcode ranking (or whatever the name of the-site-of-this-week for this type of problems). That may actually be much more fair, but it is a much worse experience.
> I’ve also had success with allowing candidates to choose take home assignments.
I was giving take at home assignments until one candidate basically scolded me for wasting his time with these. I think he actually had a point -- I would probably refuse a take at home assignment myself -- and I have since stopped giving them (we hired that guy and he became a friend of mine). They are also not very fair -- they are skewed towards people with more free time at home -- and very prone to cheating, so you have do the whiteboard discussion afterwards. Overall I find this one of the worst experiences.
> I’ve also had success doing what every other other industry does: trust their resume, ask them to talk me through projects, and talk to references.
In the same way that the whiteboard interview is the only practical method I know for junior positions, at some level of seniority this becomes the only practical way to hire people. But like the whiteboard interviews, this is practically the opposite of fairness. Like in every other industry indeed.
It's very different. If the interviewer already knows the answer and is careful not to give it away, it creates an artificially adversarial process that is completely divorced from the reality of the job.
Working together to solve a common problem is a much more accurate work sample test.
>if you just lash out the solution without revealing your thinking process, many interviewers get disappointed and may throw you another problem.
Knowing the answer upfront makes it much easier to explain your thought process in a way that makes you look smart. Many (most?) companies doing leetcode interviews are more interested in you correctly solving x number of medium-hard problems than they are in hearing how you think.
>They are also not very fair -- they are skewed towards people with more free time at home
That's why I've given them as one alternative among several.
>so you have do the whiteboard discussion afterwards.
You can talk to them about the problem and ask them to explain their decisions. Also if someone cheats and the result is that you hire a fake programmer, fire them.
>this is practically the opposite of fairness
Leetcode interviews aren't more "fair" they just test a different skill than other types of interviews. The hope is that the skill they test is more correlated with job performance than other types of interviews. I don't think that's true.
Admittedly I've never had to hire at Google/MS scale, but my best luck hiring has been like you said - talk to people and be up front about expectations and that we fire if needed.
Solving a toy problem on the fly in a language you're going to be working in seems totally fair and reasonable to me.