Unless your company can afford a month-long interview process for every candidate which the author suggests, this is the best we have.
If the current leetcode style of interviewing is the best that SV can come up with, despite having some of the best engineers and thinkers in the world and billions of dollars to spend, that's really sad.
At some point you have to accept that there is no ideal way to interview, and that there are fundamental time/precision/recall trade offs. Big tech optimizes for low time and high precision, which means you certainly will end up with low recall (false negatives).
I’m sure there are improvements to me made (and maybe the specific leetcode style is not ideal), but I don’t think the billions of dollars is relevant. It’s like saying it’s sad that tech companies haven’t improving on O(n log(n)) sorting despite their billions — it’s not possible.
Griggs applies to any hiring criteria or practice that has a negative impact on a protected class compared to people outside the class (given the set of protected classes, this is essentially any hiring criteria or practice) without sufficient evidence of probative value on job performance. It is neither limited to things very much like IQ tests, nor are IQ tests any harder to justify than any other element of a hiring process.
For all their failings, leetcode-style interviews are probably less culturally-sensitive than standardized IQ tests. They probably do have (unintentional) disparate impact, but this seems like a really hard thing to measure (and possibly correct for).
Yes, the existence of readily available statistics may make the unequal impact easier to show.
> For all their failings, leetcode-style interviews are probably less culturally-sensitive than standardized IQ tests.
“Culturally sensitive”, maybe, but they almost certainly have quite large unequal impacts adverse to protected classes (including “age, if over 40”), though the absence of external data raises the cost of proving the unequal impact. Also, certainly less demonstrably predictive of job performance in software development.
> They probably do have (unintentional) disparate impact
Probably, that, as well, but I don't think the age discrimination function largely is unintentional to start with.
Just because code challenges are not relevant to job duties doesn’t mean the results are irrelevant to job performance. They are a proxy intelligence test. General intelligence is the best predictor of success in almost any role (not just software engineering).
This is what confounds people. They think the interviews are designed that way because they are supposed to be representative of the job — I don’t believe that’s the case. They are that way because they provide a strongly correlated signal of performance after hire, and big tech has decades of data for all different interview types. I’m very confident that if they had a better interview circuit that could be done in a ~day, they would be doing that. Obviously even at big tech, referrals and recommendations count for a lot.
While I don’t like the low recall, I do think that “invert this binary tree” probably has less bias than a quiz on technology, or a design conversation (that seems way more susceptible). Perhaps it has a bias for a particular kind of computer science education and thinking, but at least that’s not a protected class. I’m not seeing the age connection, but I could imagine e.g. a computer science education in different countries emphasizing different skills over years (and leading to some candidates with a leg up on the “tests”).
This may be the case. But even if it does act as an effective proxy--and I'm not sure it does; my worst yeses in interview loops have been very adroit programmers who I passed against my better judgment when my "not sure I want to be around this person every day" bells were ringing subtly in the back of my mind--then it has a different problem. You've now set expectations with the interviewee that oh yeah, we do hard stuff here. Then they go frob knobs or write frontend stuff all day.
(This actually happened to me at my first job. I didn't know any better, of course. The hiring manager pumped up my tires with all the difficult scaling work, etcetera etcetera. Then I was writing HTML into templates for the first six months I was there because they needed a body to plug into the role.)
As humans with a strong confirmation bias, it is extremely difficult to tell what's behind your feelings for those worst yeses. It could have been the case that the candidate had red flags, you saw them, but couldn't articulate them. Or it could have just as easily been the case that the candidate had a cultural and/or communication style that was different from your own, and they also happened to perform poorly after being hired. It's important to remember that no interview process is going to yield perfect results: there will be false negatives and false positives, you can only move the trade-offs while simultaneously ensuring that you're avoiding any conscious or unconscious discrimination against protected class to the fullest extent possible. That's a hard problem to solve.
Google also found that interview performance isn't a good predictor of on-the-job performance.
Sorry, but it's hard to take this style of interview seriously after hearing about these things.
People repeat things like this without really understanding statistics and priors and what that assertion really means. Assuming that statement is true, it applies only to Googlers who have been hired by the hiring process, i.e. if you are beyond the cutoff threshold of the interview process, the ranking within that subset is not determined by the interview performance. That in no way implies that among all the interviewees, including the rejects, the job performance would not have been correlated with their interview performance, had they been hired, hypothetically.
Inverting a binary tree is not the kind of task that is a software engineer is likely to have to perform at most companies. I think companies can come up with better questions that still involve algorithms and data structures but better correspond to problems that they professional software engineer might actually need to solve.
For example, to repeat a comment I made elsewhere in this thread, ask them to serialize a binary tree to a byte stream, and deserialize the same tree from a byte stream. There are plenty of edge cases in this problem; you learn whether they understand a data structure (binary tree) and how to implement it, as well as how to write algorithms that operate on it (potentially a recursive descent on the tree; potentially a visitor, depending on how the candidate designed the tree).
This also isn’t a problem I’d directly expect to solve during regular work, but it’s useful and close enough: serializing/deserializing objects (data structures) is something I’ve had to do in reality plenty of times. (Even if you’re using an RPC library or IDL, you often have to “serialize“ in the sense of translating into the business object into one generated by the IDL or RPC client)
- Implement a LRU cache
- Implement a rate limiter
- Read and process a very large file line by line
- Combine two sorted streams
- Calculate the sum of a specific section of a MxN grid
- Parse and solve a math equation
- Find the nth most common word in a blob of text
These are all pretty realistic representations of what a software engineer can be expected to do day to day.