My platonic ideal for how engineering hiring should work
blog.alinelerner.com
blog.alinelerner.com
There is no meaningful data that any hiring process--good or bad--improves the outcome of a hire. If there were, everyone would be using it.
Instead, third-party HR monoliths have moved in and snatched up (and more or less created) the 'market' for screening and filtering applicants. Larger companies sigh, throw up their hands, and say 'well how else can we deal with hundreds of applicants?'.
As if that 'funnel' of applicants contains what you think you want. As if it's just an exercise in reductionism, each applicant a data point to evaluate.
Given this, why not hire lightly and fire lightly?
Are you saying that you had no qualifications at all before? Because that's what they're saying, just applying for any high-paying job, even if you don't have the requirements.
In both cases, I was switching industries (from ASIC design to IoT cloud software to data analytics). So, I arguably had some qualifications and arguably none at all.
Does this have some implied conditions I'm missing? Because to me it seems absolutely obvious that filtering for people who have experience/a degree will vastly improve your outcome over true random hiring.
And, also personally, I've had great experience with people who migrated from absolutely unrelated fields (e.g. several people who retrained - within same company - from support into software development). Similarly I had great experience with fresh interns turning out to be brilliant theory people.
The candidate stream is itself not "diverse" (assuming you are using the normal meaning of the word of lots of women+racial minorities). Simply randomly making offers wouldn't alter the outcomes because those people aren't being de-selected by skills testing to begin with - unless you believe the 'diverse' candidates are less skilled programmers.
Isn't it? While machine intervention might be able to weed out the most obvious low-quality candidates, I don't think there's any strong evidence that most hiring processes actually select the strongest hires - just that they, out of the set of candidates that progress to the point of human intervention (i.e. interview), select a decent one.
The problem is that interviews don't scale well, and some kind of automatic culling of the field of candidates is necessary. Engineers, managers, etc all want to feel that whatever machine solution (keyword searching in resumes, applying AI to recorded video statements, HackerRanks) they select is better than random - but there's no incentive for them to check that that's true.
Obviously randomness + interviews is better than pure randomness, but ultimately hiring processes are still pretty random. If you need to weed out most of your candidates, you can't do much better than throwing most of their applications away - and companies end up doing just that, only they cargo cult an "automated process" that claims to do better.
https://www.fastcompany.com/90462746/the-body-shop-will-star...
No, it's just the companies using it keep that data to themselves. Their hiring process is then a competitive advantage.
But I do know some companies who have a competitive advantage in the tech market, and their hiring is not great.
If hiring in tech is broken, then all of hiring is broken, whether it's fast food cashiers or aerospace engineers. Can you seriously claim that any of those have better hiring practices? From personal experience, I can say they are far worse.
Seriously, I would say that the competitive advantage of a company comes primarily from its product and the vision and competence of the company leadership. You can't just hire your way to success without that foundation. Bad leadership can destroy the greatest employees.
The company founders almost always come together in a very informal manner, and often the early employees do too. And that's the basis of the company. It has nothing whatsoever to do with some special snowflake data-driven formal hiring process.
Indeed, I would argue that once a company becomes very large and successful, they become bloated and bad at pretty much everything, including hiring. At that point they have to get better by acquiring other companies, not by hiring through their standard bureaucratic processes. The competitive advantage of giant corporations is the ability to swallow other companies whole.
I don't know if it was ever made public, I think I heard this story via backchannels. But Google did a research project at some point to discover why GPA/interview performance in general did not seem to correlate with post-hiring job performance. The project was never allowed to reach full completion because the direction their data and investigation was going looked like this:
"Our hiring process does not correlate with post-hiring job performance because we have a large and measurable bias in interviews in favour of Ivy League candidates and women."
In other words, GPA didn't correlate with job performance for Google because it was a confounding variable: they were selecting the sample for top tier universities (which select on GPA) at hiring time, but not when it came to promo reviews (which had a different process and anyway, less rhetorical ideology involved than hiring, at least at that time).
https://www.nytimes.com/2013/06/20/business/in-head-hunting-...
[1] Some Thoughts on Interviewing and Why We Do It https://jwongworks.com/blog/2018/07/24/some-thoughts-on-inte...
[2] Screening developers should be easy https://acjay.com/2019/05/16/hiring-developers-is-easy/
I think my industry (civil engineers) is a bit easier to hire for because credentialing weeds out people who are completely out of their depth, and lying about qualifications can get you in real trouble. Managers can assume you’re basically competent and can focus on hiring people with decent personality and a good reputation. Some of my software friends use brain-teasers to screen candidates, and it comes off as a bit silly to me.
If material like this had been available 20 years ago it would have made a huge difference.
Pretty much the case for every non-Software Engineering discipline.
But then again, people in the tech field will vehemently defend the system - because it's one of the very, very few fields where you don't need a college degree, can go on and "grind leetcode" for months, and land a six-figure job.
And it is because of that, that tech companies are making their recruitment process bordering the absurd. They're simply deadly afraid to vet frauds, that have "gamed" the system.
So in short - tech companies would rather let 10 decent candidates slip by, if it means they can block 1 fraud.
I've been to many engineering interviews, outside tech, and it is not even remotely the same process.
There are arguments here and on the blog post that the industry doesn't have credentials. We do have credentials, it's the CS degree, but universities suck at issuing them so we don't rely on them much. Sometimes not at all.
The people saying, hey, why can't we be like civil engineering are thus perhaps not sounding as re-assuring as they hope: why would universities be more trustworthy at teaching civil engineering than other kinds of engineering? The problems are incentive related. Ultimately the software industry is highly egalitarian and a bad university experience can't hurt your career permanently, like it could in most other fields.
I can say with confidence that certain resume features are incredibly strong predictors of interview performance (at least in the related measures, in the field and/or team I'm working in). I'm not HR, but your statement seems to also deny the possibility of this empirical observation.
> Given this, why not hire lightly and fire lightly?
Because it would be a complete dick move to have someone relocate (possibly involving change of country) and then fire them 2 weeks later? That's not even looking at the formal, bureaucratic or training overhead, or any of the other factors in this industry that make hiring and firing a bit more complicated than handing someone (and later taking away) a hardhat and a shovel.
Here are a few (at a level of detail that I'm comfortable sharing):
- Outstanding (in either direction) performance in certain subjects in school is a very strong predictor.
- Cover letters are mostly noise, but incoherent, overbearing or careless ones don't go very far. Lack of curiosity and interest is also a negative predictor (but these are of course easily feigned).
- Relevant extracurricular engagement and/or industrial experience begets more well-rounded engineers.
Let me be clear though that we consider applications on more than these factors. These are just predictors, and sometimes they are wrong. I have interviewed actuaries that couldn't calculate dice roll probabilities, C++ programmers that were confused by the class keyword and C# programmers that hadn't heard of virtual.
These are people who were in workforce 10+ years.
No offense, but that is a very silly thing to consider "foundational" in 2020. The heavy, all encompassing use of inheritance patterns these days is much less prevalent than 10 years ago, even in the C# community.
Certifications are a strong negative predictor. Someone who turns up with a lot of corporate certifications is unlikely to do well when asked difficult questions.
People who have only ever worked in low grade or low tier banks tend to do poorly. There are a few exceptions. GS has good technologists. Most big investment banks have a few programmers (older ones usually) who have solid experience - but who are unlikely to put any creative effort in and may have a "here's why it can't be done" attitude. May not matter for some roles.
People who list a very large number of short roles tend to do poorly.
People who give a list of projects in their resume experience for which they only assert they were part of the group that delivered it, without concrete claims of tech leadership, often do poorly - generally it indicates a "hiding amongst the crowd" problem in which someone is a poor performer and attempts to hide lack of any actual achievement by reference to group projects.
People who list obscure programming languages as a skill do relatively well, people who list obscure products as a skill do relatively poorly.
But what about actual job performance?
But again, I'm not claiming this to be universal. Just that it's possible.
This. I hold a slightly lower bar (meaning more willing to take a chance) on someone who is unemployed than if they're working now. Worst case if I take someone unemployed, hire them "light", and they don't work out is they're back where they already were, whereas if someone quits their job to come work here and gets fired, they're worse off than they were before.
You're still interfering with that unemployed person's job hunt.
But every company in FAANGMO does use the same (or at least pretty similar) hiring process.
Hiring is doing just fine.
I think hiring is "doing just fine" and there's room for order-of-mag improvements on precision/recall.
Source: they let me interview people at one of those fancy FAANGMO places (and I don't actually know what I'm doing). Interview success comes down to how lucky you get in the loop imo.
This is a manifestly absurd claim. It is possible to improve the outcome of a hire, and almost everyone is using those kinds of processes. The firm I work for does an excellent job filtering people for skill and compatibility. The trick is that if you want to capture the top e.g. 10% of applicants, you're going to have to pay them several times more than the median applicant. Most companies are too cheap to attract distinguished talent.
I don't have stats but anecdotally, hiring by personal reference seems by far the most effective way to hire.
> Given this, why not hire lightly and fire lightly?
Because it's a huge financial and organisational burden on the hirer and an even huger (sometimes existential if you live in the U.S. due to health insurance but that's a whole nother rant) burden on the hiree.
It shouldn't be. From my experience, the financial cost that comes from increasing the number of people you fire after a few weeks is less than what companies spend furiously guarding against making a bad hire.
>and organisational burden on the hirer
There's no reason this needs to be the case.
In a sufficiently large company you could have a specialized onboarding team, but you’d need to be careful that the specialized team doesn’t get isolated from the technical culture and needs of the rest of the company
Daniel Kahneman analyzed a bunch of data that lead him to concluded that the typical interview process did nothing to help select the best candidate. There's a chapter about it in Thinking Fast And Slow [1] and the advice he gives is summarized in this article [2]. I remember thinking after reading this book that it was just a matter of time until everyone everywhere would be denouncing interviews but here we are - old habits die hard.
[1] https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp...
[2] https://www.businessinsider.com/daniel-kahneman-on-hiring-de...
> A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go into the interview unprepared and to make choices by an overall intuitive judgment such as "I looked into his eyes and liked what I saw."
Hire people with multiple pages on a resume with a variety of experences is a good strategy increasing your odds.
I see very toxic people being rejected that think its because of their gender/color/whatever <= lots of complains but the process work
I get rejected sometimes because of misalignement from me, or from them <= feels bad man.. but the process works
I've never been hired to a place where I found that this was all a terrible idea, in 25 years. Theres been places better than others, theres been adjustments after hiring.
I've seen places hire people using simplified processes and the candidates were unsuccessful and unhappy after being hired <= process fail
Because any company that adopts such an approach will never be able to recruit anybody who wants to work in a stable environment with a consistent set of smart people who aren't in constant fear that they will be fired the next day.
I suppose if ALL companies did that then there might be some sort of fluid equilibrium state where most companies had some mix of mostly competent engineers who chose to work there for positive individual choice/reasons. But in a more winner-take-all world where the "best" need to hire the "best", because that means second tier companies get the rejects and therefore will fail to compete and the chosen few get success and riches? Set the bar incredibly high and forget fairness. (also an extreme for purpose of conversation, just on the other end of the spectrum)
It is still not easy to find good people, but the truth is: HR are not the best people to decide who is. If you have good people already and they don't try to build an empire, they will select good people. The danger there is only that they will select people that are too much like themselves or they will select the worse candidate because they fear about their own standing etc.
Where I live you have a month when starting were you can be fired with 2 weeks let-go period, after which it takes 3 months or more.
I think you should have an intensive monitoring/evaluation in the first month, and fire lightly if you think there is a mismatch. After you have determined an employee fits you, if they stop fitting you try to figure out why, help them to fit again (are they the kind who gets bored, needs new challenges?, do they have problems at home? If you found someone who fits I think it is more efficient to make them fit again then try to find someone else who fits). I believe this is the way to go even in areas where you do not have the rules about the dismissal period that you have here.
Netflix, for example, and Amazon (to some extent) have gotten away with this model but the vast majority of companies are simply unwilling to pay top of market.
Word of mouth recommendations, and yes everyone uses it.
I tried building spaceship and failed. Fiend of mine tried building a spaceship and failed too. However, I know of people who built a spaceship. So is building a spaceship a crapshot?
Reality is: 95% of HRs are incompetent. That's where randomness comes from. Take a random man from street and assign him for HR role after a brief training -- you'll achieve a similar performance to those 95% HRs.
Everybody's talking about hiring the engineer, but so fewer people talk about finding a good HR that would help you to solve a "simple" problem: how to find a good engineer with small experience and small salary that would not leave your company for the next few years and would become after several months just as good as 70-100$/hour engineer hired from top software company, while later most likely still gonna need some time to adapt to your product to become efficient.
It's comparable to how we laugh at recruiters when they ask for 5 years in a given language that has only been around for 2 years. You clearly don't know what you're talking about.
I always tell the folks I mentor through bootcamps they should figure out which area they like best, focus most of their efforts in that area, and then advertise themselves as a front-end developer or back-end developer or whatever, not full-stack. The upside down T approach. Show recruiters and hiring managers you at least know enough to know what you don't know.
And stop complaining on LinkedIn about how unfair it is that you don't have a million job opportunities pouring in or about all the applications you submitted and didn't hear back from - yes it's annoying, but if you advertise yourself as something you're not, you've likely already wasted someone else's time, so understand they probably don't want to waste anymore giving you an explanation for why you were passed over.
I've yet to meet a bootcamp grad that can efficiently architect a relational database, for example. I know plenty of Sr. level front-end developers who wouldn't think to call themselves full-stack even though they know how to submit a user form that persists to the database. Same with back-end developers who know HTML, CSS, and pick your flavor of JS framework.
If you've worked in the industry, you know things change too fast to stay up to speed on everything. It's not impossible, but it's unlikely and unrealistic to have that expectation of someone once they are employed.
Many non-tech employers careless about your stack, as long as the application performs the tasks they want and the cost is reasonable.
A full stack dev most likely won't be an expert in any particular area but should have a good understanding of the web so they can pick up different JS libraries, web frameworks easily.
And also, I don't consider full-stack exclusive to the web. Desktop apps, for example.
Suppose your employer wants your web app to interface with an industrial PLC, or fetch data from a car's CAN bus, or talk to a vending machine (ironically the sort of machine that Java was originally designed for).
Most bootcamps don't teach enough fundamentals for someone to figure out how to write both client- and server-side code for those sorts of situations.
Or on the other end of the spectrum, could you write hypervisor code to manage the sort of mass-scale VM deployments behind most cloud providers? It would be very difficult to be an expert on the "full stack", if you think about it.
So...tl;dr, most of the application layer? Probably the physical layer too, any how many of us really work with the link layer?
My company is not a hot Silicon Valley tech company either. It's a dull, boring, investment bank. I am generally not impressed at the quality level of most of the candidates we get when we hire. But still 100% of them are able to code more than well enough to pass FizzBuzz.
Now throw anything beyond the easiest of leetcode easy level questions at them and the majority will struggle. But that is still a level above Fizzbuzz and the like. And most come from backgrounds where the last job they were hired for did not leetcode them, but rather relied on language/framework trivia type interviews.
You would not imagine how many people (fortunately) get filtered by this step.
It's selecting for languages and candidates with built-in support for HTTP and json and base64. It's far from a 1h problem if the candidate has to find working libraries for each of that.
I wanted to do a challenge that really reflected the stuff that we are doing at Paystand. When you are building a Web based SaaS application, the majority of time you will be a) Interfacing with crappy APIs. b) Doing simple processing in your logic (hence the string processing and sorting), c) Dealing with GET and POST requests in JSON
That's all I tried to present in the challenge. And if you read it, you will see that it gives A LOT of hints. There's even people who have asked me why do I give so many "clues". But on the other side I have gotten candidates that email me to ask me why they API call is returning an error (it is there in the notes that you must use the proper HTTP header). People are lazy, people don't read, and those are not suitable candidates for us.
Also the point isn’t to demolish the interviewee with a difficult question, it’s to get a feel for how they are to work with (and of course, if they know the basics at all)
"Write a function that takes a set of numbers [array, list, or equivalent] and returns the average."
Couldn't even get started.
I was the third interviewer. He'd BSed his way past the phone screen and two others and I caught a whiff of "Johnny can't code" and decided to check by asking him one of our new college grad questions.
There have been many more who bombed terribly in a manner that leaves me doubting whether they could solve FizzBuzz given an hour.
I don't doubt your beliefs, but I am surprised that you've not found any candidates who cannot possibly code at all. That probably speaks well to whatever upstream filtering process your firm is using (it would be ironic if it was "coding FizzBuzz using this online coding platform").
1. External recruiter shotguns out candidates to my team manager. Having been on the other side of this process myself, any vetting the external recruiter does is pretty minimal and purely behavioral.
2. Our manager passes around resumes to the team, and we collectively review them and give a thumbs up or down based on how subjectively we are impressed by it.
3. Manager assigns the more senior members of the team to conduct phone screens. Typically we throw some of the easiest of the easiest leetcode questions, or very simple "implement this in JS" questions. I personally have failed people at this stage mostly for communication or behavioral issues.
4. On-site. Combination of easy leetcode questions and "implement this in JS" questions. I can honestly, proudly, say that my team does not look for hyperoptimal perfect leetcode solutions - we genuinely focus on the communication process more. Part of the reason for this is, as I mentioned before, most of the candidates we get would struggle at all but the easiest leetcode questions and we cannot afford to be picky. We're not a hot tech company. We're a boring investment bank.
Very, very rarely we get a candidate that has worked at a hot tech company - even a FAANG occasionally. Their candidacy gets treated like a celebrity. But so far all such candidates have been turned down by my manager's manager because "they don't have enough business (finance/trading) knowledge". Never mind that most of our current team has extremely little to no such business knowledge too. In any case I doubt we could match the level of compensation they might be wanting.
This meme has been going around for ages (at least since Spolsky wrote about it) but I really doubt it is as common as people think.
I think it’s still rational to pass on someone who got stage fright, since in that case you have no signal on which to make an assessment. But I think the “can’t code” rationalization does candidates a disservice.
Interviewing is really stressful for many candidates! I encourage interviewers to remember this and treat candidates with compassion; maybe they can tell you think they can’t code, and that makes them more flustered. Vs. a joke about how interviews are stressful that might put them more at ease.
At my current firm, I hired most of the engineering team. In the early days we relied heavily on website applications and external agencies. A non-trivial number of candidates would struggle with a task like "read a text file into memory and print out the lines". Not FizzBuzz, if anything that's even easier: FizzBuzz trips up some inexperienced candidates because they don't know about the modulo operator, which is relatively rarely used. But almost any kind of program will use files.
At some point we started pulling sourcing and recruiting in-house. We also dropped some bad agencies and got a bit more savvy about which ones were cheating. The candidate stream got better at that point and total failure became less common. Unacceptably poor performance was still a frequent issue though.
Disappointingly, to me, a lot of candidates and even people we hired would try to claim that asking candidates to load a text file from disk was somehow unfair because "real programmers" don't work with files anymore, they just use web frameworks to do everything. Needless to say, that was not the kind of software engineer we were trying to hire.
Anecdata, but from five years helping beginners through a popular online curriculum, this pattern is ridiculously common:
1. Beginner learns some HTML/CSS over the course of a few weeks.
2. Start on the JS curriculum, get stuck immediately.
3. Start to ask in forum if there are jobs [at a FAANG] just involve knowing HTML/CSS.
4. Very polite and detailed replies start to appear on forum of the form "well, it doesn't hurt to try but you might want to bear in mind that at entry level you're competing against people who have years of learning behind them", or "maybe just try to work through the slightly harder bits of the curriculum first (bearing in mind that even that isn't going give you a deep knowledge of the subject on its own)".
5. Few months later the beginner reappears talking about how they've applied to x number of jobs with no responses. And they start asking about how to get jobs working freelance. Moderators take some diazepam and start putting together more polite replies.
Anecdata as I can only see a tiny slice of self-learners via the forum; I feel for your ex though
A few ideas that I think would give these grads some better luck:
- Update your profile to say "Interested in joining a front-end or back-end team" or "tech generalist with a focus on X"
- Elaborate even further in your About section and let people know which of the two you are stronger in and what experience you have
- Swap out "full-stack" for "web developer"
Don't tell me you're something you're not. Under-promise, over deliver. Seeing full-stack on a resume sets my expectations for whoever I'm about to interview regardless of the number of years of experience.
edit: styling
Full stack interview tracks seem to be identical to backend/generalist tracks. Purely leetcode questions up the wazoo.
Some front end interview tracks are the same. But I'm seeing more and more cases where they will throw "implement feature X in JavaScript" type questions, not leetcode.
Seeing full-stack on a resume shouldn't set any expectations other than they have and intend to continue working on both front-end and backend code.
The point of my response is to say, if something isn't working, change the formula. Here's some anecdotal evidence given my experience that might help. Stop complaining on LinkedIn about it. I personally look at LinkedIn and if I see multiple posts from you complaining about how unfair things are it's going to be an immediate pass.
It honestly just amounts to "The most well compensating jobs with the most competition to attain them have a difficult application process, and that is unfair (to me) because of X, Y or Z". These posts are positively dripping in grandiose levels of entitlement.
I work for company A, I'm starting to get slightly bored or interested in doing something else. Without quitting my job, I contribute a few days of code to company B, whose project I find interesting. As I contribute more to company B, I eventually quit company A and join company B full-time.
Benefits of this approach: no interviews. Know what you are getting into. Try different things and what works for you.
How can we make this work: legal framework. California, if you are listening, pass a law that prevents companies from exclusive work arrangements, so I can always do a side-gig with company B even if they are competitors. Have company B compensate me, so we don't end up in a situation where company B is abusing job seekers by getting free work done. Maybe a law could say that I'm safe as long as my work for company B while employed by company A doesn't exceed 5% of my salary or so.
Would that work?
People sometimes want to leave a job and it's in everyones interest for that process to be smooth and pleasant. The leaver has a good final impression of the firm so they recomend it in the future to potential clients or employees, they get support in finding and trialing a new job, and the company to whihc they are seconded gets a low risk trial of a new memeber of staff.
It just seems so grown up compared to the ineffectual thrashing about of tech recruiting.
If you become a contractor rather than a full time employee, the only thing you're missing is benefits/retirement probably, in addition to longer term commitments from an organization (maybe - these days there are no real guarantees).
If you can get benefits/retirement through some other kind of government program or join an independent group that forms together (I remember there was a YC company that was doing this) to pay for group benefits - then why not just go the contractor route?
1. This process is probably going to bias your hiring pool towards young and single people that are willing to either work weekends or use vacation time to go work on a part time job
2. Forcing a company to let an employee work for a competitor opens up a whole new can of worms. This seems game-able in the same way that the Washington DC revolving door is e.g. Uber hires a Tesla autopilot engineer at the cap of 5% of his salary, and after a few years of "contributions" which are totally not industrial espionage they bring him on full-time with an eleventy million dollar signing bonus.
Or do you mean the employee would do this in secret from company A? If so, like the other commenter said, that excludes anyone who can't put in weekend work time.
Obviously no one wants to hire a long term full-time salaried employee on those terms.
However, the legal issues sound like a nightmare. Additionally, in the US many people's benefits are tied to their job, so if they needed to, for example, take unpaid leave to do this "test drive" that would also cause complications.
This fundamental problem this article is describing is that recruiters are filtering out too many candidates by pedigree to ever have them end up in an algorithmic interview with a tech company.
My pedigree is average at best. I've worked my entire career at non-tech companies (banks, hedge funds) doing decidedly unsexy work. I'm still able to attract enough attention that FAANG and many other top tech company recruiters contact me first.
Interestingly enough, it's the top companies in my current sector (finance) that refuse to even have an internal recruiter casually speak with me. i.e. Citadel, Two Sigma, Jane Street, etc.
There's also stories I see occasionally on LinkedIn, etc. of people from completely non-technical backgrounds getting hired as SWEs at top tech companies. Taxi drivers, ex-felons, aestheticians, etc. getting hired as SWEs at FAANG and unicorns.
Multiple external recruiters have attempted to submit me to Citadel and Two Sigma over the course of the past five years. I always tell the recruiter I've never heard back from them in the past, but they're always confident that they can vouch for me and get me an interview. Of course, I never hear back and the recruiter is puzzled.
Working at a well-known company can definitely help open doors. It proves that you’ve already completed their challenging interview process and it’s also assumed that you picked up some good engineering practices while working there.
However, pedigree alone won’t get you jobs at most places. You still have to go through the interview so they can see how you perform on a level playing field.
As much as online comments hate the coding challenge interview problems, they’re actually a great thing for people without impressive career pedigrees. It’s the only way you’re going to get to compete on a somewhat level playing field next to the kid who went to an Ivy League and landed all the right internships due to their parents’ connections.
Generally speaking, it’s the people who have jobs at well known companies who want to get rid of coding challenges and go back to hiring based on pedigree, mostly because it benefits them greatly and reduces their competition. In reality, anyone talented enough to work at top companies doesn’t have to study much for coding interviews. This idea that coding interviews are somehow filtering out brilliant engineers and selecting for less qualified people is largely a myth. The reality is that engineers just don’t like being challenged on the spot and would rather employers just trust their resume. That is, if they already have a good resume. The juniors love coding challenges because it’s a way to prove your skills without relying on pedigree or university or family connections.
Majority of the issues are caused by requirements not being communicated, captured, or implemented properly. That's something that's not really focused on tech interviews. I just want a diligent developer who can identify potential issues, do their research, and propose solutions rather than be able to design a system in 90 minutes, because that's never happening in real life. In fact, I'd not hire someone who can claim to do so.
The missing feedback loop from performance review back to the hiring process means that the hiring process can't get fixed. Until companies start adding that process back in, hiring will never get better.
Perhaps though if one looks at the performance review weighted to a review date as soon after the hiring, and sort of tail off how much weight is given over the coming years it'd make sense.
I've just seen a lot of bad hires get better with time. And some good hires lose steam or become jaded and perform worse over time.
Thus, as messed up as it is, one of the keenest signals is tenacity in the end. And ironically tenacity in preparation for interviews (that Leetcode grind) is somewhat correlated with performance reviews in the long run and perhaps correlated with utility to the company.
That said, "decent" engineers in the long run that don't think outside the box but check their performance review boxes slowly bleed a company's ability to disrupt. I.e., performance reviews need to be expanded a bit to think about the health of the overall company and not just the employee performing their function.
As a large tech organization, I'd prefer a complex distribution of performances over an even distribution any day - because value to tech companies is still influenced by more black swan event type of innovation. But then I have a high "beta" (variation) in terms of what I'm looking for. Companies with more of a utility emphasis perhaps are better served by a hiring process that is "reliable" with a lot of false negatives.
If they widen their expectations, and hire people that they believe they can mentor into net contributors, then their hiring practices should reflect this.
There are many ways a company can make hiring more scientific, including A/B testing (ie. hiring people they normally wouldn't, have more data around how productive new hires are, etc) but I've never seen it over than "copy Google and we should be good."
Because of that, you inevitably get "weird" results like seeing that your best employees often did relatively poorly on interviews (Google did a study that produced exactly that result—their best employees often were those that failed 1 or 2 of their interviews).
But this is just selection bias: these are candidates that got hired! Which just means that they were so spectacular in some other respect that they were able to get an offer /despite/ poor interview performance. It tells you nothing actionable about your interview process.
Really you need to take an experimental approach where you randomly give offers to people who failed the interview, and see how they turn out.
Fundamentally, hiring is simply not an easy problem. There's a small cohort of candidates who can be reasonably confident will perform well in a job (mostly people who have performed a similar job at a competitor) that everyone competes over and a marge larger group of candidates whose possible performance is very hard to judge.
You, along with however many other candidates rotate around among different companies. You maybe work for each company up to 1 week. At the end you get your offers and decide among them.
I've ended up working for too many lemons because of lack of visibility into how well put together they really are. I want to know ahead of time whether they really have their ish together or if the interview bubble is just a facade.
Also, at a job fair you have no idea what the day to day life will be in your new position. In my scenario, once you are or aren't hired you will have already lived the life you will be living if you are hired.
Takes about 45 minutes. Pretty simple, the goal is to make sure that they want to do the work that we need done, and that they feel prepared to do so.
So far so good.
How do you get these candidates in the room? Does everyone who applies get an interview?
And how do you decide between candidates? If it’s as you described, it sounds like it’s “first come first serve”.
Yeah, first come first serve.
The result is that we can bully and cajole smart, capable people, then go on to make 6 figure salaries and makes billions of dollars in revenue.
Also, bonus points if it involves OTR/PFS so you can talk details without attribution and know that since you're in a "pool" if you got through to an identifiable, in-person interview you can pretend like it never happened, but the candidate and the engineer both at some point definitely interacted in a way that they both felt comfortable proceeding to that point.
I'm interested to see what the new trend of remote work will do to the current paradigm. I think the lot of us are about to find we're not much more than highly skilled mechanics.
Edit:
On reading all the comments that came after mine, I realize I'll have a decision to make Q1 2021 when my role at my current startup becomes financially unwise to maintain (slow growth on a popular product in a very niche industry). Im a co-founder but still it will make more financial sense to put the tech product in Maintenance Mode for awhile with some offshore contractors. All the hard work has been done save for riding out the slow Change The Industry adoption curve.
When this day comes I'll be back on the market myself and in considering all this I think Im going to target large F500 companies who's core business isn't tech or internet or whatever. Companies that rely on tech as a part of their larger non-tech business.
I know folks who are devs at AMZN and GOOG and I can't help but come away with the impression that save for the RSUs they wouldn't be there either. The pain around getting the offer may not be the worst part - the worst part may be after you accept and actually have to work there. Fairly simple logic might indicate that if the dating phase is this bad, the marriage can't be much better.
big tech engineer turnover has ground to a halt (employed engineers playing it safe)
they have open positions but those were based on pre-pandemic turnover estimates
so they don't need to hire as many
the result: perfect coding challenges not enough ... now also need perfect behavioral answers, perfect 40 yard dash, perfect push-ups, perfect knife juggling, and most importantly, perfect clown show while wearing red nose and green wig.
I found this answer to both be true and unsettling.
The process has become optimized for weeding out the 490 bad applicants to eventually get down to the 10 worth talking to. It's painful for both sides, but eventually it does get there.
it's a lose lose situation
I'm talking about the difference between someone transitioning fields fresh out of a 12 week boot camp and someone with a computer science degree, full stack experience, and knowledge of what's going on under the hood. Even when you see many years of experience on somebody's resume, it's really really hard to tell what that means.
It's the combination of a really wide range of skill and a really big difficulty assessing that skill from just the resume. So the interview really has to do heavy lifting in this field.
Compare that to my partners field, where there's a licensing board, the job duties are pretty much the same anywhere, and all you really need is quick check of is this person sane, is their license and training up-to-date, and do their references check out. Even in that field, you can imagine the interviews being way more extensive if there wasn't a license requirement and mandatory regular training. It would be like a lawyer having to pass the bar exam at every interview, instead of being able to say that they've passed the bar exam.
(This is all coming from somebody without a CS degree who transitioned into the field, so I'm really really not proposing we do gatekeeping. I'm just acknowledging that without gatekeeping, we effectively have to pass the bar exam at every interview.)
If you know what skills you are looking for (hint: lots of companies and recruiters don't), this is not actually that hard.
For every skill, there are questions that should be really easy to answer for everybody with expertise, but are suprising/hard to answer for everybody who pretends to have knowledge. This enables you to check quite quickly whether the claimed skills are actually there or not.
For example, the easier sections in
> https://github.com/satwikkansal/wtfpython
might give you a hint for topics that most seasoned Python programmers should know blindfolded while they are probably rather surprising for anybody with shallow Python knowledge. Let the applicant explain how they came up with their solution for, say, some of the problems, and you will see rather fast whether the claimed skills are actually there or not.
It's been several years since I wrote JavaScript or Python on a regular basis professionally and I'd almost certainly fail your quiz. Yet I'm very confident I could be quickly productive in a Python environment.
Interviews should probe thought processing, not language minutia.
The opposite of gate keeping, by analogy, is letting everybody in. That is, hiring each and every single person who applies in. As soon as you set a minimum bar, the first person who can't clear it accuses you of "gatekeeping".
I used to be an electrical engineer. In the traditional engineering world there are mechanics/assemblers (high school with maybe a vocational certificate program), technicians (two-year Associates or vocational degree), engineers (Bachelor's or Master's degree in a specific, relevant discipline), and researchers (Ph.D.+ in a specific, relevant discipline).
In the Valley-oriented tech industry, the first three all get squashed into "software engineer". It's often not clear what a company needs, even to the company. Some companies seemingly don't care; Google, for instance, seems to hire all software engineers to the same bar then sorts them internally into the mechanic/technician/engineer bins based on what teams will take them. I think hiring is going to remain fundamentally broken until companies start making an explicit distinction between these roles when they open reqs and interview candidates.
Now, I'm not saying my former industry's way of doing things is all good. I have effectively left it for good by virtue of not having touched a circuit card or JTAG debugger professionally in 14 years. I do think this industry could learn from the separation of roles, though.
When I browse European startup offers and I interview for some of those positions, I get the feeling that their endless list of requirements has more to do with having high developer turnover and lack of funds to build a team, rather than their CTO's desire to hire a programming demigod.
Credentials for doing well in interviews, maybe. I can respect the frankness in the following paragraph though:
> Of course, one of the limitations of our approach is that we’re getting performance data about engineers from the practice interviews they do on our platform. Any seasoned interviewer will tell you that the signal one gets from a technical interview isn’t the whole story.
And really that is another big topic that requires more research imo - interview performance vs job performance.
This feels like a really big assumption. Is it true? Is there data to back this?
I've felt annoyed at individual interviewers, or a process at a given company but I've in general felt like I got more of a fair shake in the tech world than other industries I've worked in.
The phrase "hiring is broken" comes up dozens of times in HN searches. If something appears broken to you, but persists for a long time, it's time to start considering that the apparently broken thing actually serves a purpose you can't see to someone you don't know.
Minute progressive changes to systems that already supposedly work end up bringing about systems that work better.
> serves a purpose you can't see to someone you don't know.
Does not mean "good, but you're too stupid to see it."
edit: after one figures out why something is a certain way one could come to the conclusion that it must be changed immediately, and the reason it was that way was more destructive than one could have imagined.
At some point you have to accept that the fence maybe doesn't serve a purpose, or its purpose may be forgotten and not applicable, and its worth the risk to take it down and see what happens.
Aline has said some good stuff but she loses me at: "There’s no substitute for chemistry, in dating or in hiring."
Hiring is not fucking dating. You are not having sex with your coworkers, you are not discriminating on gender, race, body type as everyone regularly does in dating. Dating is not a healthy system that you should aspire to in employment. It's very much broken in comparison with swipe apps and the like, just broken in different ways.
Two possible expanations:
- What works for a big company of FAANG type is not necessarily a good idea for other companies.
- Because the FAANG companies have found a formula for printing money, they can afford such a hugely broken hiring process.
There is a long history of dysfunctional processes being replaced with other processes that often turn out to be as dysfunctional as the previous one. Just in a different way. See Scrum and Agile as examples.
Disagree. After reading and participating in the endless HN hiring discussions, I'd say I've come to an understanding of why, at least in SF, interviews are done the way they are.
It seems to be a lowest-common-denominator approach that serves candidates and companies in different ways.
For the candidate, they can do one set of study (leetcode/system design) and that effort is applicable at a wide range of companies.
For the company, they are finding the intersection of people who are at least intelligent enough to grok CS fundamentals (I know it's just pattern matching after you've studied it), and willing to jump through enough hoops.
Hiring works a very different way outside FAANG leetcode style interviews. It's mostly a chat between a couple devs about what you've worked on in the past and how applicable that experience is and trying to get a sense of where you're at in your career and if you seem like you'd be good to work with. It's a lot less standardized, so it can be very hit and miss sometimes as sometimes you get quizzed on knowledge of their particular stack. Often the study you do for one company isn't necessarily translatable to another.
> Although these approaches are rational under existing constraints, they’re neither particularly efficient nor particularly fair to the individuals subject to them. This doesn’t mean that we can’t improve the system … But before we talk about what we have the power to change, let’s dive a bit deeper into the constraints we face today.
All premature fence-pullers think they have rational reasons for pulling the fence down. They're the ones the quote is meant for.
That sounds a little like conspiracy theory. My theory is that hiring is just really hard and it’s difficult to do that at scale.
I know people who are really good at interviewing. I think I myself can interview reasonably for people who have to work with me. But that takes a lot of time and energy. So “specialists” from HR take over. But they don’t understand the real requirements for the job so they have to come up with a lot standardized indirect measures which don’t work well.
Not that hard. Talk to people like human beings and examine their previous work and employment history.
> hiring is just really hard and it’s difficult to do that at scale
Yes, talking to people about their previous work and employment history is a good way to judge their fit for a position. It also takes about half a working day of 2-3 people each time to do it properly.
But if you require that any engineer involved should spend a quarter to half of their week on hiring, and that you yourself consider anyone with less commitment unsuitable, don't you agree that scalability becomes a problem?
How many half-recruiters, half-engineers do you think are available, compared to the overall need for engineers with hiring authority? I don't have data on the former, but it's apparent to me that, if you hire people to do this full-time instead, you're hiring a recruiter. That's what gets you exactly the system everybody seems to complain about.
What does this even imply? That someone benefits from this broken system? The recruiting firms? It is indeed broken as engineers are not happy with going through the middleman gauntlet, companies are short on developers and would ideally want to hire asap. The only problem I see for companies here is find the right candidates (filter the bad candidates) but all these obstacles don't seem to be doing that at all. So back to the drawing board...
An example of this is coming up with "how many golf balls can you fit in a bus?" riddles, in the vain attempt at "seeing how the person thinks"/"problem solving"/"creativity".
When I think about how I'd like my work to have a big, positive impact on the world, one of the first things that comes to mind is fixing hiring/job hunting. I'm moderately intelligent and have felt this way for a couple decades, and I still haven't done anything about it because I still don't know how to make it not suck, and I've yet to see anyone else working on a good solution either.
I've seen a lot of very smart, capable, and talented people have trouble finding a decent job, and eventually settle for underemployment for a good while. A system that could properly utilize their productivity would be vastly superior to our current system. There's a lot of money on the table for anyone who implements that system.
I don't know what that system looks like. I just know with certainty that there's a massive amount of wasted productivity under our current system.
What I can tell you is that when I graduated college, the way you found a job was to use Monster.com--a big job search engine--apply to various openings in a tedious process, and interview. Today, you use Indeed.com--a big job search engine--apply to various openings in approximately the same tedious process, and then interview in approximately the same way.
There's money on the table for anyone who can improve the system, and no one has. This tells me it's hard. (The fact that I don't know how to improve it a bonus.)
In other comments on this article, I wrote my arguments why I don't believe that detecting good candidates is hard (I wrote suggestions there).
I believe the hard problem is rather that most companies/recruiters don't actually know (or won't say) what skills they are really looking for, so they use dubious processes to rationalize why they chose a specific candidate.
I am a hiring manager. I've also put significant thought into how we can change the hiring process, if only to get better candidates. However, it comes down to this: I only get 2-3 hires a quarter. Trying to get a meaningful result on changes I make is difficult. If one doesn't work out, did it not work out in a way that my original process would fix?
And so, we get conservative because even though it rejects a lot of qualified candidates, it also gets good candidates.
If you conceptualize a hire as "this decision costs me $100k to $200k, and six months", you'll understand why folks are conservative. That's my finger in the wind cost of a bad hire, in terms of their salary, the salary and opportunity cost of their first two weeks buddy, plus all the other related efforts to onboard and train. Plus, again, if it's a bad hire, by the time we spend 2 months getting to the termination, and a 2 month search, and a 2 month onboarding, I wait 1/2 a year before I get the employee I need.
Simple solution: Be very clear what kind of candidate the company actually needs, e.g. do you need
- a genius in algorithm design
- a person who knows the subtle details of a specific language/platform/standard X
- a nice person vs a person that will fight for "doing the right thing" (e.g. is possibly strongly opinionated)
- how important is it really that the person is also nice to hang up after work
- an allrounder vs an expert for X, Y
- a person who creates solid code that will be good to maintain vs. a person that "simply get things done"
- a person who is is already good at X vs a person who has a high potential to become good at X
Maybe it's just a matter of wording, but the implication that people who will go to bat for an important choice can't also be "nice" strikes me as strange and contradicts my experience. Good comment otherwise though.
So, do you customize your hiring process for each hire? What if multiple teams need different types of people? How big is your organization?
And if you get a discrimination claim, can you prove you have a fair process? The bigger the company, the more standard the process generally is.
My interviews aren’t heavy algorithm interviews, unless I need it. However, those questions you mention and many others are hard to analyze. How do you test that someone is an expert, especially if you don’t have an expert on staff? How do you know if the person writes maintainable code or just talks a good game?
How do you assess work ethic?
Hiring is hard.
Of course; these were just examples as the "e.g." tells.
> How do you test that someone is an expert, especially if you don’t have an expert on staff?
I gave some idea on https://news.ycombinator.com/item?id=24841552
Now, I can give you all sorts of answers here. My point is that unless you're going to put a lot of effort into redesigning the interview process from first principles for your company due to some need (like Google), or you are putting your effort into a startup like interviewing.io, you're likely going to take an existing process and tweak it. And that explains why the fence is there.
thank you for that :)
This is the exact wrong lesson to take from the proverb. Instead of considering why something is still in place, you should consider why it was put in place to start with.
And to be clear I'm not talking about having too many unqualified candidates who can't do a hello world (there's some but don't pass a screening test), I am talking about an oversupply of candidates of all levels who are able to pass the marathon of interview questions and exercises.
Sometimes when I look back at the last few people I interviewed onsite, I'm pretty sure they could all have done the job just fine. It's crazy that they're all getting rejected, there is nothing wrong with them, the company is overly picky for no reason.
Digging down there's probably 1 who got an offer and rejected it. Companies always forget to mention that half of their offers are rejected, hiring would be easier if they improved on that.
Easier for that company, but harder for the companies they beat out.
Then again my company is a boring investment bank, not a hot tech company. Top quality candidates are not in a rush to trample each other to join us. 90% of the candidates we get would not be able to brute force naively solve an leetcode medium or even a harder leetcode easy, let alone come up with an optimal solution. We would never be able to hire anyone if we rejected everyone that wasn't able to code an optimal solution in 40 minutes.
Yet if hired, I have no doubt nearly all of them would be able to do the actual job more than well enough - and that has been the case with everyone that I've interviewed and gave the thumbs up to.
It's also true that it can be ridiculously hard to get a company to respond to an application, and to get an interview.
Assuming the job requirements aren't unnecessarily restrictive, there is something in the screening process that is actively harming both applicants and hiring managers.
He doesn't claim that it has a valuable hidden purpose, he claims that it had (when it was constructed). :-)
I remind myself of Chesterton's Fence every time I encounter some WTF code.
If 10% of the work you're doing this quarter is work that you are going to be doing next quarter, you'll need to be interviewed differently than if 90% of the work you're doing this quarter is work that you're going to be doing next quarter.
If the job is narrow and predictable enough, you can just assess somebody's ability to do the tasks you will ultimately need them to do. if the job is broad and unpredictable enough, you have to assess it by proxy. Like how due to the long tail of word usage [0], it's easier to just talk to somebody to see if they are fluent, rather than quizzing them on words.
I wonder if software engineering is in some weird anti-sweet spot, where our breadth of work isn't so broad that we really need many year degrees and licenses and boards, but isn't narrow enough that you can just assess somebody on the actual duties you want them to be performing.
Not to mention that different jobs in the field really do have different degrees of unpredictability/variety, so the breadth experienced in a few years at one shop would be completely different from the narrowness experienced in a few years at another, so a resume doesn't you tell as much as you'd hope.
[0] A full 50% of the words in Tom Sawyer only occur once, for example. It's why teaching languages can only go so far, and you really just get lots of experience using a language to become proficient.
1. Resume forms. Just accept a resume. I don't have time to fill out your form that requires exact months. 2. Not hearing back. Its not hard to send a response. The ones I really hate are "we get so many responses so we'll only contact you if we like you!" come one 3. Bad recruiters. The ones who just email and call about being an automotive technician when I'm an infrastructure engineer.
Okay. In that case, I can forgive the "cardinal sin" of bad writing. Here's the problem with hiring: organizations. Hiring practices reflect organizational ladders and career progressions. Where you see flaws in the latter, of course it will backpropagate into the former. The problem with a "platonic ideal" for how engineering hiring should work is that it requires a "platonic ideal" for how a company should work. And what is that?
How do you avoid politics as you scale up a company? Sure, engineering skills are fungibly valuable, that much is evidenced by the market. But engineering career output, and promotional success -- that part is very circumstantial and specific to the company. And I would argue hiring is part of that process -- hiring is getting promoted from candidate to employee. Every company does it differently. And the dynamics are not platonic but tribal. There's no way around that but to find the tribe you want to be a part of and accept it's a little irrational.
I remember wishing and hoping when I was younger for a platonic ideal for engineering hiring and engineering organizations that never came. I grew to understand that the warfield of political economy that the business existed inside of (including and especially non-technical and executive personnel) had so much of a major impact on what engineering even means that to conceive of its retraction or engineering in isolation was a little naive and unrealistic. Is it cynical for me to say that? Is it bad for me to say that?
When your best argument is "trust me I know" usually this doesnt mean any good
The quoted email in no way supports this statement.
[0] https://epochtimes.today/chinese-idiom-point-to-a-deer-and-c...
Your link never glosses the saying as "point deer make horse". Why did you?
The grammar of the saying is normal, far more so than usual for these, and "make" is not the relevant sense of 為.
1. Finding a good recruiter is as hard or harder than finding a good employee - only do it if you expect to be recruiting for multiple roles over a period.
2. Think a lot about what you're looking for in an engineer, how you'll evaluate candidates, what experience will be shows for the talents you need and then how to articulate that to the recruiter. When they send CVs/resumes make sure you give feedback on what are the bad and good qualities.
3. When recruiting I've nearly always gone via recruiters that have gotten me roles or have worked with in the past (or atleast through them to someone else in their organisation).
I could whine on about this topic for days, I ran an IT Division that supported HR for a very large company, and we built a slick streamlined system that worked great, in spite of what HR wanted. The two things that drive me crazy when applying for a job
1) keyword matching metrics, UGH!! It's not an effective filter. If you really want to filter candidates have a job specific requirements questionnaire. As a hiring manager I'd much rather create my filter questionnaire.
2)Using the resume submission process to complete the employment application form <DUMBEST IDEA EVER>, it can be a two step process, REALLY IT CAN BE A TWO STEP PROCESS!
NOBODY READS THE EMPLOYMENT APPLICATION FORM, it has nothing to do with screening candidates. If you want to interview a candidate THEN have them complete the employment form.
I have stopped applying to places that require any more than a resume submission. Its just not worth my time.
It seems like "abstraction" is a recurring theme. The hiring manager is abstracted from the job description by the recruiter. Whether by institutional affiliation or passing a quiz, candidates are abstracted into punch cards that can be fed through the system.
Furthermore there's a seemingly overwhelming institutional bias for companies to congratulate themselves on making the right hire, and for their employees for being the right hires, that the whole thing becomes kind of circular.
Interviewing.io is different from other recruiting startups because verified candidates get to directly book time slots with engineers at companies. Candidates get verified by taking multiple mock interviews (more data points).
Other competitors, such as Triplebyte, still have a recruiter in between interview and company. Even after you get verified and have good performance on the Triplebyte quiz and interview, there is still a recruiter who acts as a gatekeeper. That recruiter can prevent you from speaking to companies even if your interview performance was 99% percentile.
Man I really can't stand job boards. I feel like the only way to make a good job board would have to aggressively vet companies posting jobs, which ofc would lead to it being very barren.
Different types of people fit into different cultures and work styles.
If there's anything at all that I would generalize on, it would be: "Be candid about your work culture if hiring and don't try to work some place that doesn't fit your style if applying."
She's been turning this crank for years, and I believe her when she cites interviewing.io having a 40% successful place rate.
Is that from a different piece? The only 40% figure in the article is in reference to reasons for people using the platform, not conversion rate...
God I love Aline's stuff.
To me it looks exactly like the kind of credentials the article describes.
Too bad I didn't get paid for my time!
I once was asked to explain the difference between 2 functions in a standard library that I don't think I had ever used.
That was the only question I was asked. They completely wrote me off as soon as I couldn't answer the question.
If they had just asked me to write an entire app from scratch (not counting frameworks) I could've done that for them no problem :)
You know that college admissions are just as "democratic" as somebody's online testing process, right? As in, anybody can take the test and if they pass, they get admitted? Like... they're exactly the same concept?
15 hours can save you 15% on someone else’s salary.
I feel used....