Engineers can’t gauge their own interview performance
blog.interviewing.io
blog.interviewing.io
- 44% of the time, the actual and perceived ranking was identical.
- 91% of the time, the difference between the rankings was at most one star.
- Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four).
- There is also a bias towards engineers thinking they did slightly worse than they actually did.
Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.
I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
- 25% of the time, the actual and perceived ranking would be identical.
- 62.5% of the time, the difference would be at most one.
- Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four).
Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
But on your point, the Jury's still out on what to do with Likert-type data like this. Purists do say that it's ordinal and therefore you can't do any numeric analysis on it, but practically speaking you can learn a lot from it by treating it as interval data.
In my opinion, statistical analysis like this is ALWAYS two parts speculation and one part science (if you're lucky). It's designed to give insight without declaring fact so you're allowed to bend the rules a little if it's in the spirit of the math.
My "gut" feeling agrees with the original post. If we treat the data as ordinal data, then we can only look at the accuracy, and 44% is, if anything, more compelling.
Cards on the table: I was involved in analyzing the data in the first place and have a vested interest in things-- but I am saying what I believe without any intentional bias.
Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
So the committees has a decent way of correcting for interviewers who use different scales.
The problem is very few things are short term tasks in isolation. So, someone that takes a while to solve 6h problems can be really efficient in the long term. Worse, short term problems need to fit the same type as what your actually doing.
PS: As soon as someone says "i don't know" you need to stop and realize they would spend a few minutes on google. Spending more time on the topic at this point is pointless.
Or, give them the answer and see what they can do with the information. IE, pretend that they DID lookup the answer and go from there. Did they grasp the concept? Could they make logical inferences from that new concept? If they can't do it when you tell them the answer, why would you think they could if Google told them it instead?
Remember the context is the Job not the interview. On the job, having not done something for 3 years is a minimal setback for most people as the retraining time is almost meaningless. In an interview that 'highly' skilled person is going to take a long time to get back up to speed.
b) Find a bug in a source code. Describe the end-user captured bug, and measure the time and the quality of the solution.
c) Simulate a distributed code review and measure the found problems. This can be extended to UX as well.
(These are just quick examples.)
Be honest now, do you do this?
1) See something or think of something
2) Decide that it's the most important point
3) Go on a "fishing" expedition to try and get the interviewee to see it
4) If the interviewee can't figure out what you're hinting at, decide
s/he's completely worthless! (Or if she gets it, decide she's the ultimate!)
I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint.
Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.
It's interesting isn't it? This was all much easier in decades past. The questions don't matter. The answer doesn't matter.
What matters are the body language cues.
It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up
If you have a true rating system then 2.5/3 out of 5 means average or more or less what you should be looking for in most cases 4 is out standing and 5 is bloody icecream pooping unicorn.
You see this every where a rating system is implemented heck in entertainment movies and games don't fight from 0 to 100 anymore but form the 90-100 if you see any score below 90 or 4 stars people usually avoid that product.
But I always cared about my interviewer's assessment of me.
So much so that I've always reserved the last 10 minutes of every interview just to find that out.
They almost always ask if I have any questions, but even if they don't I ask them anyway:
Based on what you've just learned:
- How well would you say I did in this interview?
- How well do my skills fill your requirement?
- Where do you see me in the first month?
- Where do you see me in 3 years?
- What's the first thing you'd have me work on?
- What gaps do you think I have?
- What are your 3 biggest concerns about me?
- What can I do to address those concerns?
- What is the next step?
When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
How do you phrase or bring up your praise/concerns?
I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer.
You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with.
Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
I think that would be a mistake. If the interviewer isn't able to answer those questions satisfactorily, it might be a sign that the hiree isn't a good fit. Nobody wants to change jobs to something that doesn't work out.
And "How well do my skills fill your requirement?" may not apply for some companies, where their interviewing practices don't look for a specific requirement but go for a more general set of characteristics.
Not commentating on whether this is effective or not, just that it is the case at several of the large companies you might imagine applying for these days.
Translation: The specific requirements aren't explicit.
You're better off asking questions about the business and trying to determine whether they're a good fit for you than you are deciding whether they're impressed by you or not. By assessing their business and their fit for you, you'll decide whether you want to accept any offer they may or may not decide to offer. If they don't offer, they weren't impressed, and they still shouldn't provide you much feedback because of aforementioned liability issues. If they do offer, then they were impressed, and you can ask those questions.
Are they being mean because they think I wasted their time or because I killed it and they want to push me?
P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.
The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
Sounds like a great guy to work with.
I also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com does).
I have an observation to share for the latter kind of interviews:
When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems.
I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise).
Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well.
Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.
Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.
In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer.
The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself.
This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).
The thing is that I already bring a lot of value - I could definitely bring more, but I failed to emphasize what I already have.
I found the analysis very interesting and consistent with my personal bias (I almost always think I did worse than I actually did).
We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here.
Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread.
[1] All the details at https://news.ycombinator.com/item?id=10705926 and previous posts linked there.
I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.
Do they?
Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.
I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments.
CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
That is unless you're a webdev, in which case you can call yourself head fruitloop and it would still be acceptable, apparently
An FE is required in order to be hired at a firm providing engineering services "to the public", such as a civil engineering or architecture practice. It is not required, or in some cases even valued, at product companies.
At the very least, not in the US you don't.
/has title inflated job title of Engineer, not a web developer.
I've never met a engineer who gave a crap what my title was, software developer or software engineer. Which isn't a surprise, since we pose no threat to the jobs of "real" engineers. Not sure why the word is so threatening to some software developers.
Do railroad engineers have to put up with this?
Webdevs calling themselves engineers is an insult to
the amount of work and effort that I've put in to get
where I am now
2. Definition of engineer:
"An engineer is a practitioner of engineering,
concerned with applying scientific knowledge,
mathematics, and ingenuity to develop solutions for
technical, societal and commercial problems."
3. It sounds to me that you feel hurt over credentials. What you are projecting: "I worked hard for this thing and I perceive that you are devaluing it." My response is, credentials are different in software engineering, sorry. And nobody is devaluing your mechanical engineering degree here; if it's something in your field that is valuable, that's good for you.
But mechanical engineering isn't software engineering, and you can't say x isn't engineering, you have to understand what the accepted definition is.
I once worked at a company (2007-2010) where almost every non-manager had "engineer" in their job title.
Aside from Software Engineers, our QA staff were Test Engineers, our customer support people were Customer Support Engineers, our tech writer was the Technical Documentation Engineer (that was a late addition: his predecessor was simply a Sr. Technical Writer), our pentesters were Vulnerability Research Engineers, our sysadmin was the IT Systems Engineer, and probably a few more I can't remember. The only exceptions were the tech leads in development, QA, and support, who all had the title Technical Leader (except one guy who was a Software Lead for reasons I don't know).
There was also a push for non-technical employees to all be managers, even if they didn't manage anyone. Our accountant was the Accounting Manager, person responsible for training sessions was the Technical Training Manager, the people who wrote our marketing collateral were the Marketing Communications Manager and the Sr. Manager of Technical Marketing, etc.
Basically, everybody below the director level was either an engineer, a manager, or a technical leader.
Not to mention the not insignificant number of EE who have fully transitioned over to software, some of whom may have taken the exam at some point.
Likewise, not all "real" engineers have taken the PE or FE exam. Lots of electrical engineers don't bother unless their job requires it.