Comment by weliketocode | Hacker News Reader

Comment by weliketocode | Hacker News Reader

Parent Full thread

Many people here will disagree with what I'm about to say, but if you're going to give a coding challenge...

I like hackerrank/codility style coding challenges.

gasp

They're timed. They're run against a standardized test suite. And they have support for many languages.

It's really unfortunate the number of companies that give custom take home coding challenges that run the gamut. The problems with which are manifold:

- Inconsistent technology and language choices by candidates

- Time spent varies wildly and is easily underestimated by companies - and 4-5 hours for an unpaid coding challenge in an early round with a company is a significant investment

- Good candidates will de-prioritize your company against others that move faster

I know that I only answered a small subset of the question, but it's very difficult to lump all developers together and come up with a single, wholistic, and ideal interview process. From the nature of the role (IC/Lead/Manager) to the technologies/specialty to the company's mission, the process will need to be customized to some degree.

However, do not think that your custom coding challenge will somehow be the perfect key to assessing developers.

When we were looking for a developer for our team, we created a set of small coding challenges testing the candidate's knowledge of the most relevant aspects of the language. We wrote automated tests for these challenges, and the main bulk of our interview was having the candidate solve them.

The advantages to that approach, as I see it, are:

- the developer is tested for knowledge that is most relevant for their current job (as opposed to writing general-purpose algorithms)

- the developer is showing their familiarity with a testing environment

- writing working code in a code editor seems to me more practical than writing (pseudo)code on paper or on a whiteboard

There are, of course, disadvantages to this approach as well. The developer may have hard time working on an unfamiliar machine and not in their favorite code editor (though what we had was quite standard). This might be a tad stressful (although I don't know how this compares with solving a general coding problem on a whiteboard, or with a task asking you to think not just of a way to solve a problem, but of a performant way too). And it made us focus on the candidate's coding skills too much (which some hiring managers might find objectionable).

The advantage of using something like codility, or making it a take-home thing, is that they can do it in their own time, free from the stressful environment of an interview.

I hate doing take home projects.

* It's way too easy to set a project that takes too long. I was given one the other day that would have taken about two to three full days. That simply doesn't happen if you do the test sitting in front of them - the time spent (and potentially wasted) is always capped because the test setter doesn't want to waste too much of their own time.

* Take home projects inevitably have issues ("bugs", if you will). You can't just "resolve" those problems if the person who set it isn't sitting in front of you.

* Take home projects take employer skin out of the game. It's way too easy to give out twenty tests and ignore ten of them. I was actually half way through a homework test the other day when I was told that surprise they had an open offer and the candidate had just taken the job. I doubt they'd have wasted their own time testing me IRL knowing what they knew.

From the employer's perspective:

* In the last case I also realized after googling that the answers were online and I could just have copied someone else's.

* You lose a valuable source of feedback which allows you to iteratively improve the test if you're not watching the candidate while they do the test. I suspect this is partly why the take home tests I've been given are generally of a lower quality than the in person.

* It's valuable to talk through the solution with the candidate while they do it (describing trade offs they made, pitfalls they're cognizant may crop up, potential downsides to their approach).

  > It's way too easy to set a project that takes too long.

They do often take longer than they claim. That's true.

  > Take home projects inevitably have issues ("bugs", if you will). You can't just "resolve" those problems if the person who set it isn't sitting in front of you.

In my experience, you're usually assigned someone you can call or mail with questions. Asking questions is good. Though I usually figure it out on my own.

  > Take home projects take employer skin out of the game. It's way too easy to give out twenty tests and ignore ten of them

Only if they do it wrong. The right way is to give this assignment only to candidates you will definitely want to hire, unless they produce terrible code. The candidate doesn't mail in their code, but presents it to a group of developers who ask questions about it.

This gives the employer skin in the game: they're not going to keep their developers off their work for dozens of candidates who are probably not going to make it; they only bring them out to assess a very likely hire.

I guess some employers use these coding assignments badly. You're totally right to avoid those.

  > I could just have copied someone else's.

But could you have been able to explain the choices of that someone else?

  > You lose a valuable source of feedback which allows you to iteratively improve the test if you're not watching the candidate while they do the test.

Maybe, but to many people, coding on the spot is more stressful than it is at home. Although I've also done some really enjoyable pair programming on what was probably my only long interview day (which had an interview, pair programming, and a game to get to know each other).

  > It's valuable to talk through the solution with the candidate while they do it

But it's also valuable to talk through the solution after they've done it. That's when they have made those trade-offs.

The disadvantage of that is that the interviewer no longer knows if the candidate actually solved the problem or if he got his CS roommate to solve it for him.

Isn't that one of the reasons to ask follow-up questions like "why did you decide to use data structure X here instead of Y"? If the candidate didn't write the code, their lack of coherent answers should make that obvious pretty quickly.

Exactly. That's why you let them show their code to the other devs who get to ask questions.

I don't believe someone who can't code can explain someone else's code in a convincing way.

You will favor those that used exactly same tools/frameworks/testing, but generally smarter person might get a disadvantage. Ie give me Intellij Idea and I would be a tad slower since I use Eclipse primarily. This difference would be mitigated 2 days after hiring, but would be visible during interview. One tiny issue with the env can maybe block their effort measurably.

I get the advantages, I guess everybody does, but there are disadvantages too.

I recently did some codility tests, and they were totally fine.

I'm also fine with take-home challenges, though. But you're absolutely right that they're often vastly underestimated, and you shouldn't give someone a take-home assignment when there's only a small change they'll get hired. It should only be done once you're sure you want to hire this candidate if they're as good as they seem to be. A take-home assignment is too big to sift the chaff from the wheat in a large group of candidates. But once you're sure you'll hire them if they do well on it, a take-home test is totally fine.

Though such a take-home programming assignment should also involve presenting the result to a group of developers to explain their choices. If you're not willing to commit to that, you can't expect them to commit to putting in the time to do the assignment.

Timing a code challenge is a bit like chess boxing. Increased cortisol levels diminish or shut down the reasoning function of the brain, so expecting someone to code fast under the threat to lose a potential employnentbopportunity is not exactly productive.

Yeah, those challenges are explicitly timeboxed, and I know there is a relatively objective grading criteria (which is often as simple as “passes enough of the hidden test cases”).

I refuse to do other types of take home challenges, but am fine with these.

The hidden test cases can be super frustrating, especially when you're passing 7/10 and the timer keeps ticking down. It's impossible to debug besides just staring at the code and guessing what else they might be looking for. Then to ultimately have the company immediately go, "Sorry you didn't check enough green boxes. You've failed. Goodbye."

Although sometimes this is true, most of the time you can't pass all the test because you have written a messed up code that you yourself can't understand. Learning to write the simplest code that works first time one of the most critical skills in programming and in real life also there are no test cases written for you.

Reply on news.ycombinator.com