garyjlin

It's idealistic and naive to say that we'll replace the entire onsite interview, and I wouldn't even recommend we go in that direction. What we're trying to do is clear out the technical evaluation part of an onsite so that there's more time for engineers to have insightful conversations with the candidate, and to get to know them more as humans and potential colleagues. If you're spending too much time with a candidate staring at a whiteboard going over code, you don't have enough time to gauge culture fit.

However, because we're still early, it's not fair to ask clients to chuck out the tech eval part of their onsites, since we haven't proven effectiveness to them yet. So instead, we'll replace whatever they use mid-funnel, and also early funnel auto-interviews. We still recommend someone talk to the candidate both before and after the Litebulb interview.

I think you make a good point here, you're right, because leet code style interviews don't work for most companies, but they do work for some companies with specific needs. I recently spoke to a prospective client where they mentioned they need a quick way to filter through hundreds of candidates pre-funnel and they only want to know if they can write basic code. If you can write basic code, you pass and your resume gets a look. They specifically didn't want anything too deep or difficult. I actually recommended that they use a tool like HackerRank or LeetCode instead of us because it made more sense for their use case.

This is exactly the problem we're trying to solve for. You have too many candidates, you want to be fair to all of them, but you don't have the resources to manually talk to every one of them one by one. You could give them a LeetCode-style auto-test question, but that's not much better than flipping a coin on every candidate. You could also use a service like Karat to do the interviews for you, but before you know it you'll be out $200K. Give them a Litebulb interview, let them do it on their own time, everyone that pushes up a solid solution gets scheduled for a call.

Yeah we generally discourage users from using Litebulb as a "weed out" exercise, first of all because we just don't agree with that practice as a principle, and second because Litebulb is a terrible pre-screening filter. As a pre-screener, the only people Litebulb would filter out are the good devs that have options. ie: if I have companies A, B, and C that I want to check out and C needs a multi-hour take home before I even talk to someone, you can bet your ass I'm not applying to C.

That's not what Litebulb is for though, we're mid/late funnel. As a candidate, if I have to choose between a 4 hour onsite or a 5 hour take home, I'm probably going to do the take home.

In terms of pay, I know some really good companies with a healthy eng culture and heavy pay that do take homes (not as a pre-screener, for sure).

Btw from my experience, resumes are a pretty weak indication of skill. I have interviewed people with 15+ years of experience, some at MAANG, that couldn't comprehend basic systems infra scenarios, and I've interviewed interns that had production-grade code. You need to actually try it out, one way or another, to be sure.

> If you want to test someone's ability to do the real work, then give them an obscure error code from a little-used utility and see how long it takes to find the fix.

This is actually a type of interview we're trying to support! A blocker right now is that once it's leaked, you basically have to throw the interview away. The good part about the "feature building" type interview questions is that it almost doesn't matter if the prompt + codebase is leaked, because your solution is probably still not optimal. "Good code" is super subjective and takes years of experience/learning to master, but an obscure bug fix shifts the end result to a boolean state, which is easy to leak (thus hard to detect plagiarism).

Appreciate this tons, because we'll definitely need the help! Would actually love to connect over email (gary@litebul.io) or directly in a chat (https://calendly.com/gary-j-lin/30min).

Yes! Gumroad.com is not hiring until late April, but their initial coding challenge is async, untimed, and uses Litebulb: https://www.notion.so/Jobs-f43f816013b2405aa41ddefb663a4a38.

Some other companies are On Deck, Mashgin, Dover, SnapEDA, Evidence.dev, getatlas.io, and Okteto.

I completely agree, our marketing message needs to hammer in the concept that a Litebulb interview is pretty much just picking up a ticket off the backlog. We've actually had candidates say that halfway through an interview they forgot they were in an interview.

Also, the biggest problem prepping for tech interviews is that it's a different vertical of skills you're optimizing on. As you said, getting better at LeetCode doesn't necessarily make you a better engineer for the specific job you're being hired on to do. One of our goals at Litebulb is that if you're a good dev, you shouldn't need to prep for any Litebulb interviews. If you do prep, all you're doing is making yourself a better developer in general.

Really appreciate the support here and you make some excellent points! We're trying to make something that already exists just a bit better. :)

1. That point is actually one of the reasons behind Litebulb! For most LeetCode-style tools, getting 100% is easy if you just google the question, copy the top answer, then change up a few variables. For Litebulb, a legitimate submission has multiple commits, so when we do git history matching, no two legitimate submissions will have the exact same commit history, which means submissions either have a <5% or >95% match. The >95% matches are plagiarism and will be flagged.

2. Litebulb is meant to replace the initial LeetCode question + the mid-funnel technical screen. We still recommend doing a hiring manager screen at the beginning and team onsite at the end, but using Litebulb correctly should lessen an engineer's interviewing burden, not increase it.

Thank you! Data science interviews are definitely on our roadmap, but unfortunately we won't be able to get there within the next quarter. Biggest reason for this? Nobody on our team has data science experience! I'm not going to pretend like we know how to interview data scientists when none of us actually have the experience, but we will be spending time collaborating with (if not straight up hiring) a few data scientists to help build out our first few DS interviews.

That's perfectly fair, in fact I don't even recommend using Litebulb for senior positions. The problem for senior openings is always going to be sourcing (there's just not that many good engineers to begin with that meet that bar). We've found that there's no discernible difference between the code of strong intermediates and seniors anyways. For seniors, you're looking for qualities like ability to mentor, to give direction, to architect, to produce specs given vague client demands, etc.

I do however think there is a non-negligible boost in signal accuracy for intern and junior candidates for the first 6 months. Predicting long term growth comes down to things like work ethic, willingness to learn/grow, and resourcefulness, but immediate production-readiness is an important attribute for many companies, especially earlier stage.

These virtual interviewing pain points all make sense, and if I'm not mistaken, https://www.dover.com might have solutions to some of these. Full disclaimer, they are a client of ours.

I certainly hope so too! And yeah getting to know the team is really important as well.

Yeah this is actually why we recommend liberally-timed take homes instead of a sit down 45 minute block. It's super frustrating to get 95% of the way there just to be cut out by time.

Great set of questions!

1. Complex cases:

We can spin up isolated AWS or GCP environments with some resources pre-filled, and the rest is up to the candidate to build. Basically a systems design whiteboard challenge, except instead of a whiteboard, it's actually in a cloud environment. Since "input" will be defined, and specific "output" will be expected, we can then run tests against this environment to test for sufficient robustness. For example, if the prompt is to build a load balanced CRUD api, we can run create resource a million times, then try to read, update, or delete, and measure total response time. For something like testing concurrency safety, we can run a bunch of parallel threads attempting to access the same resource, then test for timeouts (deadlocks) and inconsistencies.

2. A solution that's code complete but has a minor bug that prevents compilation

For a solution like this, code completeness scores are unfortunately going to be 0, simply because there's no way to test for functionality, performance, event logs, if it doesn't compile. However, our static code analysis will still work, so they may still get high scores on code style, code complexity, naming, etc. Something we might be able to add is a way to flag to employers that a submission has high scores for everything except a specific metric, and so we'd recommend manually diving into the code.

3. Cheating

We can't prevent cheating if the candidate gets their roommate to do the interview for them. However, if someone just blatantly copies an old solution their friend sent or they found on the internet, we can check for that. We can run a diff between the submitted git history and all previous submission's git histories. 2 legitimate final solutions might be close (ie 60-70% match), but you'll never have 2 organic git histories with multiple commits be close, (it'll pretty much be either 0% or 100% match). Looking things up on the internet isn't considered cheating, I don't know of a single dev that could survive the day without SO :D

Ah ok so that flow needs to be sharpened a bit. It's actually not a video, it's an interactive demo where you can play around with a small subset of the app in a controlled environment. It's a series of guided DOM snapshots of the actual app itself, but ultimately I realized it's not clear at all.

I'm debating on just not having that interactive demo at all, and instead to be a bit clearer in the rest of the site about what we do, and how we do it.

Yeah for now it's primarily for frontend, fullstack, or backend cloud devs. We have an iOS Swift interview already built but it's not polished, so it's in the backlogs for now, and dev ops and data science interviews are coming up next. Hardware is going to be a tough one though, not too sure how much support we can easily provide there.

This is a really good point, thanks for bringing it up, @dang has already helped us update the HN post title, and we've made that copy change on our website as well! From now on, our messaging will be "coding interviews", not "technical interviews". That is, until we start supporting all the other technical specialties ;)

Engineering management skills are very, very hard to evaluate. From what I'm aware of, you can get a general idea of someone's competence as an engineering manager by seeing how their engineers perform under them vs under someone else. For example, if a team is doing so-so, then a new EM comes in, and a few months later the same team is now suddenly all rockstar devs, this is a sign of a good EM.

Also, the best engineers aren't necessarily the best engineering managers. See https://en.wikipedia.org/wiki/Peter_principle.

We don't believe tech management interviews should at all be automated. You really want these to be as qualitative and human-driven as possible.

We're mid-way through building a V2 of the scorecard. This "Senior - Weak" grading model isn't working, since everyone has different ideas of what a senior engineer is.

Instead, we're going to just expose raw data about somebody's code submission, and let you decide what those numbers mean.

Some examples:

1. 7/10 unit tests passed

2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB

3. Linter returned 10 warnings, 0 errors

4. Cyclomatic complexity counter returned max of 24

5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars

I've thought about this as well, the biggest problem here is that I have Github repos from 7 years ago with very, very, poor code quality, and also just a bunch of incomplete test repos. I wouldn't want that to lower the perception of my current skillset.

If I could select which repos to include, that could be very interesting. Also interesting additions: open source contribution analyzer, Stack Overflow Q&A quality analyzer.

We have something for this in our roadmap.

Currently: candidate does a Litebulb interview, gets a report, that report can be shared with whoever they like, hopefully speeding up interviews at other companies.

Next up: candidate does a Litebulb interview, gets a list of actively hiring companies that use that stack and would like the candidate to be inserted mid-way into the interviewing funnel.

Yup, our clients have brought this up with us as well, which is why we recommend they offer 3 choices for the candidate to do the interview:

1. Hop on a call, do the interview in one sitting together and share thoughts/questions

2. Hop on a call, make sure set up is smooth, prompt is clear, questions are answered, then candidate goes off on their own. Hop on another call later in the day to demo the solution and talk over design decisions.

3. Do it completely async, submit the solution whenever you have time

From what we've seen, this flexibility has simultaneously reduced dropoff rate and increased candidate experience, since we're catering to the candidate's needs. Some would prefer to do it face-to-face, others prefer to work on their own.

I wouldn't necessarily put it that way, primarily because some of our interviews require less than 20 lines of code! It's a bit more about selecting candidates with the right fundamentals. You don't need to write too much code, but the code that you do write, is it clean? Are you following best practices? Is it efficient? Have you thought about how your new service affects the rest of the system, and what the implications are?

I do agree though that some of our frontend interviews are a bit heavier on code + css and less on systems design thinking.

Btw just to clarify, we have a policy where we won't host any interviews that are just open dev tickets to build a feature that will be used in production. Like, we DO NOT tolerate companies using interviews to get free labor.

Yup! That's why we recommend using Litebulb as a mid-funnel tool. Initial screening should be with a hiring manager (or a founder if it's an early stage startup), final meetings should be with team members/managers to get cultural fit. We just try to make the technical coding evaluation part easier.

This makes a ton of sense to me. Seeing how you solve a difficult problem with someone collaboratively is a great insight on how you'll continue working together, because that's exactly what you'll be doing. It's also very valuable to see how someone receives suggestions or criticism and how they respond. We'd actually like to build a lot of these concepts into Litebulb, ie: responding to code review requests?

As an add-on, an interesting question one of our clients asks as a conversation starter is "what's the most technically complex project you've ever worked on?". This question sets up the space such that the candidate is the expert (since they've already worked on it), and it almost becomes a session where the candidate teaches the interviewer.

I completely get what you're saying. We've seen many candidates that have expressed the concern of "if I'm going to put X units of time into this, you should also put X units of time into this". A good approach of running a Litebulb interview is to offer to do it in a sit-down session over a call with the interviewer, or to do it asynchronously. When we did sit-down sessions, we offered candidates a choice to just go off and do it throughout the day, whenever they can find the time. 100% of candidates chose to do it async. As it turns out, trying to code something with an interviewer breathing down your neck wasn't a good experience and made them very nervous.

Note that it's an option, not mandatory, to do it async. If the candidate wanted to do it in a sit down session, then we prioritize making time for it to be there the entire session. The reason we offer this option is because devs that have busy lives often can't easily find a single consecutive multi-hour session to do an interview. They have full time jobs, and might have family obligations at home and over the weekends. Basically, the only way to actually do a full onsite interview is to take a day off work to go do the interview. We're giving the flexibility for the candidate to find a chunk of time here, a chunk of time there to complete the interview.

We tested this above approach, because just as you said, we did see a huge dropoff when we just emailed candidates the interview link. After we offered to hop on a call and to be available for support, we saw dropoff reduced to less than 15%.

Also, a good chunk of candidates that drop out of interview processes are senior engineers that can't be bothered with lengthy processes, which is fair. I wouldn't give a Litebulb interview to a senior engineer to begin with. Debatably I wouldn't give any kind of technical interview, actually. If they have decades of experience leading teams at big or high-growth tech companies, those achievements speak for itself. The interview process for seniors should be more geared towards finding a product-team-eng fit, rather than an evaluation.

Yeah so this is a common concern we've seen. "How do I know your idea of a senior engineer is the same as my idea of a senior engineer?" And the answer is you don't. We're modifying the report to no longer say things like "Strong Junior dev" but rather to just give you a bunch of raw data, and then you decide how that data is interpreted.

Some examples:

1. 7/10 unit tests passed

2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB

3. Linter returned 10 warnings, 0 errors

4. Cyclomatic complexity counter returned max of 24

5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars

Also, the concern with outsourcing to a cheap contractor is a very legitimate concern, because we ALMOST actually did that. Instead, we decided that it was a bad move and to double down on building product + surfacing clear, unambiguous data. That's also why we're intentionally staying away from ML (at least for now), because it's inherently unpredictable.

That would be super cool, and we'll be adding support for new stacks by popular demand. We've already had a few people mention support for Unity, so it's on our roadmap (but likely won't get to it until Q4)!

Thank you, and love that you promote great interviewing practices! What were some examples of prompts you routinely used for interviewing? And any learnings on great (or not so great) interview design?