And how do you prevent someone from cheating if there's no engineer there on the interview?
1. Complex cases:
We can spin up isolated AWS or GCP environments with some resources pre-filled, and the rest is up to the candidate to build. Basically a systems design whiteboard challenge, except instead of a whiteboard, it's actually in a cloud environment. Since "input" will be defined, and specific "output" will be expected, we can then run tests against this environment to test for sufficient robustness. For example, if the prompt is to build a load balanced CRUD api, we can run create resource a million times, then try to read, update, or delete, and measure total response time. For something like testing concurrency safety, we can run a bunch of parallel threads attempting to access the same resource, then test for timeouts (deadlocks) and inconsistencies.
2. A solution that's code complete but has a minor bug that prevents compilation
For a solution like this, code completeness scores are unfortunately going to be 0, simply because there's no way to test for functionality, performance, event logs, if it doesn't compile. However, our static code analysis will still work, so they may still get high scores on code style, code complexity, naming, etc. Something we might be able to add is a way to flag to employers that a submission has high scores for everything except a specific metric, and so we'd recommend manually diving into the code.
3. Cheating
We can't prevent cheating if the candidate gets their roommate to do the interview for them. However, if someone just blatantly copies an old solution their friend sent or they found on the internet, we can check for that. We can run a diff between the submitted git history and all previous submission's git histories. 2 legitimate final solutions might be close (ie 60-70% match), but you'll never have 2 organic git histories with multiple commits be close, (it'll pretty much be either 0% or 100% match). Looking things up on the internet isn't considered cheating, I don't know of a single dev that could survive the day without SO :D
You could always have someone look at their work. Or better yet walk through it with them to try and ascertain their thinking. This is what I do with my team, multiple times week, in my actual job. Seems better than immediately excluding them
We are not talking some CRUD app where it is trivial to make each component work or a Leetcode problem that you can spit out in 20 minutes.
In the latter case, is it possible for a developer to operate that way and still produce a great solution given more time? I suppose so - but in my experience, the people who are not in the habit of incrementally checking their work are virtually never strong developers.
There's also a time management question here. If a person knows they only have 45 minutes, that should create extra pressure to start testing earlier. Someone who still waits until time's almost up before checking anything is probably a fairly extreme case of this antipattern.
Something I've been thinking about, not necessarily in the interviewing world, would be to have a simulation of distributed systems for non-production code: in theory I think you totally could have a simulation layer for databases / server loads and test how your code performs in such environments.
I guess it exists in production: but haven't seen it for either a) learninga bout distributed systems, or b) for interviews