The big advantage of remote work is that you can easily try out someone as a freelancer first before bringing them on full-time.
It's mind-boggling how broken our hiring practices are, yet how easily it could all be fixed.
There are certainly cases where that is true, but my experience leads me to believe that they are the marked exception, and not the rule.
Another is to assess your ability to brainstorm and talk people through your ideas. If it was about coding ability alone you wouldn't need the whiteboard - you could just be sat in front of a computer. I can think of several people I've interviewed who have produced very good code, but have lacked the social skills required to integrate with a large team - even working remotely. So I don't think the solution is quite as easy as you think.