There is nothing wrong with testing people on simple tasks as part of an interview. This isn’t a long task, likely 30 minutes work!
It’s to ensure what they say they can do is correct. Using AI tools is fine but we need to see an understanding too.
I do have an idea I suppose. Present them with a problem and a solution and have them lead a discussion of the merits of the solution, what alternatives they might propose and so on. There’s some issues to avoid with this approach tho, like presenting a pair with a correct response in mind and then judging people against that standard rather than the standard of “how are they thinking through this problem, the drawbacks and benefits, are they presenting their ideas effectively and respectfully, etc”
It also primes them to expect a certain sort of environment coming into the job, participating in meetings and collaborative solutions instead of just being a code cowboy. Adapt your specific discussion to how you expect them to conduct themselves, probably present your company collaboration guidelines and meeting CoC as a precursor to the meeting, idk I’m just spitballing here. I’ve struggled with this problem a lot hiring people.
You can tell in an interview if they are focused on the interview or the job. If there is never any talk about the job, if it's all about some insanely drawn out interview process you know that's all they care about.
I've seen startups (even some here on YC) who have been hiring the same roles for literally years, they don't know anything else.
No, it means they know how to use tools to solve certain small programming tasks.