> but people won't be able to see the artifacts

But that is correct and why any proper study should first train the participants on the type of artifacts and trade-offs. Like the reference to JXL choices: think its creator mentioned that AVIF might look nicer for faces because it was "smoothing" them but it was a worse representation of the original. So the study participants would need to be conscious of that trade off and be clear in terms of what they're asked to evaluate - whether the result looks nicer or whether it represents the original better.