Users are most likely not “considering” them at all. We’re not machines going over one tile at a time and generating a score. Most of the time you’re looking for something specific and just trying to find it.
This type of metric is the worst. I really hope this data-driven fad dies down and we start designing with human factors in mind again.
If by human factors you mean basic heuristics based on human intuition, that fails too. It might satisfy 50% of viewers but there's too much diversity in taste for this to be a good model.
I'm not defending the metric as described, but there's obviously good data driven approaches and bad ones. The trick is to find a solid proxy for the business problem and to try to weed out any dark side effects by applying it.
The issue is many-fold: 1) formalising multiple half hour interviews into something you can A/B test is non-trivial, 2) given the logistics of these interviews the sample you end up with is small and therefore bias.
Having a user testing session is relatively rare, saved for special one-off products/features. Actually having feedback from those sessions distilled into product feature proposals, and getting them into the product roadmap is even rarer.
In the meantime, hundreds of A/B tests based on metrics like these would have been ran and resulted in a permanent change to the product. It’s just so much easier for everyone involved to trust “the data” vs something that requires nuanced interpretation.