Mobile Benchmark Cheating as a Service | Hacker News Reader

Mobile Benchmark Cheating as a Service

anandtech.com

51 pointsuluyol14 comments

Towards the end of the article, they talk about high performance modes added by manufacturers and say "The phone essentially goes into a high-power mode throwing away any attempt to be efficient; it’s a nonsensical mode that is unusable in every-day use-cases beyond getting high benchmark scores"

What makes it "nonsensical" ? Are they exceeding the TDP of the phone? If so, then couldn't benchmarks just be made much longer, so that the phones would throttle or fail if they were cheating?

Or is it just about battery life? If so, why is it unreasonable to have a phone plugged in to get great gaming performance?

> why is it unreasonable to have a phone plugged in to get great gaming performance

People don't always game near power outlets.

There's a difference between "don't always" and "unreasonable." It seems like if you can get a phone that performs as well as something that costs 2x as much, and the compromise is gaming near an outlet, or carrying an external battery, that's a choice some people would like to have.

And if it is the battery that's "unreasonable', then I wonder why the review sites don't just factor the cheating into their review by having a category like "battery life running PCMark" or something where they measure how long the phone will run the benchmark.

Customers are not informed, they just use the number - if you game the number you are unethical even if you pretend like its for consumer choice and enhanced information - no matter what that's not how consumers use it and you are juicing a number to get additional sales.

The problem is basically a variant of Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."

Publications want to use these benchmarks as a convenient proxy for real-world performance. Phones have "benchmark detection" and run their power management with different settings when they detect that a benchmark is running, defeating the purpose of running the benchmark.

BTW, I don't think this is about gaming vs. non-gaming workloads (there are different benchmarks for those), or even running plugged-in vs. on battery. My understanding is that some of these "benchmark modes" are so aggressive that the phones would become too hot to hold in your hands. No sane phone manufacturer could use settings like these for real-world usage.

Reviewers should simply say, "we've adjusted these numbers to account for benchmark manipulation", and give those phones worse reviews.

It sounds like it's likely to overheat if you're gaming with it. That's a pretty big problem.

And while making benchmarks longer helps, it doesn't tell you if that phone's new mode would have huge problems in a slightly warmer environment, or a case, and it only works on full-out gaming benchmarks. You could easily still get misleading results for something that tries to be a mixed user workload.

>If so, then couldn't benchmarks just be made much longer, so that the phones would throttle or fail if they were cheating?

This. I think Geekbench needs an additional mode running it at 10min plus instead of the current 2min max.

To your last point, I find it completely reasonable and something I configure for all mobile games I have on my phone. My Samsung Note 8 isn't able to handle smooth gameplay for many applications without tweaking, which I have configured through Samsung's Game Tuner app which does warn about potential overheating concerns. There are actually phone cases made specifically to help with phone cooling these days; tangent, but this reminds me of how CPU cooling gets more advanced as high performance CPUs generate more heat under heavy workloads.

Most popular benchmarks used on mobiles are pretty much flawed.

I.e.Geekbench is highly optimized for Arm but it doesn't use AVX2 on PC, making ARM seem much better than it is.

Geekbench runtime is too small. Good laptop reviewing websites like notebookcheck use many rounds of Cinebench R15 to test if scores go down due to thermal throttling and by how much and after what time.

Something similar can be done on mobile phones instead of just running Geekbench once and writing down the number.

Yes, some manufacturers are trying to cheat but benchmarks used are bad and many reviewers do a poor job.

There's a long history of this kind of nonsense: https://techreport.com/review/3089/how-atis-drivers-optimize...

(Drivers detect the name of the executable as QUAKE.EXE and behave differently!)

That was a really interesting blast from the past.

Isn't that kind of exactly what GPU companies have been doing all the time with game-optimized drivers? And since they cannot just skip work, but merely re-prioritize things, these cheats should be easily detectable by a reduced battery life.

> game-optimized drivers

These generally actually make the games you're playing have somewhat better performance (via game-specific hacks). Applying tweaks to a mobile phone benchmark, but then not applying those same tweaks to other apps is just misrepresentation of actual performance.

Reply on news.ycombinator.com