Apple’s statement on the Consumer Reports battery test | Hacker News Reader

Apple’s statement on the Consumer Reports battery test

marco.org

23 pointsarm9 comments

Stop. Please stop. This is nuts. I know we all hate the new MBP, but this is just not an honest take. Consumer Reports found a bug exercising a feature that nobody in their right mind uses in real life and it invalidated their test results. That's it. Consumer Reports was not trying to test cache disabling. If they had been, they could have easily shouted from the treetops "Oh god, look everyone, cache disabling eats battery life!" But they weren't trying to test that. No sane person would. They were trying to reproducibly test page loads and they failed to get meaningful data because of a not-their-fault problem with their testing methodology.

Consumer Reports did get very poor and inconsistent results from their battery test, which was a reasonable and valid test

Wrong. Yes it was a reasonable test, but their results are invalid.

They disabled the browser cache to put mild strain on the machine. Instead of visiting 10,000 pages to make sure the wifi is used over several hours, they visited the same handful pages and retrieved the whole data each time. If you visit then same pages with caching enabled, you barely use wifi because all data exists locally, which means it would not be a real-world test.

Unluckily, disabling the cache in Safari triggered a bug that drained the battery. Disabling the cache is still a valid method to simulate actual web browsing, in my eyes.

>>Wrong. Yes it was a reasonable test, but their results are invalid.

I'm confused. In what way are the results invalid? The results were as a result of a bug in Apple's software, no?

I'm not the person that you are asking, but I agree with them. The premise of this test is to measure battery life in a common use case. To achieve this, they use an option that nobody apart from testers would use and this inherently makes a use case not so common. This is a methodology error in my opinion. What they could have done instead was to use incognito mode of safari which they reopen each cycle.

I use this feature quite often when testing site development. I'm sure other web developers do the same. I actually run two safari browser windows at once. One with private browsing and one normal. This helps to see how a site functions under different circumstances.

Do you actually refresh the pages as often as they do though?

Nope :)

Let me put it another way with a car analogy. Everyone loves car analogies.

Scenario: You want to test how far your car can go at its top speed on a full tank of gas. You're too lazy to actually sit in the car with your foot on the accelerator the entire time so you go and find a brick to lay across the pedal. Unfortunately, when you place the brick across the pedal, you accidentally, unbeknownst to you, deploy the drogue parachute[0]. You walk away, hit the remote start button in your pocket, and the car takes off. RACING RACING RACING, except, you know, parachuted. Actually you don't know. You didn't do it in purpose. You didn't know it deployed. You didn't even see it, because you were walking away. You only found out later when you got back to your car and saw, oh shit, you accidentally deployed the chute. What a boner. Why was the deploy button so close to the accelerator where a carelessly placed brick could bump into it anyway? What a stupid design mistake THAT was.

Now. Do you call your test valid?

[0] https://en.wikipedia.org/wiki/Drogue_parachute

Then the conversation should be about how Safari was defective, not the machine itself. Implying that the test revealed a problem with the machine is not correct, simply because it didn't. Installing a non-Safari browser is trivial.

Reply on news.ycombinator.com