Hacker News Reader

Comment by bla3 | Hacker News Reader

Weird that the Safari team gets worked up over this when https://apple.com/safari regularly shows numbers based on years-old versions of Chrome.

They just updated it so it's now only Safari prerelease versus Chrome 94, but it used to compare against a one-year old version. Right after the update, it's still comparing prerelease to an outdated stable Chrome version.

Apple is very willing to intentionally make skewed comparisons to make themselves look good, but Apple apparently also gets very upset over an apparently honest mistake where a presentation that even tried to make them look good didn't make them look good enough.

View on HN

So I read your comment expecting to be mad at Apple, but a quick look at the Wayback Machine [1] disproved almost everything you wrote.

> They just updated it

Last month would be a more accurate description [2].

> but it used to compare against a one-year old version

What actually happened is that they hadn't updated the data on their website for a year. That's very different from "comparing it against a one-year old version." They were at no point comparing the latest version of Safari against a year old version of Chrome.

> Apple is very willing to intentionally make skewed comparisons

It looks more like you're willing to make intentionally skewed projection of facts.

[1]: https://web.archive.org/web/*/https://www.apple.com/safari

[2]: https://web.archive.org/web/20211026001113/https://www.apple...

> They were at no point comparing the latest version of Safari against a year old version of Chrome.

They tend to update that page after a Safari update, which is yearly. They tend to highlight benchmarks that Safari looks good in. If Chrome and Firefox then optimize these highlighted benchmarks and catch up on these benchmarks within a month, Apple still has that page show the old data for a year. And for next year's release they again pick the benchmarks that look best for them at that moment in time and keep the numbers up for a year. Even though this update cadence guarantees that the numbers are outdated and misleading for the majority of the time they're up.

> They tend to update that page after a Safari update, which is yearly

So you very well knew Apple weren't "comparing against a one-year old version."

> And for next year's release they again pick the benchmarks that look best for them at that moment in time

They're using the exact same benchmarks as last year, and the numbers seem to be improving for Safari. May I remind you I actually checked the past version of Apple's website on the Wayback Machine?

I'm not the GP author, but I don't think they actually said that _exactly_. I agree it was a bit vague, but the latter point of them using an old version of Chrome for their most recent update was more pertinent. They are comparing a pre-release to a stable in both cases and then not spending any effort to contrast that later.

Maybe both of you are right?

I'm sure Apple's page was factual and up-to-date at the time it was posted, but Chrome's faster dev cycle means that the Chrome data is outdated long before the next year when Apple updates Safari.

That's very different from "comparing it against a one-year old version.

The outcome of Apple not keeping their page up to date is that they were misleading the user by comparing Safari to a version of Chrome that is no longer current, or relevant. The action (or inaction strictly speaking) might be different, but the result is the same.

Apple have enough staff that they could keep a page up to date if they wanted to. They chose not to.

> misleading the user by comparing Safari to a version of Chrome that is no longer current, or relevant

If they had continued to update the version of Safari in their comparisons without updating the version of Chrome I agree that would be misleading. But leaving the comparison frozen because they haven't updated their page is just the normal difficulty of keeping things up-to-date.

(Disclosure: I work at Google, speaking only for myself)

The updated numbers generally seem to favor Safari. The outdated numbers were only misleading in the sense that it benefited Chrome, not Safari.

That's not how that works when Safari gets much more infrequent updates.

What you're implying is highly unlikely since browser performance doesn't change so dramatically every few weeks as to make your claim possible.

https://arewefastyet.com/win10/benchmarks/overview?numDays=3...

It changes over the course of 6 months. So after 6 months the comparison is very outdated, and the comparison wouldn't benefit Chrome.

Multiple updates for Safari are delivered over the course of six months. And note that the graph I've shown you plots the daily performance of browsers for the past year.

I agree with you that there was nothing wrong with Apple posting a page after a Safari release, comparing it with the Chrome version at the time, and then not updating the page until their next significant Safari release.

However, it is true that Safari gets feature updates much less frequently than Chrome or Firefox. Historically, there's the integer update in the fall (13 to 14 to 15) followed by the .1 update in the spring. Those are when features are added. The Safari updates that happen in-between are only bugfixes; some bugfixes may have performance implications but they don't include improvements to language support (HTML, CSS, JavaScript).

Safari 15.1 is already out but I think they only used that version number because it includes a significant UI change (to tabs), it's not an indication that they're moving to a faster release cadence for features.

I mean, it would've been the newest version at publication a year ago. Nobody expects Google to update these slides over the next year either.

Being a dynamic website vs a static slide from a presentation happened at a specific point in time, I think you are comparing apples (err..) to oranges.

They do have a clickable footnote on one of their claims which currently reads:

> Testing conducted by Apple in September 2021 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 13-inch MacBook Pro systems with Apple M1, 8GB RAM, 256GB SSD, and prerelease macOS Monterey. Tested with prerelease Safari 15.0 and Chrome v94.0.4606.61. Performance will vary based on usage, system configuration, network connection, and other factors.

And in August 2021, it read:

> Testing conducted by Apple in October 2020 by measuring page load performance of snapshot versions of 10 popular websites under simulated network conditions. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Big Sur. Tested with prerelease Safari 14.0.1 and Chrome v85.0.4183.121. Performance will vary based on usage, system configuration, network connection, and other factors.

In September 2020, they posted a clickable footnote with:

> Testing conducted by Apple in August 2019 using Jetstream 2, MotionMark 1.1, and Speedometer 2.0 performance benchmarks. Tested on production 1.4GHz quad-core Intel Core i5-based 13-inch MacBook Pro systems with 8GB RAM, 256GB SSD, and prerelease macOS Catalina, and Windows 10 Home, version 1903, running in Boot Camp. Scores represent browsers that completed the test. Tested with prerelease Safari 13, Chrome v76.0.3809.100, and Firefox v68.0.2 on macOS, as well as Chrome v76.0.3809.100, Microsoft Edge v44.18362.267.0, and Firefox v68.0.2 on Windows Home, with WPA2 Wi-Fi network connection. Performance will vary based on system configuration, network connection, and other factors.

Could they constantly update this piece of information? Sure. But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here.

I think it's a pretty high bar to expect that all web pages must always show the latest possible data.

Edit: I actually pulled slightly different footnotes accidentally, since the pages have updated over time. But in general, all testing clearly points to the months where the testing occurred. I think the worst thing I saw was in 2018, I think they had some tests which used data in August and October. Which I don't love. But I didn't notice that happening recently, and they did clearly disclose it.

> September 2021 ... prerelease Safari 15.0 and Chrome v94

It looks like in September 2021 Chrome v94 was in either Beta or Stable depending on when in the month they were testing. [1] Assuming they picked their versions early in the month, this seems very reasonable.

> October 2020 ... prerelease Safari 14.0.1 and Chrome v85

In October 2020 Chrome Beta was v86 and then v87 (2020-10-15), so this is less reasonable: prerelease Safari vs stable Chrome.

> August 2019 ... prerelease Safari 13, Chrome v76

In August 2019 Chrome beta was v76 and then v77 (2019-08-08). Assuming they picked their versions at the beginning of the month, this is fine.

[1] https://chromiumdash.appspot.com/schedule

(Disclosure: I work for Google, speaking only for myself)

They say "prerelease" Safari because it's the version they're marketing, but they test it before they officially release it so the marketing materials are ready. That version of Safari would've been current at publication time.

That's a good point: if they are literally about to release it, it's much more similar to stable than beta.

I tried to look at historical timing, and it's kind of confusing. For example, Apple said they tested "prerelease Safari 14.0.1" in October 2020. But the initial Safari 14 came out on 2020-09-16, an 14.0.1 was a security update released on 2020-11-12 (which I would expect not to have any web compatibility or performance changes).

My overall takeaway is that what Apple is doing here is unremarkable and not misleading.

> But I think they're being pretty up front and clear about what they're saying. And it seems like they do similar testing every year, at a fairly regular interval. I see no evidence that they're doing anything wrong here.

Is this a failed attempt at sarcasm?

Apple takes the small print to an art, in addition to being several pages of scrolling away from the actual claim. Even presented with normal typography here on HN, the text is so unreadable that my eyes try to glance off of it, and I'm a technical users that knows what the text means!

If you want to publish this kind of "snapshot" information, either publish somewhere where that is clear from context (like a blog), or at the very least make sure that the inline content makes it very clear! For example: "In October 2020, we found that Safari caused the battery to last 10% longer the then-latest version of Chrome".

Publishing this kind of claim as evergreen and then pointing at the small print when anyone complains (as if anyone in the actual target group actually read that) is effectively fraud.

I like their honesty though it's in the small print "Not all features are available on all devices."

Reply on news.ycombinator.com