Is this an overreaction, or a subversive call to improve the decode perf of JPEG-XR? I'm certainly on the performance-is-a-feature bandwagon myself, but 1% ux impact is pretty small. And shouldn't we assume that decode perf will improve in the near future? Is there any reason to assume that this problem is permanent and unfixable?
The best you can say here is that it’s within the margin of error, in which case the effect is so small it’s literally unmeasurable. In that case the null hypothesis still wins, and you should not switch.
> Is there any reason to assume that this problem is permanent and unfixable?
IE11 as tested in the article is not getting anything other than security patches. The users who use IE11 are probably also the users most affected by poor performance since they most likely have the oldest hardware.
> The best you can say here is that it’s within the margin of error
What is preventing JPEG-XR from becoming faster and a net ux positive, tomorrow or in the future? Isn't the best I can say that today's results might be a bug or fluke or a quick and dirty implementation, and not a permanent problem?
> IE11 as tested in the article is not getting anything other than security patches.
The title and conclusion both said we should avoid JPEG-XR on the web, not that we should avoid it on IE11.
You're right, the article focused on the problem with IE11. Is it already a net positive on Edge? (The article didn't say.) Should the shim to make it work in the old browser that fewer people use really override the potential benefits on the newer browser that more people are already using?
Does it seem reasonable or unreasonable to make future web decisions depend on an old browser that's on life support today and has small and declining market share?
And even if you don't, taking a 1% hit to your conversion rate and hurting user experience just because you want to use a fancy new image format is probably not a good idea.
The may be a spurious figure.
"we were able to statistically verify a -1% negative impact on core business metrics including conversions"
Ultimately though the issue is more fundamental: you can't make assertions about performance without publishing actual figures. All we have to go on is the author's assertion that the test methodology was sound and the results were out of the bounds of chance. That's not really enough.
I have worked with many, many teams who earnestly believed they were running "scientific" MVTs, with solid statistical analysis free of bias. They weren't.