Hacker News Reader

Comment by devmunchies | Hacker News Reader

> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regression in the metrics gives us a signal to abort the update and revert

How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

View on HN

> How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?

That's assuming that the update contains a UI/UX change. A lot of updates that will roll out won't include that, they'll be fixing or optimising things.

You acknowledge that fact and if you're making a severe change you either run a longer test or you break up the large change into manageable chunks (Google Maps is a great example of this).

For something that is effectively binary and can't be changed incrementally the only way is to extend the evaluation window or make a call with limited information.

regressions due to "users are used to" usually goes away after a while, you'll see a temporary dip in the results.

If you are convinced it's better for the long tun then what's the point in measuring ?

Have you worked at a large company before? You don't!

Reply on news.ycombinator.com