The bigger problem is when these engagement bumps are valid but destroy long term value in the cohort. The auto binge optimization makes viewers more cautious about opening the Netflix app altogether.
This happens with things like fast food restaurants. Subway expanded rapidly, and then tried holding up their sales with discounts. How long can someone eat at Subway weekly or daily before they never want to go there again? The bar is so high, you are way betting off starting a new franchise brand than trying things like removing yoga mat chemicals from your bread or hiring a new mascot who isn’t a child rapist.
Bread is one of only a very few foods that seem to be immune to "food exhaustion", the tendency to want to stop eating something you've been eating. (Milk is another, but I have a feeling that only applies to lactose-tolerant people.)
So if you varied the fillings, probably indefinitely. How long can you have cereal for breakfast before you never want to eat cereal again?
I believe the veggie option is still yesterdays hummus crudely spread around a bun with bell peppers, black olives, an raw onions. The meats are cafeteria tier, in fact I think the meatball sub recipe is the exact slop served in my old dining hall.
They undermined their masterful subliminal marketing on the entire millennial generation by raising the $5 foot long sub to near chipotle bowl price points, and in my city at least cheap excellent street food is already on every corner.
I only ever settle for subway, it's not a destination.
A/B testing causes way more bad decisions than good ones because people run the tests wrong and they interpret the data wrong.
But I've found explaining that to people to be an uphill battle. They get sold on an "objective" way to optimize the UI and then don't want to hear that the method is actually very subjective when not done correctly. And it's hard to execute and interpret correctly so every A/B test I've seen ran has been very subjective.
Meaning have one person who talks to all the stakeholders and every type of user. Who just knows your product backwards and forwards. And thus can drive what changes you need to make to push things forward.
What exactly the means depends on the nature of the product.