For one thing, it's easy for A/B tests to show something that looks good in the short term, but has longer-term negative effects. If your goal is to iterate quickly and stop your test as soon as you think it shows a positive result, you'll never see that.
But more to the point, there's no guarantee that the "user engagement" metrics Twitter cares about are actually a proxy for what users really want. If it takes me more time and more clicks to find the information I want -- enough to be annoying, but not quite enough to drive me away from the site -- that probably gets counted as "engagement".