I've personally run tests and seen tests achieve statistically significant (and very large) results that weren't merely fixing bugs or moving buttons within reach of a user. It could be things like adding a modal, changing copy (yes, literally just adding or changing words on the page), removing content, changing an image, or reordering funnel steps.
Think about that: by merely changing some sentences, your website could be 10% more successful next month. But you won't know whether it's working without a/b testing.
You don't just create tests randomly, but based off of an understanding of the product, goals, signup funnel, etc. which may involve talking to users, but ultimately verifying the change is for the better through a/b testing.
I would agree that there are a number of people running a/b tests that don't understand the math or know what they are doing, but that isn't an indictment of a/b testing.