Agreed. That's the sort of rework that I think can be avoided by a good partnership between the PO and the stakeholders in the nascent stages before physical work begins.
I guess the reason why this happens is this weird psychological block in people's minds that software is totally mutable and flexible and therefore can change to whatever you want it to be, even after the fact. You'd never go into that mindset when building a physical structure, but the amount of re-architecture required to enact such changes in software can be just that monumental.