Move fast and break things itself doesn't imply you shouldn't go back to patch it up and make it cleaner. People should be moving fast so they don't end up in endless discussions regarding or spend too much time on creating a foundation for a solution that doesn't work, and to inhibit perfectionism. People should also be transitioning from "make it work" to "make it good" once it works, prior to delivering or finalizing it.
This is largely a problem with people unable to shift practice according to the context, lazy developers and managers thinking "it works" means "ship it and never look back". Unfortunately, there is no cure perfect cure for lack of foresight and willingness to listen to the guy saying PoC code will cause problems at some point down the line.