Seems strange that we'd optimize for making modifications easier, when such modifications are much fewer and farther between than the everyday living that fills the time between, during which people value a solid wall with no gaps or seams.

Of course, not everyone values the same things to the same degree, and a homeowner could cut a 4x8 piece of drywall into smaller pieces and have more easily removable panels like you're imagining.