Hmm..at least you reasoned that it is a "problem" (and I would say it's more analogous to VLOM switchover than turbo lag ROFL).
For now, if you infrequently do full builds (and instead do incremental builds) and are mainly dragging things around, adding small features, etc. you're better off with less cores, even if those cores do run slower.
I don't really want Apple to "care deeply" us [slowing our phones], so I'mma not help illustrate further because a proper solution could involve a massive Engineering task (maybe Nuvia knows) and even with $3T market cap, they pretty much cheap bastards unless it comes to food courts and office chairs.