I wonder if particle and laser physicists tend to be excepted from this trend, since these depend on specialized equipment.
Would you be working with different faculty, besides your advisor? Would your postdoc research feel like a distinct research project, or would it feel like malingering continuation of PhD research?
I think the preference for moving institutions is to avoid seeming "stuck." Some PhD students malinger on postdoc work at their home institution, making it seem like they don't know how to finish a project or know when it's time to scope/start a new project.
So, it might prove beneficial to ensure your projects do not fit the stuck/malingering narrative.
I think this bias is unfair, because there are plenty of reasons someone might want to stay in one place: house, family, etc. But like any bias, you can meet it head-on by controlling the narrative instead of letting the biased narrative play out.