It's fine if core migration happens during the invalidation loop - the core migration itself surely must wipe the non-shared cache levels thoroughly, otherwise nothing would work.
EDIT: Actually if the big and little cores are used together, and not exclusively, then this might still be an issue, yeah.