Processes get interrupted and rescheduled between tens and hundreds of times per second (depending on OS, the scheduler may take over somewhere between 100 and 2000 Hz, although it doesn't necessarily deschedule every time). This is how SMP works at its most fundamental level. Every time that happens, it involves a full flush from the hardware perspective. It's always context switching, savings the registers, clearing the buffers, etc.

When a process gets descheduled and then later rescheduled, there is no particular reason to reschedule it on the same core it was on previously. Desktop class hardware doesn't have NUMA. Last level caches might have locality-dependent performance characteristics, but this is usually pretty nuanced, wildly variable between hardware microarchitectures, and not accounted for / not worth accounting for in the OS's scheduler, since there's a good chance that data won't be around next timeslice anyway.

Binding a processor to a single core affinity-wise is not changing any of this.

If you really have a throughput-bound process that needs to squeeze every single microsecond of compute and/or you're super latency bound for some reason, you schedule a process with what's called realtime priority, which changes the scheduling decision of WHEN to deschedule it. Not every operating system has equivalent mechanisms for this, but that's more in line with the phenomena that you're describing. But a process may well cause itself to be descheduled anyway by doing something that puts it to sleep. You have no control over this.

In reality, looking at which core a process is on is mostly the wrong way to be thinking. You're looking at this on a 1-2 second window (the amount of time it takes for Activity Monitor / Task Manager to update its UI). A full second of time is an absolute eternity to a process. Large amounts of work happen on the order of microseconds and quicker. And in that 1-2 second snapshot that Activity Monitor or Task Manager gave you, the process probably spent time on EVERY core -- and you're just seeing wherever it was in the brief moment when Activity Monitor / Task Manager updated its UI.

> gamers have been binding to single processors/cores for years for that reason, it's a huge cause of microstutter type behaviors

It wouldn't be the first time gamers prescribed placebo tweaks because of magical type thinking.