I think your mistake is to take “free will” to be something that exists or could exist (ie: in the same realm as physical forces). Which is, of course, just as “outside of physics” as the concept of a spirit or God.

“Free will” is a cultural-linguistic construct. How could it be anything else? The very concepts of discrete individuals, let alone agency, is effectively arbitrary as far as physical forces are concerned.

The “things” that free will concern are already many, many layers of abstractions (and cultural-linguistic construction) away from the individual deterministic interactions of what we currently understand to be physics.

And, like all abstractions, these are leaky and imprecise. So trying to model or analyse their behavior in terms of deterministic physical interactions has always seemed misguided to me.