Hmmm. Very interesting. I wonder if you could speed up your clever approximation still further with this approach https://arxiv.org/pdf/2205.09120.pdf
Or, in simpler terms - if you have just ones and zeroes, and minus ones, you can remove zeroes from calculations, but that's it. No good method to figure out which ones are more important than the other ones.
Also, there are no bits left to store positional information when bucketing.
There are some paths that could be explored in this fashion, but it would require a redesign of the algorithm from the ground up.