6? That can't be right. I don't know how big a GCU is, so the scale could be up to 1 OOM off, but a full redirection of all simulation capacity should let it integrate out further than that.
The error scales up exponentially with the number of (ball-to-ball) collisions.
So if the initial ball position is off by "half a pixel" (=> always non-zero) this gets amplified extremely quickly.
Your intuition about the problem is probably distorted by considering/having experienced (less sensitive) ball/wall collisions.
See: https://www.lesswrong.com/posts/JehyrC6W3YTtdxw6S/a-primer-o...