This might be possible if the binary format encodes a sequence / tree of CSG operations. A binary blob might be difficult to merge / conflict-resolve for humans.
Forget CAD, we don't even have a good way to version control images, something programmers work with every day, besides "use Git LFS and pretend an image is just a pointer".
edit: just saw GumTree mentioned above. Guess I have something to check out...
This sounds extremely nice actually. The more semantically aware the version control system is, the more likely it can gracefully handle merge conflicts. A textual diff is a very low level transformation primitive; composing them doesn't guarantee very much at the higher level.