Yes, and git doesn't play video files either, because it is a version control tool for source code, not random binary data.
I most definitely used Git with binary data without any issues. It makes no assumptions as far as I can tell. It's just not great with very large binaries.
I think you cannot make a completely generic indifferent system in finite time. Maybe your system is optimized to efficiently work with 5 1TB files, or 1 million 5KB files, or just nicely handles a reasonable amount of text files in a neat way while being not extremely complex or hard to maintain & deploy.
If Git does version control better than other control systems but only does it for text then I might be fine with that, I think it's reasonable to have a scope.