Git will deduplicate all of your binaries across branches (and if you are clever, across repositories, but that's another story) so worst case you will only have one copy of any binary file no matter how many times it appears in your history.
That is not to say that git may be a poor fit for your current project+organization for other reasons but a blanket assumption of ("repository would be too huge") is not generally accurate.