Consider even just the chance that at some later point you want to remove the secrets. Now you have to rewrite history and also force that on all contributors.
you also have the option to cut a new repo. it's a small team, you don't have behemoth inertia.
(secrets in repo if your code is open-sourced is indeed not a good idea at any scale. it's also a bad idea if your secrets cannot be easily meaningfully rotated, like putting your social security number in a secret.)