If the big files are in git history then I think the only way to remove them is to do a full git filter-branch.

https://git-scm.com/docs/git-filter-branch

But this will sever history with all cloned copies, much like rebasing master would.