It's kind of surprising to read about the same experience and even reaching exactly the same conclusions.
Anyway, I've recently experimented with checking the build directory in only in production/staging branches to avoid constant merge conflicts and the repository bloat. My approach is to have those build-excluding lines in .gitignore file present in development branches and to comment them out in production/staging branches. So far it works well, but it's sometimes quite confusing to other people on the team.