OP's first sentence refers to limitations of "version control $HOME" ... What are the limitations?
Well, you can't have different configs for different hosts. Other than that, I can't quickly recall what other limitations are, I see none. I really like the simplicity of the "pure git" approach.
My dotfiles repo dates back to 2018, I'm happy user of this git one-liner for the past 6 years.
Though you can use git branches, but I prefer not to
You can? Git is a DVCS, that's kinda the point of the Distributed part (although most people use it as a centralized VCS.
I have not only host specific configs, but also (host specific) work configs. It's really easy to merge/cherry pick changes between them.
I rename .git directory to .dotfiles in my $HOME