Having a high average comment score helps a lot (maybe a bit too much as you've said) but AFAICT the average is only among relatively recent posts.
I'm pretty sure it is something like the average of your last $X posts that matters rather than average score of every post you've ever made. So while it may still result in a "rich get richer" situation, it isn't quite as dire as it would be if your entire post history worked into the average.