Where Brahms is concerned, very few composers matched his mastery of the craft. That being said, Bach is one of them.
I think Bach is so consistently regarded as the GOAT because (AFAIK) no one has matched his output while maintaining the high level of quality he did.
Also, he wrote St. Matthew's Passion which is arguably the greatest piece of music in the entire western canon. I can think of very few other reasonable contenders for that spot.
As far as composers being greater than Bach in any particular area, I 100% agree. Brahms was probably the best chamber composer, Chopin wrote the best music for piano etc...