There's a lot of people using X, and the vast, vast majority of them don't care much for Musk or actively dislike him.
THe problem with X is that it is (and always was) sort of an echo chamber, there's no average X user, everybody sees very different content based on what they follow. The average HNer probably won't notice the huge swathes of people who are on there just for sports content or their favorite celebrity. Those people usually don't care much for Musk, except when he appears in some funny meme.