It's worse than that. The way our minds work, labels are part of our self-image. Once we attach a label to our identity it becomes part of who we are and we often try to conform to the expectations of the label.
Biphobia, which is just a modern label for a collection of attitudes that predate our modern concepts of sexuality, would still be a thing if no one called themselves "gay" or "bi" (or pan), just as same-sex attracted people got lynched for going against some people's beliefs long before there were words for their attractions.
Labels aren't the problem. Dogmatic adherence to narrow concepts of gender and sexuality is the problem. It's helpful to identify the way I do so people who also identify that way can find me and compare experiences with me.
Labels for gender and sexuality are best treated as a way of finding others like yourself to increase self-knowledge and protect yourself against people who want to hurt you because of the feelings those labels represent.
Labels have their downsides, but the upside is an enormous communication advantage over other species, one that brought us the ability to foster life and evolve other species with us. In effect we are now _better, more nuanced labelers_ than we were in the past, which is really the goal--eliminating labels stunts our ability to identify and solve problems, so using them better is the only way to go.
TLDR: "We label too much" itself is a label for a problem; even though it necessarily leaves out nuance, that's also the very factor that gives a labeler more leverage. To move up and away from that problem, we will need to find new labels that get us better traction in that area. The amazing thing is that there are so many billions of ways of labeling a problem, each with respective advantages/disadvantages.