A second reason is that while mathematical logic is rigorous and precise, unequal mathematical objects are similar to each - even objects that on first glace seem nothing alike. For instance, the two types of inverses you mention, or sets and linear spaces, or groups and tetrahedrons. And because of the diffuse nature of mathematical objects, it is inevitable that the same notation will be used for unequal objects. Because, it advances our human understanding of mathematics to use the same notation for two unequal but similar objects.
This second reason, once understood, is one threshold between the mechanical mastery of the intermediate student and almost artistic use of mathematics by the advanced student.