That's because you're using the unsupported "compact" mode. Switch back to normal mode and the speaker only shows on hover (in addition to the "PLAYING" text on the second line, which is omitted in "compact" mode).
I won't be switching to "normal" mode as long as I use Firefox in a smaller window, displaying chat applications, videos, or whatever as I go about other things.
Mozilla seems to assume people will use Firefox in a maximized window, which isn't true for me 90% of the time, but I am getting exceedingly tired of having to fix the interface with custom styles every 2-3 releases.