The play button points at the direction the tape will roll in your cassette player. It had a very practical function once upon a time.
Did these buttons first show up on cassette players? The reel-to-reels I can think of had multi-position switches.
Here's an interesting reddit discussion about it with an image of an older style play button.
And an older more in depth discussion on the same thing
http://www.stackprinter.com/export?service=ux.stackexchange&...
This machine is apparently from 1961, so it must have been quite early in the 60s: http://www.reelprosoundguys.com/images/1962_GRUNDIG_TK_46.jp...
For a very basic and obvious example, possibility to turn right vs turn left signs look different not just because of convention but they resemble the shape of the road.