This was not intentional. I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1136300.

I've had this problem with Readability before (the icons appeared by default to the left of the back button, which I didn't like; they kept reappearing). I want to say I solved it by going to about:support and resetting things.