The problem, for me, is that it's not as easy to understand, read, or write. By this, I mean that most of the time that I'm reading an arithmetical expression, I don't really care what it means. I am only reading it because it's easier to just read it than to look for where it ends and continue. It's fluff. Not being as comfortable with prefix, and probably never being as comfortable with infix, it's not as friendly to the eyes.
When I need to understand it, I'm basically converting to infix-- I think in infix, not prefix. The parentheses in the infix expression are mental and implicit, and I think my mental notation is more than adequate. Prefix form gets in the way here.
And obviously, it's more of a pain to write fully parenthesized arithmetic than not.