It can also be extremely wasteful - if I leave a tab open for hours or days, you'll have to waste your resources and mine to keep pushing me stuff I won't see, while now I just hit refresh when I want to see new content.
In terms of usability, it's often jarring to watch content change when you're interacting with it - that's why even sites that implement real time notifications often have a link or button that you have to press to update the UI. In many cases, doing that completely negates the benefits of pushing.
Also, see the thesis on "Architectural Styles and the Design of Network-based Software Architectures": http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
Polling is a good default for the web; it fits most use cases (content that rarely changes) in a simple and economical way. WebSockets are useful for the exceptions.