1. inability to represent arbitrary query results. Results get mapped back into 'domain-objects'. Now you've removed yourself from the extreme expressiveness of the SELECT statement as well as GROUP BY, pivot tables etc. etc. The best you can hope for is domain objects that are only partially initialized with values.
2. lazy loading of attributes - certainly configurable, but if you rely on SELECT statements being generated while you are navigating a 'persistent' object, you will run into unexpected performance issues. To solve those, you then start to eagerly load domain data for certain access patterns. More annotations, more configurations and more ... stuff - where a nice SELECT query would have sufficed to get the data you need.
3. Troubleshooting is a pain. Especially if you rely on dynamically building queries. Have fun getting the actual SQL that was run and then translate back and forth between SQL and the query builder syntax to optimize it - if you can even optimize it.
4. Doesn't treat data as data. First of all data coming from SQL queries is stale the moment it arrives in your objects. Trying to pretend otherwise and having an object graph that pretends to be the living counterpart of the state in your DB is foolishness.
5. No/limited control over writes/updates. The moment you rely on your ORM to create UPDATEs and INSERTS to reflect the state of your object graph back into the DB, good luck with your chosen locking strategy and good luck optimizing those. Pessimistic locking is the best worst you can hope for.
Stay away from ORMs (as in Hibernate and its ilk) and treat data as data. Use structs/records.