You usually end up losing a lot of flexibility when using ORMs or frameworks. If you don't do things exactly the way the ORM or framework forces you to, you'll be in a world of pain. Of course, there will always be situations where you need to customize things to your particular case. ORMs and frameworks are notorious for making this far more difficult than it should be. I've seen teams waste more time twisting a framework or ORM to handle a unique situation than it may have saved them in the first place.
It's naive to think that ORMs and frameworks necessarily have fewer bugs, or better quality, or better documentation, or more tests than custom-written code. I've worked on enough teams that have had a hell of a time due to ORM or framework bugs. It's worse when you're using a closed-source ORM or framework that you can't easily fix directly.
Somewhat related to that, for any sizable or complex application, it's just not possible to completely avoid understanding the inner working of the ORM or other frameworks that you're using. Eventually you'll need to dig into them, whether it's to fix a bug, or to figure out how to do something that's poorly documented, or perhaps to figure out performance problems. This can be very time-consuming, often exceeding the effort needed to write, debug and test custom code.
If you do need to make additions or fixes to an externally-developed ORM or framework, this can very easily cause headaches down the road when it comes to upgrading to a new version of the framework or ORM. You end up having to choose between using an old, patched version of the framework that lacks critical features or bug fixes, or you can upgrade and try to re-apply your fixes/additions, or maybe even moving to a whole new framework altogether.
Having to deal with issues like these can make custom code look very compelling.