ORMs and query builders can also constrain the implementation from a performance and flexibility perspective though.

Your anecdote and reflections are all absolutely true in my experience, but I've found just as many problems with off-the-shelf ORMs.

The key is the clean boundaries/interfaces and keeping things simple - leaving the bridge intact so that your enemies may retreat.