You said it yourself, if you want to store a graph (in memory object instancss referring to each other by reference) in a non-graph database (say postgres or mysql) then an ORM is the thing to use.
This is literally what everyone uses them for. You defined their use, and that's what we're all doing with them and finding immense value from them doing it simply.
It isn't rare. It is literally the common case in every web app everywhere. There are more examples of an ORM being good and providing value, in the real world than not.
I'll just carry on using them and finding them valuable, and in the rare corner case their limit is exceeded then take the harder or more complex option.
Best tool for the job.