If you don't use a b-tree, you still need some other in-memory data structure like a hash table to hold the pointers. "Direct pointers" don't help you if you have to read a node from disk before you can dereference its pointers.
In my case, I was working with a graph consisting of about 10M nodes and 35M rows of edges (parent-child relationships, where a parent can have many children and a child can have many parents). Many of those 10M parts weren't actively used anymore, so a minority of nodes had a majority of the relationships. I don't have access to the data anymore, but I recall being able to retrieve the transitive closure on a subgraph of ~10k nodes in 5-10 seconds, and the entire graph in about 7-8 minutes. This was on a small, possibly underpowered SQL Server instance. It's possible that Neo4j would do it faster, but I decided it was not worth moving all my data into a different database and introducing that ETL latency, to improve upon performance that was already acceptable for the business need.
I have heard anecdotally that SQL Server's CTEs are more performant than PostgreSQL's, but I don't have evidence of this, and it could be outdated if Postgres has made improvements since I heard that.