I have one possibility of what might be going on.
When you have multiple joins that aren't one-to-one mappings you get all of the different combinations in the result set.
Let's say that you have 5 people, each with 5 addresses and 5 phone numbers. The result set wouldn't have 55 rows (5 people + 25 phones + 25 addresses), but 125 rows (a row for each combination). The difference gets larger as the numbers get larger (for 100 addresses and phones per person, it's 1,005 vs. 50,000). Your ORM also needs to figure out that semi-mess and when you get to large numbers of rows, that's a lot of rows to process.
John | Address 1 | Phone 1
John | Address 1 | Phone 2
John | Address 1 | Phone 3
John | Address 1 | Phone 4
John | Address 1 | Phone 5
John | Address 2 | Phone 1. . .
So, you're potentially pulling back a ton of excess data. Phone 1 will appear 5 times in the result set and then the ORM has to de-duplicate that. Likewise, Address 1 shows up 5 fimes in the result set and John shows up 5 times in the result set and the ORM needs to deal with it. Likewise, it's a lot of excess data over the wire and a lot of excess data to even process into a hash.
Now, Hibernate has a way around this. It's been a while, but it's something along the lines of setting it to fetch related entities in batches (EDIT: doing slight Googling, but not checking it in code, it's the @BatchSize annotation and @Fetch annotation that you'll want to look into). In that case, Hibernate basically does a "SELECT * FROM people" and then a "SELECT * FROM addresses WHERE person_id IN (list_of_people_ids_from_last_query)" and a "SELECT * FROM phones WHERE person_id IN(list)". That way, Hibernate gets 5 people records plus 25 address records plus 25 phone records (55 rows in total) and there aren't any duplicates that need to be weeded out, etc. It's three queries which is more than 1, but a lot fewer than the looped queries without the cartesian product.
This is the behavior that Rails' Model.includes(:relation) and Django's Model.objects.prefetch_related('related_set') use. There was a discussion (possibly years ago) on the django-developers list about why select_related('related_set') didn't work on reverse foreign keys and this is the reason why.