do you know how far madagascar is from easter island? if you're talking about mediterranean and river travel, yes you're right. but the pacific ocean + indian ocean are utterly massive.
It's easier to sail to and from Madagascar to much of Asia than it is to sail to Madagascar across the Madagascar channel.
[0] - https://commons.wikimedia.org/wiki/File:Corrientes-oceanicas...
At the time there was no Suez canal. Going from europe to China by sea would mean go around Africa which was a challenge by itself.
Although there was a Suez canal at various times in antiquity, albeit one between the Red Sea and the Nile, not the Mediterranean directly.