The real kicker is labor cost per ton. Bigger ships don’t have a proportionately bigger crew. And sailboats were quite labor intensive.
Sailboats are much taller than fossil fuel cargo ships of the same size. A lot of harbors have bridges in their path.