It's also important to consider that reducing or eliminating tariffs is effectively a tax cut for large corporations. Whether trade increases economic efficiency is kind of beside the point, even if we were to agree that economic efficiency is something that national policy ought to attempt to increase.
Eliminating income tax would probably increase economic efficiency as well, but we don't do it because the money to pay for government operation and services needs to come from somewhere.