The reason it makes the clothes more expensive is that US workers are paid more, work in better conditions, and have more rights. That's worth spending more to support.
Unless China (or another supplier) chooses to pay the tariffs to make their products more competitive. I've heard allegations that they do sometimes export products below cost.
Almost literally, as the goal of the tariff is to change buyer behavior. At least, of the tariff even has a coherent goal.
If there are no tariffs they won't cost more. They could use other means to onshore jobs without causing a huge shock to the economy, then raise tariffs.