There's no evidence, for instance, that the fourth amendment was meant to prevent customs checks at the border. Inside the US, customs checks would be unreasonable searches, but at the border they have always been legal.
Phone searches are nasty, invasive, and not like a customs check at all. That's why we have courts to distinguish them.
The only reason it doesn't say that is that it's inherent in the term "warrant". A warrant is exactly the legal authority to use force on behalf of the government contrary to the rights which the law would normally guarantee, which is a prerequisite for performing any search or seizing any property (absent the owner's consent, of course). Any law purporting to authorize searches is a warrant, though not necessarily a constitutional one. There is no such thing as a legal search or seizure without a warrant. There are only unauthorized, and thus illegal, searches and seizures, and unconstitutional warrants.
If "reasonableness" were enough on its own—in other words, if obtaining a warrant were not a necessary part of the process of conducting a search—there would never have been any reason to place further restrictions on the issuance of warrants, and the entire second half of the 4th Amendment would be void.
I understand the pragmatic argument, but not the implicit presumption that an upstanding citizen would come home with contraband.
Here's an explanation of reasonable suspicion in comic form (part of a larger chapter on 4th amendment law): https://lawcomic.net/guide/?p=1833