Because I can install modified hardware that performs more complicated attacks like sending the PIN for your phone or your iTunes password over the network? And since it’s a hardware modification, it’s persistent forever and nearly impossible to find. Malicious hardware is not part of the threat model that phone manufacturers design around and it’s cheaper/simpler to pair components to fight against that attack vector than to come up with protection mechanisms (eg restrict the memory that the component has access to and various Hw measures to make sure you can’t fuck around in the analog domain - it’s really really complex to get right and a flaw means your entire run is vulnerable until you fix it if you even can without doing major redesign work).