How so? Your ISP is still the channel over which the integrity information is sent. Why couldn't they just swap the SRI hashes as well?
A browser could render a similar security warning to what it already does, if the signature doesn’t match or if the hash is wrong.