It's not an occasional wrapper. It's the block of cheese I finished off when cooking dinner yesterday, the container of mushrooms I emptied, the bag of tortillas. Most household trash just changed from "toss in the trash and it gets dumped somewhere" to "bring to someone to check each item for completeness"?
I'm sure we'd end up with a lot less single-use packaging as a result, but we use single-use packaging for a reason and your proposal amounts to a near-ban. Let's not pretend this is a simple "just charge a deposit and littering stops". It's "charge a deposit, discover the deposit is non-viable, and single-use packaging as a whole stops".