It's not perfect, but Norway has a tax on drinks containers that is set up so that the starting point is a tax on drinks containers. However any drinks containers you recycle is offset against the tax. Then there's a government endorsed recycling scheme that handles all but the collection for you, but requires that you participate in collecting containers irrespective of whether you sold them or not.

So the default assumption is that if you do your job, the tax is more of a deposit. If you don't do your job, you, or rather your customers are still paying, and your customers can drive down the cost of your competitors by helping increase their tax offset if you make it a hassle to return things at yours.

By creating the presumption that you ought to be able to collect and recycle most of the recyclable products you sell (return rate for cans and bottles is well over 90%), the tax/deposit can be set fairly high. High enough and you create secondary businesses taking the hassle of returns for those who can't be bothered (don't want to return your bottle in Norway? odds are someone who needs the money will fish it out of the trash to collect the deposit), and there's a strong incentive for businesses to take back anything they sell subject to such taxes/deposits and deliver them to whichever scheme is approved to offset against their tax bill.

This sounds relatively close in principle to an implementation of what you're suggesting. with penalties etc. implemented basically by tallying up the tax per unit sold and then reducing the liability per unit recycled, so the penalty is simply the default if you fail to recycle.