Interesting idea, I like it.

The biggest problem I see is that this is going to (eventually) be used by spammers.

I can send a mail with a FROM anywhere with a link. That link will be advertising for something, and your service will mail it off to some unsuspecting person.

This means your mail server's reputation will end up being black listed everywhere.

Enforcing DKIM/SPF validation would help a lot (but not perfect). I verified that you are not currently doing any validation today. I was able to spoof my gmail address from my residential IP. You do not have very long to fix it before it gets abused.