Unless you do deep packet inspection, people may get clued in to your "mundane destination / port filtering" setup and start using other protocols like bittorrent on that port.
I actually saw this back when I ran an open network, in trying to prioritize bittorrent below ssh and interactive traffic. Some remote seeders will operate on port 22 or 443 because that's the only way they can reach the outside world. If an "Open Wireless Movement" using a standard VPN setup took hold, a similar thing might happen with that port.
Of course, without DNS only the savviest users would initially be able to exploit mundane destination port filtering, but it's only a matter of time before one of these users puts together a tool for the masses. So I still wonder about the plausible deniability of such a setup.