Your thinking of the firewall that's built into most consumer routers. The exact technicalities of NAT (PAT actually) are not super well defined, and many implementations will not track the destination address.