If your public address gets a random packet then the packet is addressed to your router, not one of your local devices. This is insecure if the router is providing some insecure service on that port, but otherwise it's irrelevant to this discussion.
The problem comes when your router receives a packet addressed to one of your local devices. Many people in this thread believe that if the router is NATing outbound connections, it'll drop this inbound packet, but that's not what happens -- the packet will actually be routed to whichever local device it was addressed to, just like any other packet would be.
This should be fairly obvious, because the only thing NAT does is change the apparent source address on outbound connections. It does nothing at all to inbound ones (and you can demonstrate this to yourself by setting up a test network in some VMs and NATing traffic between them with Linux and iptables).
(You can configure a NAT to do things to inbound connections using port forwarding rules and/or a DMZ, but those rules only apply to specific inbound connections. Connections not covered by those rules are treated as normal, and "as normal" for a router means normal routing.)
NAT isn't a firewall, because it doesn't drop connections. All it does is rewrite addresses on connections, not decide which connections can and can't be made.