"but keep doing stateful tracking of outbound connections just like a NAT would, and drop everything else on the floor"

Here are two important behaviours which come with "just like a NAT would".

1) UPNP - a protocol which allows an application to request it be exposed to the internet

2) Hole punching. E.g. two hosts send packets with matching ip/port values to cause a direct connection to be established between them

Applications don't really do those things on v6...

And, FWIW, ISPs should be giving residential users at least a /56 - if not a /48. Sites should be able to have enough address space to route within the site and still use the features of IPv6 which require a /64. Route aggregation and routing table size is the constraint of the v6 world, not address space.