Well performance really isn't something I would point to at all. While I believe it can become problematic with great scale (e.g. those expensive CGNAT boxes), it's more the architectural and connectivity aspects that are problems:

Architecturally, dealing with two different address spaces is a complication. A flat address space is easier to reason about and more flexible. This is amplified by the fact that IPv4's RFC1918 address space is very small. Have you ever had to merge IPv4 networks from different companies? It's rarely the case that they don't have overlapping ranges.

Connectivity-wise, stateful IPv6 firewalling is very different from NAPT in IPv4 because the applications cannot reason about the ports being used (without rendezvous servers).