My guess is that a massive majority of websites will never notice any performance difference.
I don't think the difference in performance between Nginx and Caddy would mean you need 3 servers versus 1 of the same spec. But of course, you need to run your own benchmarks on your own config to determine which is best _for you_.
Benchmarks cannot be done generally, because the config drives so much of what happens at runtime, and everyone has different needs.