You simply cannot emphasize the information security enough if all your Internet traffic is audited, censored and manipulated by a number of adversaries supported by (authoritarian) governments and what not.
I preferred to use wildcard certs, which requires a plugin for the dns
Besides, I don't use wildcard certs. I use caddy to reverse proxy a number of self-hosting things, and manually assign domain names to each of them. Caddy can handles many certs just fine.
How's your experience with Caddy regarding memory usage? I am currently serving a static site with 500 MB, but this is with very low to zero traffic.