I guess that's fair. But: if you buy into this idea, there's a much, much better thing to do: look into Canarytokens. Canarytokens, unlike port knocking, are criminally underused. Really, you should do something similar for your off-port SSH service; don't actually _run_ SSH there, just run a stateless unprivileged service that spoofs a bit of SSH protocol and generates loud alerts.
The networks I'm describing aren't "ideal" or "highly secured". I am describing table stakes. While I was at Latacora, most of the clients we engaged with were already at this level of maturity when we joined up.