Port knocking as it's usually done is easily sniffed. Perhaps using a dynamic TOTP-like time based seed to constantly rotate the ports might help. But it sounds overly complex.
It feels very like a "key under the third plant on the right" kinda thing. Not a solid security measure.