Yeah, that really requires calling their bluff. Ideally it's a bluff.
Like you, I’m also not a lawyer, but I did engage the services of a firm when I left. My lawyer informed me in no uncertain terms that my noncompete agreement was enforceable and cited several cases where a) their firm took employers to court to have the agreements tossed and lost and b) unsuccessfully defended people who did what you did and were sued by their previous employer.
In general, “fuck off, sue me” usually isn’t a great legal strategy.
That's different than "totally". It's harder for a janitor, who might not have the means to fight back, to try and call that bluff.