That's why you don't use cosine activation and always limit yourself to Lipschitz functions, I guess?