Take a well known function: sin(x). Call it G
Now make a function H that makes hairy version of functions. H takes a function f, a slope m, and a probability p.
H returns a version of the function where every value is the close to f(x), and p percent of the points have a derivative of m. The remaining points have the job of undoing the derivative of m and jumping back to f(x).
If you are dealing with real numbers then things can get very hairy without ever deviating from f(x). You could turn p up to 99.999999% and m to a crazy slope.
(I am not a mathematician, I don’t know if these musings are misguided)
My point is that if you had a hairy function, you wouldn’t be able to use gradient decent on it, because a derivative takes the slope at a point, and these slopes are misleading. However, you could use a numerical derivative, that plots the slope between a point and another point separated by a non zero distance D. The hairiness would not be detected.