One problem I have with this approach is that it requires you to pick a threshold after which the response is "too slow." This number can change a lot over the course of an application lifetime, and would be hard to pick objectively anyways.

Median latency -- perhaps with (the smoothing-effect of) a rolling median -- would be more robust to outliers without having to resort to hardcoding of "too slow" thresholds. It would still require the human to connect the dots (e.g. median latency of >200 is "too slow") but it's an improvement on mere average response time for reasons noted.