Still struggling, if requests from X were slow, caching their responses would only drop or maintain p99. If those requests were fast, p99 might increase & alert... but if you were caring about performance at a user-level, you can imagine you're dimensioning on users too, and you'd see the p99s for each user hadn't changed, and would shrug and bump the alert threshold up.

If it were me, I'd probably start by caching the slow requests (that are above the p99) first & adjusting the alert thresholds to match the new performance.