Sampling is another useful tool but I don't see any sampling in the API for this. If you do sample though you could end up with inconsistent or divergent state if you just sampled a flat % of traffic, so in some contexts it makes sense for sampling to be determined per-user/tenant/etc. rather than per-request.
This is pod based so you could easily slice by pod. If you have a pod with a lot less traffic, more impactful traffic, or that is harder to debug (example: pod that periodically aggregates and exports metrics but you believe it's dropping data sometimes) you could just enroll that and have some kind of dev shard of a portion of the stack.
I could also see this being very useful if you believe it to be too big a liability to record actual user requests (or implement something special to scrub out sensitive stuff) or let devs peek too close. You have a way of investigating some classes of issues without necessarily looking at sensitive data or requiring extensive back and forth in a support case.