That’s a more subjective task of balancing trade offs: how much it saves versus how annoying the change really is. It’s not the job of the benchmark utility to make these subjective decisions.
So that's why CIs are interesting and p-values are less so.