>Following that logic, we might as well continue trimming along the x-axis until nothing is left.

The data is still useful in determining the statistical confidence of the set, but not reliable for representation as its representation completely changes the understanding conveyed by the data set. So the data stays in, and the X-axis doesn't get trimmed. Only the data point, as represented in the last, non-robust interval, gets trimmed.