Since XFS works by multiple allocation groups spread across the volume, that would require you to migrate data out of the area you want to reclaim, which would require a pretty significant reshuffle of the volume contents.

It's easier to just backup and refill with a single set of reads and single set of writes than it is to suffer the huge overhead of all that random metadata updating and seeking.