I think it filters dupes within a certain time frame. You don't want important front-page news at a site being considered a dupe because something was big on the front page a year ago.
(actually that's a lie, I saw it mentioned in the CSV challenge thread. But that was just a few days ago, so the point still stands)