This is a negotiation tactic by the NYT to drive up the licensing price. Period.
The Napster/Music Industry analogy has no resemblance to this situation.
The only meaningful question that might be answered as a result of this is, what permission and access rights do crawlers have to content that is publicly and legally available.