At its core, this case comes down to "is this fair use" (there's no claims that it's not copyright infringement, IIRC). And while fair use analysis considers the four factors, in actual practice, the four factors are merely a way to explain the reasoning behind the gut decision, which usually boils down to "did the copyright owner deserve to get paid?"
The fourth factor is "the effect of the use upon the potential market for or value of the copyrighted work," which is where the monetary aspect is going to be considered. If the judge is pointing out that the publishers actually saw market growth as the Internet Archive expanded its lending, that could signal that they are going to find the fourth factor in favor of the Internet Archive--and that would utterly destroy the publishers' case here.