More small pages means more opportunities to show ads. And in the case of search, a larger portion of the first page can be given to sponsored results. :-(
Somehow the web has evolved to a place where the poles of infinite scroll and excessively short pages are two of the great annoyances.