It's for controlling search results; blocking overzealous crawlers is a convenient side-effect.

robots.txt also blocks indexing of non-html files.