Robots.txt was created long before Google and before people were thinking about SEO:
https://en.wikipedia.org/wiki/Robots.txt
The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.