Found this[1] via the Wikipedia's Talk page for the robots.txt article. It showcases that early on robots.txt was designated to help maintain bandwidth performance of web servers. Back then it would have been due to bandwidth contention, today it may be bandwidth cost to some operators which robots.txt help mitigate.

[1] https://yro.slashdot.org/comments.pl?sid=377285&cid=21554125