I mean the main reason is that robots.txt is pointless these days.
When it was introduced, the web was largely collaborative project within the academic realm. A system based on the honor system worked for the most part.
These days the web is adversarial through and through. A robots.txt file seems like an anachronistic, almost quaint museum piece, reminding us of what once was, while we stoop head first into tech feudalism.