If there was a similarly simple and effective solution, either in 1994 or now, he would have suggested it. But he didn't. We have to guess that he'd want something involving a declared robots-rules-'link' elsewhere.
But layering robots-rules as a 'link' from the headers of the root page, or markup tags (if the root page happens to be HTML), still requires an initial investigative hit to a site -- and if at the '/' page, probably involves more bandwidth than a compact robots.txt or 404.
Assuming a hostname is a unified 'site' is not perfect -- but in 1994 and now, that's the only inherent and well-defined unit of administrative control provided by the HTTP protocol.
By going with this easy-to-understand, easy-to-implement convention, webmasters have had enough freedom to opt out, and crawlers enough freedom to collect, to enable ~15 years of dazzling growth in powerful search applications. And robots.txt files built to the 1994 standard still work, handling 99.999% of what webmasters need to communicate to crawlers.
That success satisfies my design aesthetics.