It's not binding, but it's a hint they may listen to where you can say things like if there's a certain URL structure where your server would keep serving content infinitely and just waste the crawlers time and resources, you can hint to them they maybe they should avoid certain URLs, but hey if they wanna waste their resources crawling that anyway be my guest!
I think the main issue is with scrapers that don't adhere to robots.txt/X-robot headers.
Google doesn't either, they just don't index the paths written in robots.txt but bots still hit them.
No, because robots.txt is a friendly request and not a demand. It can never be a demand.