If they respected the robots.txt, the quality of the search engine would be much lower unfortunately. Many of the most visited websites refuse to be indexed these days. Brave search gives good result because Brave doesn't care.
I understand Brave wanting to have the best results for their search, but that doesn't seem to be their primary motivation if the lawsuit is to be believed. It seems to be they're doing the whole "chad third party scraper meme"[1] and then making big bucks selling those results to the other AI companies like Perplexity that have already gotten in trouble doing that same thing in the past.
I wonder in the future if they're be a whole underground/darkweb or p2p/distributed scraping economy to get scraping data from people/countries that have no restriction on where they get their data and have no qualms about getting sued about it, like Anna's Archive. Or some kind of local client people can run on their home computer that works as a peer to peer service, slowly scraping websites from their residential IPs and getting paid a few pennies to submit the results to a huge service that collects it then resells it to the big AI companies.
[1] https://knowyourmeme.com/photos/2615826-twitters-view-rate-l...