OP mentions using robots.txt to avoid crawling but even google ignores this now correct?
1. https://www.searchenginejournal.com/google-robots-txt-noinde...
1. https://www.searchenginejournal.com/google-robots-txt-noinde...
FWIW, what I specifically have in robots.txt is
User-agent: *
Disallow: /
which seems to work well for me so far (i.e., I do not find my house documentation site on any search engine).[1]: https://developers.google.com/search/docs/crawling-indexing/...
I think the point of it was that you could tell Google to crawl some pages (for links) but not index them?