Something I've always wondered: when you sign up for a proprietary "intelligence management" platform like Palantir Gotham, do they give you access to a bunch of OSINT datasets they scrape and manage themselves, to combine with your proprietary intelligence dataset for enhanced understanding? Sort of like how BigQuery has "BigQuery public datasets"?

If not, seems like a missed opportunity for value-add / lock-in.

And if they do provide such data, it'd be nice if one such platform would make either the scrapers, or the scraped data, open-source. It's all just cleaned+normalized forms of already-public data, after all. (Like how the Yellow Pages could always be seen as a cleaned+normalized form of phone numbers listed in public view on shop windows.)

This is on my mind recently after noticing that there's actually no public dataset of "all WHOIS data for every domain on the Internet", despite WHOIS data being something explicitly designed for public querying. Almost certainly, intelligence agencies compile such a dataset themselves in order to fill in some gaps in cybercrime network-analysis graphs. Almost certainly multiple of them do, such that the number of crawlers doing it probably hurts the WHOIS services. Almost certainly the WHOIS services would be better off partnering with one of them to act as an "official crawler" to build such an index for everyone's use.