Sharing for anyone who didn't know there is a very good dataset you can use now. If you don't have a nvme ssd in your computer, I highly recommend getting one for fast i/o.
http://commoncrawl.org/ http://commoncrawl.org/the-data/ http://index.commoncrawl.org/
related.. Mark's blog is amazing and worth more than any data science degree imho.
https://tech.marksblogg.com/petabytes-of-website-data-spark-... https://tech.marksblogg.com