Ask HN: Where can I find a text file with a set of representative URLs?

1 pointbrad02 comments

My google-fu is failing me today.

I'm looking for something similar to an a list of phrases but for URLs.

A phrase list looks something like this: https://www.phrases.org.uk/meanings/phrases-and-sayings-list.html

Is there a standard text file with various URLs of random web sites?

Ideally some URLs would have the path, query and fragment components.

With the chrome extension Link Klipper you can export all the links on a page to csv, then make whatever kind of list you want.

I'm looking for more of a large set of links (say 10,000) that are representative of a wide range of URLs used over the web. Theoretically I could use a web scraper but it's better to see if there's already something out there first.

Reply on news.ycombinator.com