What are some examples of sites that are not "most real world sites".
Is HN a "real world site".
What percentage of sites submitted to HN are "most real world sites". (IME, it's a minority fraction.)
Why not just delete or change the user-agent line in scrape.go before compiling.
(Personal experience: I have been successfully retrieving information from the www for decades without including a user-agent header. The number of sites I have found that require this header is relatively small. It does not rise to the level of "most".)
HN replies usually fail to include even a single example.
Similarly, do examples provided for scraper programs and libraries ever include "most real world sites".
OP is making a very well known day-one point about this topic. It is somewhat surprising that the library doesn't offer a way to dynamically set it.
Edit: and this isn't even getting into stuff like Tls fingerprinting, header order, etc
Anyway, I’m not interested in a pedantic debate. Most (but not all) people who have attempted to scrape any popular site, or unpopular site behind Cloudflare at above-minimum protection level, in recent years should know exactly what I’m taking about.
Edit: In case you’re not aware, many of the sites I have in mind are/were capable of defeating puppeteer-extra-plugin-stealth. User agent is only the most basic thing, like level 1 in a hundred-level dungeon.
Real world example, I was looking at scraping unjobs.org for a friend the other day. The need for JS rendering turned the job from 15 minutes of requests and beautifulsoup into a full-blown session with selenium, geckodriver etc.
I'm not saying the linked framework isn't nice, I've not looked at it, but tools for simple scraping are plenty and easy to use.
There's a lot more that a new framework needs to do to distinguish itself. I'd love something that makes JS rendering, proxy-rotation and catchpa solving easier, in a nice package I can deploy myself.
Below is a simple demonstration using only common UNIX utilties and minimising the number of TCP conections. No browser. No Javascript. No Selenium. No Geckodriver. No proxies.
Step 1 requires 40 TCP connections and completes in under a minute. Step 2 requires one TCP connection and completes in 10min. (NB. The connection minimisation used here, what RFCs used to call "web etiquette", is not possible using a popular headless graphical browser.) 1.htm is 1.5M, 2.htm is 40M
# step 1
x="some user-agent string"
# e.g., https://raw.githubusercontent.com/51Degrees/Device-Detection/master/data/20000%20User%20Agents.csv
n=1;while true;do
test $n -le 40||break # unjobs.org only shows listings 1-1000
case $n in 9|17|25|33)sleep 10;esac # need a delay after every 8 requests
echo "GET /new/$n HTTP/1.0@Host: unjobs.org@User-Agent: $x@" \
|tr @@ '\r\n' \
|openssl s_client -connect unjobs.org:443 -ign_eof -servername unjobs.org
n=$((n+1));
done > 1.htm
# step 2
n=1000;
grep -Eo -m$n /vacancies/1[0-9]\{12\} 1.htm \
|tail -$n \
|sed "\$!s>.*>GET & HTTP/1.1@Host: $x@Connection: keep-alive@>;
\$s>.*>GET & HTTP/1.1@Host: $x@Connection: close@>" \
|tr @@ '\r\n' \
|openssl s_client -connect unjobs.org:443 -ign_eof > 2.htm
IME, Wordpress sites do not require a user agent header. Contrast, for example, with Squarespace sites which do require a user agent header.
IME, if send a user agent header with a particular value, then some sites will block, depending on the value. Whereas if _do not send_ a user agent header, then almost all sites will accept.
The so-called "developers" who publish code and commentary about "scraping" almost always include a user agent header. Usually they use fake values. They try to guess the "correct" values to send.
Im not referring to sending fake values. Im referring to not sending the header at all. No "spoofing" is involved. This works for me, for decades, across thousands of websites.