Hacker News Reader

Comment by snailmailman | Hacker News Reader

I’ve always been tempted to set up SearXNG, but all the search engines already make me do captchas and anti-bot checkboxes every few searches. Wouldn’t a meta search engine run into the same issues? Or if I had a local server scraping search results pages wouldn’t I instantly get banned from all the major searches? I guess I already feel pseudo-banned from them so maybe it’s not actually a problem.

Just a few minutes ago I dared to do a google search and Google refused to show me results because apparently my iPhone with latest iOS seems like an “automated search”.

View on HN

No, they do a pretty good job at getting around it. I only run into issues with Brave/Startpage, Google/DuckDuckGo/Wikipedia all work fine. And when you're fusing together multiple sources getting CAPTCHA'd is not actually a big issue, it's more so a thing to fix later.

I would say also -- if search engines are making you do CAPTCHAs every few searches, this is probably an issue with your exit node. I notice a sizable increase in CAPTCHA barriers when I'm on my VPN (most annoyingly Google/YouTube, which sometimes will serve you an infinite number of CAPTCHAs w/o letting you through).

By contrast, my SearXNG instance is making ~100 searches a day from a known good IP address (being my VPS), so rate limiting is less of a problem. You can also set up API keys for many search engines.

I wish it was a problem with my vpn/exit node. But I get these captchas when I’m not on a VPN. I assume it’s just because I don’t login to Google and I enable tracking prevention stuff in Firefox? But that’s really only on my desktop. I get them on my iphone too which is vanilla iOS safari. I get captchas and the cloudflare checkboxes extremely frequently, and sometimes cannot get past them at all.

I saw the articles about residential proxies on TVs recently and honestly have been worried something like that was the cause, but I’ve kept an eye on my network traffic and I don’t notice anything suspicious. Pretty sure I’m not part of a botnet or proxy or anything; but maybe other nearby IPs are.

Maybe I’ll give SearxNG a try. Searching is bad enough as it is, it can’t make it too much worse.

On the off-chance that you were searching in incognito (that's where I run into that the most), iOS now lets you pick a different search engine in incognito mode. Personally I use DDG there.

i dunno how searxng avoids it but i've been using it regularly lately with a custom agent, and i've never had a blocked search

Reply on news.ycombinator.com