Comment by abakker | Hacker News Reader

Comment by abakker | Hacker News Reader

I have a version of fixing this that I would personally enjoy a lot. Leave google alone, let it crawl the web, prioritize what it wants to via algorithms. But, give me a version of that which ONLY surfaces results from discussion forums (including SO, Reddit, HN, etc). For most of the stuff where I am actively searching and not just looking stuff up, discussion forums of motivated, self-selected contributors have the stuff I need with the context I need. It used to be that blogs had answers, but that media has been categorically ruined by SEO.

Now, one of the deficiencies here has been examples. Try this: "best miter saw". you will not find any websites that actually discuss the answer to this question, despite it being a product category with a lot of price variability and performance tradeoffs (weight, capacity, power, cord vs cordless, accuracy).

Nearly any product reviews for large purchases follow the same pattern unless consumer reports has decided to dig deep (e.g. washing machines).

How about guitar strings? Sandpaper? Printers? google's algorithm has allowed profit motivated websites to displace the commons to too great an extent.

How can they tell it’s a discussion forum? Does the scraped search spam that copies stack overflow content look enough like a discussion it fools their heuristics? Is it a manual process (in which case you can bet it “doesn’t scale” and won’t be built) this is the problem they face. Literally nothing is simple given the size of their dataset, the scope of their user base, and the adversarial nature of the very world around them generating new data they must work with in order to do the job.

There’s definitely an element of “we got our profit so fuck it” with respect to the search engine advertising business and Google’s incentives to make search quality better, but that doesn’t change the difficulty of the underlying problem. If I wanted to pay Google $5 per search for super high quality results, even $50, they can’t just make this product better to get my money, they are fighting an ongoing war against adversarial SEO which prevents this from ever being better than stalemate at best or more likely due to economics, the slow slide into declining quality we see due to the SEO side having more money with which to pay for engineering brainpower.

You may be interested in searX - it refers to data sources as Engines; you have the ability to run your own instance (or use a public shared one) and only enable engines you want results from (reddit, stackoverflow etc.). Build your own meta-engine recipe, basically.

https://searx.space to learn / get started. Find one and visit it, click Preferences upper right then follow your schnoz.

The results include all the SEO spam that infected Google, ie. SO clones. How is it better?

The GP commented about using curated sources. One can disable all those (google, bing, etc.) and choose to only enable results from reddit, wikipedia and so forth in searX, which directly queries based on a config inside the project.

My current solution for this is to just tag `site:reddit.com` to the beginning of Google searches. A Google search for `site:reddit.com best miter saw` has a lot of relevant results.

Marketers/SEO people are starting to infiltrate this as well, but since they can't control and SEO the content on Reddit nearly as much, this still works pretty well for now.

Of course! this is a tip I got from HN years ago. it gets old to add Reddit, PracticalMachinist, Fine Woodworking, etc. I really want a proxy for user generated content where nobody got paid to write it.

Fun story: I knew a person who worked for a home building website part time. She got paid to write stories on home renovations. She had never done _anything_ she wrote about. Mostly, she gathered up other blogspam and recycled and rewrote it without citation. Sometimes she went to forums, sometimes reddit, sometimes youtube. But, the universal part of it was that she had to produce 2x pieces of content per week endlessly. Just for a local LA builder. Most of the content wasn't "wrong" but, it also wasn't exactly incisive and didn't include any details that would have been useful. Instead it was just filler. The worst part is that it consistently improved that company's search ranking.

Content farming needs to die.

https://search.marginalia.nu has a very interesting approach to this:

Use tracking especially and JS generally as a weight in ranking so sites that contains much of any of these needs to be exceptionally high quality to float to the top.

This means sites with limited ads and tracking, typically enthusiast driven pages float to the top.

Now always when someone discusses a novel way to combat webspam someone will immediately counter: if this becomes popular SEO hackers will immediately start doing this.

Well - if reducing page size and removing tracking becomes a leading SEO trick I can deal with a bit of SEO hacking :-)

Yet for some reason I feel Google won't start using this very simple metric :-)

Another big factor in what I do is prioritizing the opinions of certain indieweb sites when ranking domains, basically a segment of the graph consisting of humans with a particular dislike for seo spam. This makes ranking manipulation much less effective.

Good to see you in this thread! I just added Marginalia to the recommended search engines in a new search tool I'm building, to get programming answers faster. The search assistant builds queries for specific sites with "site:targetsite.com , programming question" (that comma is not a typo). When doing a query like that I get no results but these warnings:

/!\ The term "," contains characters that are not currently supported

/!\ Try rephrasing the query, changing the word order or using synonyms to get different results. Tips.

sample: https://search.marginalia.nu/search?query=site%3Astackoverfl...

Please make your engine ignore the comma, it shouldn't affect the search.

Either ignore the site:... expression or filter sites accordingly.

Thanks a lot for creating Marginalia!

Ignoring comma seems doable, I'll have it fixed in a few days, currently away from my work computer.

site:-queries are supported, but only at the first domain level. (e.g. site:marginalia.nu; not site:search.marginalia.nu). I might tune it so that it strips subdomains automatically, that is pretty trivial.

I've made the changes to the syntax (as well as fixed a bug that broke site:-queries), although I think it's probably not a great search engine for searching big mainstream websites like SO, the crawler is designed to de-prioritize them over blogs and what have you. Disregard what I said in the other comment about the domain names, use the full domain name.

e.g. this works fine: https://search.marginalia.nu/search?query=site%3Amemex.margi...

If you just do a site:-query without any search terms, you'll get a dump outlining the crawling status of the site. Like this:

https://search.marginalia.nu/search?query=site%3Astackoverfl...

You can see it's crawled 80 pages but not gone much farther since SO ranked so low.

Reply on news.ycombinator.com