Hacker News Reader

Comment by ryanisnan | Hacker News Reader

Forgive my naivety, but wouldn't a simple way for a search engine (like Kagi) to avoid falling victim here to detect affiliate link programs? There's got to be a small handful of patterns for affiliate link tracking:

1. Domain Interception & HTTP redirects 2. Tracking codes embedded in the URL directly

View on HN

It looks like Kagi does this[1]:

> Kagi surfaces shopping results featuring unbiased reviews and no affiliate links to help you identify the best product across categories. Top results include discussions focused on helping you find the best item to purchase - you are not bombarded with affiliate links and ads. Continue to scroll and you will see product comparisons across multiple vendors so you can pick what best suites you. Kagi's shopping search will always return a detailed discussion of which product to buy not a competition amongst advertisers promoting where you should buy. Kagi is focused on providing you the best results to make an informed decision not polluted by affiliate links and advertisements.

1. https://help.kagi.com/kagi/features/shopping.html

Kagi is better than Google, but just yesterday it returned the top link to Amazon-linking “check the price for this item on Amazon right now” affiliate spam. So it’s not perfect.

Oh, OK. I thought it was perfect.

I think it’s pretty good. I understand sometimes we root for the good guys like Kagi and look at their work with rose tinted glasses and even think they’re perfect, like you did, but it’s not really feasible.

A pattern I’ve noticed from people not working in tech, or more junior people in tech, is that they have a thinking that software can ever be perfect. If either of those categories describe you, I can tell you from experience no software is ever perfect!

In adversarial environments, this is true. In non-adversarial environments, software of a certain defined scope can at least get very close to perfection.

Ultimately a site can get around affiliate / referral detection with internal links that redirect though I suppose. So the crawler would need to follow and track all links to detect this. Also sites will surface different links / content based on user agents, so I don't know how effective this is.

Maybe there is room for one search engine that does this - for paying customers or as an investment to be better

Kagi is a paid search engine to avoid the manipulation that Google welcomes. To avoid every trick and targeted tricks it may be cost prohibitive. Fortunately Kagi isn't popular enough yet for targeted manipulations to have a noticeable impact.

I tried a search and found a site with amazon affiliate links in their featured "Shopping" snippet

https://i.imgur.com/vPRpBYJ.mp4 (yes the recording is very broken but you can still see)

Maybe the affiliate stuff is still there but ranked lower?

Sure but it sounds to me like it shouldn't be featured in the "Shopping" widget when it has 9 affiliate links?

When I hover over the "i" icon it says:

> Information provided by Looria, Amazon, Reddit and other sources. Shop confidently with Kagi - our results have no affiliate links.

The sites featured in the "Shopping" widget were:

- Reddit

- RTINGS: has 8 affiliate links (just from one article I picked, it was featured several times)

- NYTimes: has 47 affiliate links (just from one article I picked, it was featured several times)

- Techgearlab: has 9 affiliate links

Even if you ignored the widget, the first result was a site with 7 amazon affiliate links.

On second read, I think what the Kagi doc means is that they don't provide first-party affiliate links (i.e. if you click on an Amazon.com search result it's not using their affiliate). They will still show you search results that have affiliate links embedded (but presumably don't affect or negatively affect ranking).

Currently, Kagi has (if you hover/click the shield icon to the right of a result) an indication of the information it knows about a website (as well as a way for you to rank it higher or lower for yourself).

One of these is "ads/trackers". I imagine that it would be feasible for this to include some of the more common affiliate URL types, or third party lead/affiliate tracking bounce hops like awin.

Clearly there will always be some amount of ability to "defeat" this kind of measure by obfuscating links, but eventually the user needs forwarded to a link that has a referral parameter or a site that sets an affiliate cookie or similar.

The "tracker category" also can give a bit of extra information - things like "invasive fingerprinting, advertising"

I am working on a search engine that checks a site for affiliate links from known providers and demotes sites that have a large amount of them.

In addition:

* it demotes sites with popups (think newsletter sign ups)

* it demotes sites that block (or complain about) ad blockers

* it demotes sites with a high number of ads and favors sites with no ads

* it demotes sites using certain sketchy ad companies.

* It demotes sites that have paywalls

* It detects possible link networks and flags them for human review/removal.

* sites with RSS feeds get promoted.

* There is a toggle to hide all sites with ads or external trackers, but it is still WIP (The whole project is).

There are many other features. No idea if I am going to make it public, I created it to update my skillset. I actually thought about setting up a nonprofit and making it open source, but I haven’t decided.

Very interested, any way to follow you?

Me too.

I don't know if this would be a very long-term solution if the big ones (ok, google) did it. Advertisers would catch on very quickly, and some legitimate review sites which might get funding through affiliate links unrelated to the product being reviewed would lose out to straight-up paid for "reviews" that are funded wholly by the manufacturer and just don't use affiliate links.

What would be the difference between an affiliate link program and a web ring?

I don't know if discovery is actually a bottleneck to be automated away. It might be the fun part. I'm thinking back to the Napster approach where you could browse other people's libraries for music ideas.

Reply on news.ycombinator.com