Creepy Website Similarity
memex.marginalia.nu
memex.marginalia.nu
If unfamiliar, google sets would return other terms related to the term the user input. E.g., input: cisco output: brocade, juniper, hp, arista etc.
@marginalia_nu, if you are ever looking for another project to do with your scraping data, a sets.marginalia.nu could be super useful.
On topic, the top results returned for my queries were spot on. Some others were initially surprising, but probably correct. Creepy website similarity or an on-the-fly web ring creator.
I wonder if you could get anything useful from just upvoting/downvoting entries in your own feed for a few days/weeks this way, or if much more signal would be needed for a real algorithm.
Although I guess you could use link data (or some other source) to base suggestion off.
We can keep a generalized utility search as the default, but I wish I could somehow tell a search engine "show me stuff you'd show to my friends from this community". I.e. what would HackerNews' community search group look like? What would this friendgroup that for some reason includes a lot of people who work in botany but that isn't the defining feature of the group. Maybe they all also happen to like board games, are allergic to social media, and are weirdly well-read on continental philosophic traditions.
I think the key is that people have exit. My main gripe with things like TikTok and twitter is the algorithm tries to optimize results for what you'd like most. But people don't really work that way. People grow over time. People join different communities, put on different hats, try different identities. Sometimes it happens over time or sometimes people are in phases of their life where they're constantly in between multiple communities. I would feel more comfortable with the "profile" these algorithms made for me if I had some sort of say in it and could switch between at will or do an "incognito"/fresh slate mode whenever I want
Unfortunately Marginalia itself requires javascript. Could you please do something about this?
I would use Marginalia a whole lot more, except that if I do more than two or three searches an hour, all I get is this:
search.marginalia.nu
Checking if the site connection is secure
Enable JavaScript and cookies to continue
search.marginalia.nu needs to review the security of your connection
before proceeding.
Also, what does any of this have to do with email spam? We're talking about HTTP(S) here, not SMTP, right?
1. How does marginalia build up the 10M bit vector for a site it has not seen before?
2. How do you do a cosine similarity on a 64 bit bloom filter?
======== Example: https://explore2.marginalia.nu/search?domain=zombo.com
Results seem quite good.
1. As I understand a 10M bit vector was created for zombo.com target, features were domains(10M of them), and values were booleans of whether site links to zombo.com .
So on the fly to save space 10M bit vector was converted to a 64 bit bloom filter. So no false negatives and a certain percentage of false positives - so far so good.
2. Those 10M sites all have already precomputed bloom filters. If I understand correctly features would be 64 now and we calculate cosine similarity on those?
2 -- The bloom filter is used to not calculate the cosine similarity directly. It's used to eliminate cases where the vectors are trivially orthogonal.
It's this part in the code example
if ((a.hash & b.hash) == 0) {
return 0;
}
hash = 0;
if (values.size() < 128) {
for (int v : values) {
int bit = computeHash(v) % 64;
hash |= 1 << bit;
}
}
else {
hash = ~0L;
}
I don't perform the calculation on the fly, but rather I've pre-calculated the similarity between each domain and store up to like 75 results given they have good similarity. This took about a day.
Maybe if I'm looking for even more I could put my feeds in and get more content.
My toy, which relied on scraping <ul> and <ol> lists of people's favourite bands from personal websites, had almost nothing to go on by comparison. last.fm might have been able to do something good with the data they collected, but they seemed to die on the vine once they were acquired.
HREFs sound like much less manual work.
Edit: > ...able to find similarities even among websites that Marginalia doesn't index
Yeah, I joined LinkedIn specifically because even without my input it was way too good at figuring out my actual social graph. With my input, it no longer is :)
Lagniappe: https://www.tandfonline.com/na101/home/literatum/publisher/t...
It reminds me of the hand-wringing about the youtube recommendation algorithm. Billions of people discover amazing stuff is not a story, a few people discover disturbing stuff is.
The problem with the similarity technique is that it can find associations between sites that do not link to each other, regardless whether and how they try to hide. It's enough that someone thinks they are related.