Mojeek: Independent search engine with its own spider
mojeek.com
mojeek.com
I searched for one of the keywords that rank one of my blog posts on top in Google and it shows someone that copied my article word by word -years later- instead of my site.
Not sure how the algorithm works but you may want to look into original sources and copycats. In this particular case it's not a critical issue (just me, some random guy, losing a visit and ad revenue) but at a larger it could be more serious.
Overall, I think the name isn't too catchy, but it's nice and unique. You definitely need a logo or something more identifiable. For now, the light green / lime tone is not that nice to look at, I think a more "boring" grey-tone or something blue would be nicer.
Nice work! Looking forward to this becoming a big deal!
What a BLESSING! All I ever find on Google/Startpage/Bing/DDG is last weeks product listings, sponsored reviews and influencer "content". Let's embrace randomness!
EDIT: i just managed to find some serialz (for the lulz). Almost like the old days. What a blast!
This might explain why GigaBlast has a problem:
Because of bugs in the original Gigablast spidering code, the Findx crawler ended up on a blacklist in Project Honeypot as being “badly behaved” (fixed in our fork). That meant quite a bit of trouble for us because CDN providers, which are a very powerful hubs for internet traffic, put a lot of weight on this blacklist. Some of the most popular websites and services on the internet run through services like Cloudflare and other CDNs – so if you are in bad standing with them, suddenly a large part of the internet is not available, and we weren’t able index it.
extract from: https://web.archive.org/web/20190921180535/https://privacore...
Mojeek follows the robots.txt protocol so if a site doesn't want to be crawled by MojeekBot we respect that wish. There is also a generous crawl delay between pages on the same host.
Generally a 'badly behaved bot' will ignore robots.txt or hit a site too hard with requests.
Our bot uses a specific user agent which you can verify via DNS. https://www.mojeek.com/bot.html
What's the order of magnitude of this delay? milliseconds? hundreds of milliseconds? seconds? I'm curious what's considered 'polite' in this realm and how the various parties come to form opinions on this.
User-agent: bingbot
Allow : /
Crawl-delay: 10
Does this mean your spider is a fork of Gigablast? Is there some additional interesting technical information about how your code/infrastructure is set up?
https://blog.mojeek.com/2019/12/100-server-build-and-install...
We'll be writing about our tech stack in our next FAQs series; 3 of 4, this is 1 of 4:
https://blog.mojeek.com/2020/11/frequently-asked-questions-a...
I remember Gabe Weinberg mentioning on here that DDG became profitable pretty early on, so I'd be interested how easy it is to meet that threshold and how sustainable it is to create search competitors!
This means that they just need to sell enough advertising (which I think bing provides / forces them to use) so basically for DDG, more visitors = more revenue and breakeven means that the traffic pays for the costs they have.
If I were to create a privacy first search engine, I would likely not go general purpose and would instead try to focus on specific use cases first for specific audiences (that would be willing to pay). In practice this would look more like an extended version of DDG's instant answers than a Google-alike, but I think the world is convinced that general purpose web search is a) necessary and b) the only valid approach to finding answers to questions.
Many search engines already exists for specific use cases:
- www.bailii.org for UK law related searches like court verdicts
- tachiyomi.org for searching manga
Etc. Generally speaking, search engines that cater for a specific use case are simply called "aggregators".
You will always find:
><Search Engine> is <bad/good> and the results are way <worse/better> than <Another search engine>.
><Search Engine> is only a frontend for <Parent search engine> <but/and> does(n't) do their own indexing.
>Results for <Topic (usually technical)> (in <country>)? on <search engine> are garbage. Search results for <purpose> are poor.
What you will see as a response:
>You should use <search engine> instead! It uses <different parent search> instead so the results are better!
>I agree, I didn't like <search engine> so I had to go back to Google.
What you won't see:
>Nowadays I only use <search engine> for some search, most of the time I use <specific source(s)>.
>Maybe you should use specific searches for specific purposes?
The two that you cited are very valid, and I'm sure many people use specific sources for specific things (most doing it unknowingly via apps). The main thing seems to be that (on especially desktop) the single purpose all-encompassing web portal seems to be the way to find something. The phrase "Google it" is a catch all for 'do your research'.
DDG also uses Yandex for web search (mostly Russian content).
I've used DDG near-excusively since 2013.
Mojeek has been built from the ground-up meaning we do not depend on anyones technology. We have our own servers, crawler, index, ranking and so on.
So unlike most we are not using Bing, Google, Yandex or anyone else's. Our road to sustainability is a different one; we are a technology company.
Here are two posts on how we are funded and our business model; that post also covers privacy and (lack of) surveillance practices.
https://blog.mojeek.com/2020/10/who-funds-mojeek.html
https://blog.mojeek.com/2020/12/frequently-asked-questions-a...
Do you disclose what goes into your ranking algorithm? I think having that transparency and perhaps being able to tweak some ranking parameters would be go a long way to being able to verify it's built in the best interests of the users.
A balance in ranking for those seeking information and those looking to promote content will always be an issue; at any level of transparency. Done right, more transparency can benefit seekers too.
This is a case of our bot blocking being overly aggressive, where bots tend to use quotes.
We will rectify this in the coming days, thanks again.
Then, I'm wondering do you guys have the notion of "product brands" or something like that?
For example, when I search for "SaaSHub" - a product (I work on) that has been online since 2014 and have quite a few mentions around the web, the first result is some "random" Wordpress theme. i.e. if I'm searching for a particular product, I'd expect the product homepage to be the first result.
The feedback is appreciated and your example is one for a generic challenge that we are actively working on. I note that your site shows as link #2 so someone looking for your site/brand should see it; but still your point stands.
Given that you run your own spider, presumably you provided it one or more urls to get started spidering. I'm curious what those were.
Everyone wanted to understand how very few search engines are real crawlers, should open "The Search Engine Map" https://www.searchenginemap.com created by Mojeek.
Btw I'm not a native English speaker and mojeek seemed like a great name to me immediately. Definitely easier to remember than metager and YaCy, on par with qwant (I've been looking into alternate search engines recently). There were (and still are) people who thought DuckDuckGo is a terrible name - and even if that's true, they've shown it's not a significant obstacle at all. Good luck with your results, those are the only thing that really matter.
Sadly, the name is pretty horrible.
If you have to spell a domain out letter by letter when telling people in person, it’s not a good name. You also don’t appear to own common misspellings, like mogeek.com or moejeek.com or mojeak.com or at least they don’t redirect to the main website.
Still, I hope it gains market share and usage and I hope in a few years you may have the funding necessary to rebrand it into a more user friendly name.
From: https://blog.mojeek.com/2020/11/frequently-asked-questions-a...
How do you pronounce Mojeek?
It is pronounced ‘moh-jeek’ with a hard ‘j’. In many countries this is an unnatural sound, and so we encourage the use of ‘moh-yeek’ to those folks who find our name near unpronounceable.
https://blog.mojeek.com/2020/12/frequently-asked-questions-a...
"Search engine companies typically earn revenue from advertising (in many different forms). Other sources of revenue can include APIs and partnerships, including site and enterprise search. Recently search companies have started to explore a subscription model and/or micropayments. At Mojeek we are exploring options for our business model and all of these are potential routes for us; indeed we have had small revenues from some of them without being focussed on revenue streams. In 2021 we will start to focus on which revenue options to pursue proactively. Enabling people to search on Mojeek without tracking (aka surveillance) is a something we will not compromise on, so any routes we go down will be guided by our privacy-by-design principles."
Mojeek generates a signed token that IS authorization, you match it against a list of valid tokens, with no other information required.
This could be used to federate subscriptions, and hosting of front ends such that they could stand alone without having any knowledge of the users, while only serving valid users.
Flickr has something like this that grants access to an otherwise private portfolio, yet the user can revoke it if necessary, without ever sharing passwords, usernames, etc.
[Edit] - Example: On a linux machine, how could you give access to only one file in the whole system? Answer: By setting the permissions on every single file other than the one in question to deny access. Set the permission to allow access on the one file you care to share.
With Capabilities, the token IS the permission... and it doesn't really take much to implement it, once you completely grok the idea.
Of course, that's just one search, but based on this one search alone I've added Mojeek to my bookmarks and will be using it to find the more off-beat paths on the net now.
MoJeek, where are you based?
edit: UK. Thanks. I'm giving you all a try as my main engine.
https://blog.mojeek.com/2020/10/who-funds-mojeek.html
https://blog.mojeek.com/2020/12/frequently-asked-questions-a...
Extract from 2nd link:
What happens if a certain government authority comes knocking, demanding all logs for a certain time period and IP range?
User IP addresses are not logged, so it wouldn’t be possible for us to provide that information; we simply do not have it.
Have you ever received any requests for information from any authority?
At the time of writing we have not received any requests for information from any authority. If we were to be compelled by a court or similar to give over information, all we would ever able to hand over would be uncoupled search queries, referral data, and requested pages. We log, on aggregate, where traffic comes from at a country level, but we do not and never have logged or stored individual IP addresses. Any information about browsers used is uncoupled from search queries.
Does being in the UK subject you to surveillance under the Snooper’s Charter?
The Snooper's Charter (or Draft Communications Data Bill) was only a bill, it was never implemented. The similar ‘Investigatory Powers Act’ was passed, but that applies to Communication Service Providers, not us. As a non-tracking search engine which respects privacy, we don't store any IP addresses or other information that would be useful to the UK Government if they were looking to identify individuals and if we were compelled in court to hand it over. This all being said, we encourage all Mojeek users to use a wide variety of anti-surveillance and pro-privacy tools when browsing the Web.
Can you be a private option whilst being resident in a Five Eyes member country?
The Five Eyes is an intelligence alliance comprising Australia, Canada, New Zealand, the United Kingdom, and the United States. As Mojeek does not store IP addresses, there would be very little useful information to hand over to the Five Eyes nations were they to compel us to give them data or to allow them to access to Mojeek’s backend.
Stemming (grouping similar terms like "mortgage" and "home loan") seems bizarre. I did a search for my last name, and it came back with mostly results where the name was using the German spelling, which made the results mostly worthless. (the difference is the German spelling ends in "l", the French version ends in "le" and the English spelling ends in "el"). So, the resulting search results were mostly useless.
Hope to see more of Mojeek!
Interesting point about universal lists. Yes, there are a lot of dead sites out there but there are also a lot of good sites that buried way down in many search engines, by big brands and SEO.
However, I'm presently seeing very few results from popular sites that have a lot of user content. Github, Steam Workshop etc.
Searching "ge_tts" (an admittedly very niche project of my own) yields no useful results, where as the project page is first result for both Google and Bing.
EDIT: Increasing the weight for fuzzy text match search results would also be nice. "threejs" doesn't yield a Github result for at least the first 5 pages, but "three.js" works as expected.
Our index is 3.64 billion as of today; new servers in the next month will allow us to grow that much faster.
We are crawling/indexing those two sites but maybe not your content, yet https://www.mojeek.com/search?q=site%3Ahttps%3A%2F%2Fgithub....
You could add a maximum of once per day per domain if you are worried about abuse.
One thing I noticed was that searches often return pages in foreign languages, e.g. searching 'Wikipedia' pulls up de.wikipedia.org and es.wikipedia.org as search results in addition to www.wikipedia.org. I'm sure you're looking more into how to best handle page languages, but I figured I'd mention it for the sake of user feedback :)
https://www.mojeek.com/search?q=shirts+without+stripes
Good luck to Mojeek, but at some point we'll need to have a search engine that's not dumb as bricks five times a day.
__________________
[1] Mojeek's technology has been developed entirely from scratch by Marc Smith, mostly using the C programming language, and uses no pre-existing search or web crawler technology. All technology and IP is fully owned by Mojeek Limited.
1) I am building a browser [1] for the semantic/static web, would I be allowed to integrate this as a search alternative to searx.me and wiby.me?
2) What's your policy on submitting URLs? Do you support opengraph descriptions? Do you support dublincore (which would be awesome)? Should browser users be allowed to submit things they couldn't find?
3) I couldn't find detailled API docs, only the opensearch description. Do you have a JSON API that would help reduce the load of your web servers? Also, what syntax do you support?
2) We used to have an add URL page but a significant proportion of the submissions were spam. We will look into ways genuine submissions could be accepted as lots of people have asked for this.
3) We do, docs can be found at: https://www.mojeek.com/support/api/search/
If this isn't implemented, any idea on when or if you'll implement it at all? :) Love the idea behind the project, but I'm feeling a bit apprehensive seeing as a lot of these projects to build more privacy focused engines only lead to poorer results with a lot of the pages having questionable practices still.
I get the lack of personalized tracking, makes sense. I don't want that. But do they at least track, for a given query, which results are clicked or not (not attributed to a person, just a query string) for relevance evaluation and optimization? For most search engines, this is usually foundational.
I'd be genuinely curious if they use a different method for this sort of thing.
One thing I did notice because I mistyped the search string while adding it manually as a search engine in Firefox, is that if you hit a URL on http instead of HTTPS, you'll get a 403. Might be smoother just to redirect. I say that having seen someone affiliated with mojeek in the comments.
Do you plan to leave your index uncurated, will self-moderate or will allow reporting content like spam and link farms?
My opinion is that leaving it uncurated isn't really perfect, but moderation isn't ever perfect either. I wish I had some insight into this. I'm curious.
Our current thinking is that curation is needed but from a position of defaulting to uncurated.
"über allen wipfeln" --> 403 Forbidden
"über" --> works
I've tried a few other cases like those. It seems to me that Umlaute are fine when only a single word is used, even quoted, but a quoted string of several words where one has an Umlaut leads to the error.
The most "correct" result was 4th, but even this goes to a very specific page (2) in the documentation rather than what it should go to (3).
I don't know how to build an index, but language and date seem like low hanging fruits.
(1) https://www.mojeek.com/search?q=lynis+hardening
We have been tweaking our language detection and intend to roll out some changes in the coming months to give heavier preference to the user's preferred language
There's also an API feature that allows preferring newer documents over old (datewr) [1] It will eventually make its way as a search preference you can set for your own convenience. In the meantime it works by typing it into the URL.
We are also working on the core algorithm to better identify when the main content of a page has changed.
[0] https://www.mojeek.com/support/search-operators.html
[1] https://www.mojeek.com/support/api/search/request_parameters...
Searching for a 'domain' related term - out of 10 top results 9(!) are Godaddy.com, just on different country sub-domains showing duplicate content.
I expect to find the official SQL Server docs in the first results, but it was not in the first page of results :(.
https://www.mojeek.com/search?q=Aishwarya+Rai&fmt=images&img...
For the default setting we only show images from pixabay, which as you point out may show nothing:
We've put answers to some of these issues and notably these questions :
What data do you collect? What happens if a certain government authority comes knocking, demanding all logs for a certain time period and IP range? Have you ever received any requests for information from any authority?
here: https://blog.mojeek.com/2020/12/frequently-asked-questions-a...
We are exploring working with commercial partners who want data to deliver services (shopping, ads). We will only pass along search queries and location.
We will never provide IP; even though we know some partners pressure for it, or even insist on it.
"Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for www.mojeek.com. The certificate is only valid for the following names: dock.shp.mcafee.com, *.dock.shp.mcafee.com
Error code: SSL_ERROR_BAD_CERT_DOMAIN"
Could be caused by the extensive (68k lines) of blocking in my /etc/hosts file, I suppose.
You might want to talk to Mcafee about their classifying your site as dangerous, or just curse them out like I've done when they listed my software as malware. For what it's worth, they do the same to emacssurvey.org so it's not just you :)
I recommend this tagline for Mojeek :)
As it happens we were toying with taglines today and minded to go with these:
Want different results? Value your right to privacy? Trying to escape Big Tech?
> As it happens we were toying with taglines today and minded to go with these:
I very much like the "rediscover" part. The web has indeed lost it's charm when the results only show Facebook, Twitter and 10 or so other huge sites.
As for tagline, I'd suggest something along the lines of "Rediscover the web and escape your filter bubble!".
Wish you all the best and happy coding!
P.S. I'd be very curious to get insight into if you have considered open sourcing the search engine? This would tie into transparency and trust, and would also enable contributions! I'd love to contribute to my day-to-day search engine, especially if it is privacy respecting!
Also, I second a subscription model for sustainability. I do not want to be the product.
Kind regards, Robin
Open sourcing merits a whole post of it's own; it's a topic we will be writing on in the future and a bit about in our next FAQs post.
Subscription model under consideration: https://news.ycombinator.com/item?id=25385333
We are currently focusing on Romance/Latin European language documents as those languages in themselves have a huge amount of content to crawl and index. As our index expands we will increase coverage to index documents in other languages.