ColinHayhurst

>Basically all 'independent' search engines piggyback off Google or Bing

Incorrect: https://www.mojeek.com/about/why-mojeek

The blocks for MojeekBot, as Cloudflare verified and respectful bot for 20 years, started before the robots.txt file changes. We first noticed in early June.

We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.

Kagi pays to use APIs from Mojeek and Google

Kagi uses at least Google and Mojeek

edit:

> Realistically, there are only 2 search engines now.

https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.

Information neutrality is preferred to unbiased (which we used in 2018) but a bit of a mouthful and not widely understood. For more recent and nuanced views on this key topic see: https://blog.mojeek.com/2022/05/freedom-to-seek-matters.html

With no ties, direct AND indirect, that does make Ladybird uncommon, like Pale Moon.

Our own approach to privacy is as radical as it gets in search: "No Tracking, Just Search". As we often say: tracking, not ads, is the fundamental problem. Contextual ads do not need necessarily to have tracking. Though the duopoly of search ad networks makes that a hard road too.

Good luck. Excited to see how Ladybird progresses.

This is a welcome initiative speaking from a personal and professional perspective, and as CEO of an independent search engine; we are all too well aware of the power of money and defaults.

This immediately comes to mind as akin to the Signal vis-a-vis WhatsApp etc. Here there is an obvious reason to use Signal and a well-understood proposition. What might it be for Ladybird? And how will you differentiate?

Congratulations to Vlad and the whole team. They are showing how a patient building of product and customer base can succeed, where a VC backed approach for a similar business model and product did not sustain. RIP Neeva.

We are delighted to support you as a customer and see your growth. Alternative search options are needed more than ever and becoming increasingly appreciated. We too are doing our bit in search, with a different business model and shared values about empowering a "human-centric and sustainable web that benefits individuals, communities, and society as a whole".

As proved by Mojeek; a full web scale international search engine built from the ground up with £3 million in funding, and no dependencies. Admittedly longer than a couple of years; 5 years as a hobby project, 10 years building, and more recently GTM [0].

I declare a bias as CEO, but the determination of our founder Marc Smith deserves more recognition, so here is the story up to 2021 [1]

[0] https://www.mojeek.com/about/technology.html

[1] https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...

Without opt out it would be a no go, so that's great to hear. What's the downside of opting out?

Another option is to use multiple sources. For example Kagi who use Mojeek (self-disclosure) and others; "Our search results also include anonymized API calls to all major search result providers worldwide"

Mojeek, fully independent index. self-disclosure: CEO

It's not just that. LLMs by compressing knowledge, and providing one answer [0], demote diversity. Search engines demote diversity but you can still get to many viewpoints; if you are willing to do the work to tweak your query, or go past top ranking results. On major search engines there is suppression but it's done page-by-page, or at worst site-by-site. With LLMs diverse viewpoints are suppressed through universal "safety" policies.

[0] https://blog.mojeek.com/2023/11/one-model-to-rule-them-all.h... as pointed out here, search engines are like Librarians that point you toward context, whereas LLMs are more like Doctors that protect you from context.

thanks. tried everything I can think of, still no luck, run out of ideas.

If Anthropic want some business they can get in touch. For Claude direct, at least until then, I'll carry on using you on Poe for chat. Will go elsewhere for APIs.

And without a mobile number. What is it with these US AI companies requiring mobile numbers?

Code verification failure on UK mobile number. So is it a EU launch not Europe?

4. Default search on Chrome, Android...

5. Default deals with Apple, Samsung...

> My impression was that it essentially does a web search and then uses an LLM to summarize the top few results.

Yes that's a good short summary. More details on how we do it here: https://blog.mojeek.com/2024/04/mojeek-search-summary.html

Not the whole web; LinkedIn and a few others block us and we fully respect robots.txt, but we have ~8 billion pages.

edit: from article, "Doing this for a few urls is easy but doing it for billions of urls starts to get tricky and expensive (although not completely out of reach)" - indeed so, but we have now done embeddings for about half of those ~8 billion pages and are using them for mojeek.com.

We have an API with many features including uniquely authority and ranking scorings. Embeddings could be added.

https://www.mojeek.com/services/search/web-search-api/ used by Kagi, Meta and others. Self-disclosure; Mojeek team member.

Kagi uses the Mojeek API as one of it's main sources.

> There is no such thing as an "unbiased" ranking, just like there's no such thing as "unbiased" journalism.

Correct. But you can use techniques, to reduce bias. And not use others, notably tracking, so eliminating what can be bias or perhaps better put influence.

Besides that the bias of publishers and platforms matters less the more diversity we have. We have many news publishers for example, but in search the dominance of Google means we have little diversity.

We can run contextual ads, and have API customers: https://www.mojeek.com/services/search/web-search-api/

https://www.mojeek.com/ self-disclosure, mojeek team member

Kagi uses the Mojeek API.

Or perhaps not. You may note that yesterday Head of Engineering and Product for Bing said [0]:

"Many people ask me whether LLMs will make web search engines less valuable, but the opposite is actually the case. LLMs make web search more critical than ever, since the combination of LLMs + web search is what produces fresher and more accurate chat results."

This is true, but what he didn't mention is where the the biggest source of pre-training data is from. This was revealed earlier this week by his colleague Mikhail Parakhin, CEO of Microsoft Ad and Web Services, in the Google antitrust trial. Apparently he revealed "Bing Chat and Google Bard leverage indexes." [1].

A point on which I speculated on back in February [2].

[0] https://twitter.com/JordiRib1/status/1707103407050477772

[1] https://twitter.com/jason_kint/status/1706866080130498939

[2] https://blog.mojeek.com/2023/02/will-chatbots-kill-the-searc...

"The money generated by the Google deal was a key reason why Apple declined acquiring Bing, according to the people" .... "with knowledge of the matter"

At current P/E and an estimated $18bn the deal is worth half a trillion $.

For one thing mainstream media is exempt from the bill.

It has been there, just not directly. Always ask yourself; what is the deeper unstated reasons why a story is being covered widely by mainstream media?

All parties have supported and encouraged this bill. There have been a few individuals in each party willing to question it, and improve it, but they have been more or less ignored by all sides.