jeffreyw128

This was my favorite course in college!

This is cool!

If you want to add embeddings over internet as a source, you should try out exa.ai. Includes: wikipedia, tens of thousands of news feeds, Github, 70M+ papers including all of arxiv, etc.

disclaimer: I am one of the founders (:

Hey Paulista,

Cofounder Jeff here. Both!

Exa | San Francisco | In person | Full time | $130K-350K

Jeff, cofounder of Exa.ai here. LLMs represent a brand new opportunity to organize humanity's knowledge, in a way that hasn't been done before. We're an AI research lab focused on AI-powered search algorithms (using embeddings), currently applied to vast swaths of the web (we make our money as a search API).

A little about us: - Raised series A a few months ago. https://techcrunch.com/2024/07/16/exa-raises-17m-lightspeed-... - 15 people, fully in person in SF. Our team - https://exa.ai/team - Our mission: https://exa.ai/blog/superknowledge

We're hiring pretty broadly across engineering - AI research, high performance Rust (e.g., we build an in-house vector DB), and full stack. If the mission of organizing the Internet motivates you, it's a good fit :)

https://exa.ai/careers

Exa | San Francisco | In person | Full time | $130K-350K

Jeff, cofounder of Exa.ai here. LLMs represent a brand new opportunity to organize humanity's knowledge. We're an AI research lab focused on AI-powered search algorithms (using embeddings), currently applied to vast swaths of the web (we make our money as a search API).

A little about us:

- Raised series A a few months ago. https://techcrunch.com/2024/07/16/exa-raises-17m-lightspeed-...

- 15 people, fully in person in SF. Our team - https://exa.ai/team

- Our mission: https://exa.ai/blog/superknowledge

We're hiring pretty broadly across engineering - AI research, high performance Rust (e.g., we build an in-house vector DB), and full stack. If the mission of organizing the Internet motivates you, it's a good fit :)

https://exa.ai/careers

Exa | San Francisco | In person | Full time | $130K-180K

Jeff, cofounder of Exa here. LLMs represent a brand new opportunity to organize humanity's knowledge, in a way that hasn't been done before. We're an AI research lab focused on AI-powered search algorithms (using embeddings), currently applied to vast swaths of the web (we make our money as a search API).

We're hiring pretty broadly across engineering - AI research, high performance Rust (e.g., we build an in-house vector DB), and full stack. If the mission of organizing the Internet motivates you, it's a good fit :)

https://exa.ai/careers

Missed exa.ai! Embeddings-based search engine with its own index

Exa | San Francisco | In person | Full time | $130K-180K

Jeff, cofounder of Exa here. LLMs represent a brand new opportunity to organize humanity's knowledge, in a way that hasn't been done before. We're an AI research lab focused on AI-powered search algorithms (using embeddings), currently applied to vast swaths of the web (we make our money as a search API).

We're hiring pretty broadly across engineering - AI research, high performance Rust (e.g., we build an in-house vector DB), and full stack. If the mission of organizing the Internet motivates you, it's a good fit :)

https://exa.ai/careers

That's true. Although should be much harder

Do our best at getting the right content.

For paywalls/login - we play pretty straight, always obey robots.txt, etc.

Yup, definitely still gameable but if the model learns what high quality content is like and what high quality webpages there are (which it does), then the only way to game would be to be great :)

For your search - I would recommend turning autoprompt off and searching something like "Here is a great summary of the best computer mice to use:".

Our embeddings model is trained on how links are talked about on the Internet, if that helps with querying. So you have to query like how someone would refer to a link before sharing it

The issue with traditional search engines is that keyword-first algorithms are extremely gameable.

Try https://search.metaphor.systems - it's fully neural embeddings-based search. No keywords, only an embedding of what the actual content of a webpage is.

So in the mentioned example of searching for Youtube downloaders, with Metaphor you'll get only Youtube downloaders (https://search.metaphor.systems/search?q=This%20is%20the%20b...)

Full disclosure - I work there :p

This is the dumbest thing I've read in a long time

Check out http://metaphor.systems/ - neural embeddings based search that is really good at finding the longtail of high quality content.

Really good for finding personal pages, niche blog posts, etc.. Algorithm doesn't at all weigh website popularity explicitly.

(Disclaimer: I'm one of the cofounders)

It’s especially terrifying that misinformation compounds multiplicatively with AI because it happens in 2 layers - once at the retrieval layer (where AI-generated content is worsening the problem of bad SEO content) and again at the retrieval augmented generation (RAG) LLM layer.

(shameless plug) At Metaphor (https://platform.metaphor.systems/), we’re building a search engine that avoids SEO content by relying on human curation + neural embeddings for our index + retrieval algorithm. Our mission is to ensure that the information we receive is as high quality and truthful as possible as AI adoption marches onwards. You (or your LLM) can feel free to give it a try :)

Hey HN!

We wrote a blog post about our adventures in building a (neural) search engine in the post-LLM world. We hope it gives a perspective on how we're thinking about the future of search and the role that tools like Metaphor could have. In brief, we think that LLMs will do more searches than humans, using tools like Metaphor.

You can play with our search here (http://metaphor.systems/) or check out our API here (https://platform.metaphor.systems/). The API is free to use up to 1000 requests/month, and if you're a student or nonprofit, just reach out and we can probably do more.

Cheers!