karterk

Typesense (from the latest version) has automatic schema detection: https://typesense.org/docs/0.20.0/api/collections.html#with-...

It is quite powerful and you can even choose to "stringify" or co-erce types automatically to deal with dirty data.

Thanks. Typesense takes a different approach to indexing than Algolia.

a) While Algolia (from what is available publicly) has indices which are pre-sorted on a set of ranking factors, Typesense allows dynamic, on-the-fly sorting.

b) Also I believe Algolia memory maps their indices, but Typesense stores the raw JSON documents on-disk and constructs the index from scratch on start-up.

We don't have a proper design document (but I certainly think we should have one). I will try to offer a high level summary:

At the heart of Typesense is a `token => documents` inverted index backed by an Adapative Radix Tree (https://db.in.tum.de/~leis/papers/ART.pdf), which is a memory-efficient implementation of the Trie data structure. ART allows us to do fast fuzzy searches on a query.

All indices are stored in-memory, while the documents are stored on disk on RocksDB. All underlying data structures were carefully designed, benchmarked and optimized to exploit cache locality and utilize all cores efficiently.

Cool demo. Searching for phases like "there was a" and "and there is" take a really long time. I presume that since the words are common, the document IDs mapped to those individual tokens are too long as well, so intersections etc. take longer?

Happy to benchmark if the dataset is shared!

Really cool demo. I'm curious to know what kind of hardware this is this hosted on.

Full disclosure: I work on a similar fast, typo tolerant, fuzzy search engine search engine called Typesense (https://github.com/typesense/typesense).

I've been following HN for 10+ years, first as a lurker and then getting into the whole "build something people want" thing. Over the years, I've "launched" quite a few projects here. Some have failed, while others have succeeded far beyond my modest expectations. But in a pre Product Hunt era, launching on HN was the only way to get exposure to your product. Even today, for a number of highly technical projects, HN is the best place to get the word out.

While HN crowd has a reputation of being too cynical at times (the most famous example being the original "Show HN Dropbox"), over time, pre-empting how the HN crowd will potentially react and what kind of criticism a project might attract has actually helped me improve the product before launch!

> I mean one day you got traffic 100K on the website. Good. But just for one day.

My latest project, Typesense, which is an open source instant search engine (https://github.com/typesense/typesense) literally found traction only after posting here on HN. Yes, it was a ~50K single day traffic, but it had a permanent impact on the baseline traffic. So nothing is as useless as it looks :)

Apart from the value I've gotten out of all these Show HNs, there is an incredible amount of value in the comments on HN. In fact, I often just skip the main post and just skim through the comments. Also, unlike certain other forums, snarky/toxic comments are discouraged and moderated.

What kind of hardware are you using to host the Postgres instance?

I was referring to an integration between Postgres and an external search system such that the data sync is automated. That was what the parent comment was asking about (I think).

As I posted in another comment [1], the biggest problem is handling updates and deletes. Requires that you do some kind of logical deletion with an `last_updated_at` timestamp to fetch updates from a primary table.

[1] https://news.ycombinator.com/item?id=25417856

Keeping an external search system in-sync with the primary DB is certainly a pain point. The biggest problem is that the records in the DB will likely be normalized, while the records in a search store will not be.

You can "poll" for new records for indexing by using a query, but handling deletes/updates is the real deal breaker. You will then have to use both the binlog (atleast for mysql) and querying, at which point it becomes quite complicated to reason about.

What features are you missing?

Tries are insanely good for implementing fuzzy search as well (say based on a Damerau–Levenshtein distance).

Using this approach for a typo-tolerant instant search engine that I am working on: https://github.com/typesense/typesense

"Cost of living" is not that straightforward to compute. In many Asian cities, the greatest cost is rent/mortgage. The cities are typically expensive and overcrowded because suburbs have no good schools and other amenities. However, most of these open salary calculators take raw purchasing power (like cost of commodity) to arrive at a rate.

I was faced with the GPL vs AGPL dilemma when I started working on Typesense (https://github.com/typesense/typesense):

I wanted to protect future potential commercial interests without stifling the spirit of open source collaboration and development.

I asked around for advice and eventually just chose GPL over AGPL because a lot of legal teams seemed allergic to AGPL and some companies even had a blanket ban on it.

I also thought that the greatest risk for my project was not an existing or future competitor: it was death by obscurity. To that end, I did not want to slow down adoption by choosing a license that screamed "$$$" or made somebody's legal team uncomfortable. In any case, if a big company wants to copy you, they will and can with the resources they can throw at it.

On the other hand, there are popular projects that do use AGPL (MongoDB) so it might not be that big of a deal in the real world :)

I can think of plenty of companies that failed simply because they were too ahead of their times. One cannot "time" the market. However, the playbook to survive in an early vs late scenarios are different. When you are entering a market late, you don't have to work hard to educate the market. For example, Apple literally created the touch screen mobile phone segment. Android phone makers only had to sell into that hype. However, when you are entering a crowded market, you need to have a really strong go-to-market strategy because you will have to find a way to break the strangelhold that your competitors have over the coversation.

On the other hand, when you enter the market too early, you might have the "novelty" factor but it's really difficult to stay under the radar: clones show up fairly fast. So unless you are able to stay underground and still grow fast (improbable but not impossible), your early-to-market advantage is really not that big of a deal.

Finally, the size of the addressable market also plays a big role. A large market can easily support new companies that can execute better.

Happy to hear any feedback on what I can do to make it easier. Typesense does not require you to define a full index-time sorting order (except for a default sorting field) so it's pretty flexible on what fields you can sort on at query time.

I've implemented Google Custom Search Engine on a few sites before, and it's not clear if this is the same (repackaged?) or different. The biggest question (as with many Google products) is when this it going to be shut down, given that Google Search Appliance itself reached its end of life in 2019.

If you want a dead simple way to integrate search onto your site/app but find the learning curve of something like Elasticsearch steep, take a look at this project that I've been working on:

https://github.com/typesense/typesense

Typesense is an attempt to offer a great search experience out of the box -- with minimal configuration. It also supports clustering/HA, so you can certainly go a long way before considering something like ES (which is a great product but also pretty beefy).

I don't have a direct answer but want to offer a different perspective.

I've been working on an open source search engine for a few years (https://github.com/typesense/typesense) that's far more easier than Elasticsearch for a number of use cases and is a great alternative to Algolia.

I was faced with this same dilemma when I started out: how do I protect future commercial interests without stifling the spirit of open source? Eventually I just chose good old GPL and decided against BSL-like licenses for a number of reasons.

GPL is a fine balance. When you are small, your greatest risk is not an existing or future competitor: it is death by obscurity. To that end, you don't want to slow down adoption by choosing a license that screams "$$$" or makes somebody's legal team uncomfortable. In any case, if a big company wants to copy you, they will and can with the resources they can throw at it.

I wanted to offer another perspective for this trend: Many states in India, especially in the South, have affirmative action policies that make it really hard for these people to find good jobs or colleges in India. So they head out and prosper in the US.

There is relatively very little money in open source. Also the scale of running a large application is very different compared to running an open source project most of which ironically are centralized in some way if not via GitHub, then certainly by the presence of a BDFL who drives the project.

You've the order wrong. This is merely the world reciprocating.

a)Allowing US Internet companies to compete fairly instead of using the firewall as an excuse to promote state controlled tech monopolies. b) Having a fair and transparent judiciary to handle cases of intellectual property violations.

I agree with you on the overall perspective. But I do think exceptions can and do exist.

This not just a case of “they did A, so we will respond with B”. It's a case of “they did A, B, C, etc. so we are responding with X after 10+ years“. To be fair, successive US administrations across party lines gave CCP a long enough rope.

Certainly: CCP has been able to fool the world for long enough. US policy of hoping that China would open up in the long run has proven to be extremely naive.

It was tolerated in the hope that over time China would open up. But that did not happen. When facts change, time to change your strategy.

US is merely reciprocating the policies of CCP: let them allow US companies to compete fairly in China as well.

Because many US companies have been at the receiving end of unfair trade practices, copyright theft and blatant IP violations from the mainland. It's good that we are finally doing something about that.

If you used a Trie for the inverted index, you can do fuzzy searches on tokens. This can help handle typo errors. This is the approach I've taken with Typesense: https://github.com/typesense/typesense

This was suggested when TikTok was banned in India. However the reality is that people just moved onto other clones.