HNHacker News
TopNewBestAskShowJobs

francoismassot

831 karma · joined April 25, 2021

Co-founder at Quickwit

https://quickwit.io https://github.com/quickwit-oss/quickwit

submissionscomments
francoismassot··on Show HN: Hacker XP – Hacker News styled as a Windows XP Outlook email client
thank you for this hilarious moment, the internal server error was just too good
francoismassot··on Quickwit 0.2 brings full-text search to ClickHouse and Kafka
oh nice. You should come to the discord channel to discuss that, we are still thinking about how to solve this complex problematic.
francoismassot··on Zinc Search engine. A lightweight alternative to Elasticsearch written in Go
Another alternative in Rust is Quickwit[1]. Only search is currently distributed but indexing distribution will soon come up.

Disclaimer: I'm a cofounder.

[1] https://github.com/quickwit-inc/quickwit

francoismassot··on Ask HN: Who is hiring? (November 2021)
Quickwit | Senior Rust engineer | Remote (EU, ASIA) | https://quickwit.io

Quickwit is building the next-generation open-source search engine for petabyte-scale data. We are also behind tantivy, the Rust search engine library that has rallied a great community of users since its inception.

We are looking for an engineering generalist with extensive experience writing Rust code and leading technical architecture. Working with large data sets, dealing with complex distributed systems, and developing backend services should be second nature.

More info about the position: https://quickwit.io/jobs/distributed-software-engineer

francoismassot··on Show HN: Highly cost-efficient search engine in Rust
We started with the idea of a double license model: AGPL or commercial. A saas model is a possibility but we are too far from it today.
francoismassot··on Show HN: Highly cost-efficient search engine in Rust
thank you! We solved 3 challenges as described in the blog post. The first one is really gathering metadata of different index files to know in advance where to read exactly the index data that will be needed when receiving a query. We call this the "hotcache". The hotcache is specific to quickwit. Then we also modified the index structure so that we only need to make only 3 subsequent queries on the storage to fetch the relevant data for return the matching documents. The index structure is coded in tantivy. Finally, with parallelisation of queries, we get the right throughput from s3.

We definetetly need to write a blog post on that and explain what we did in some meetups and confs.

PS: it's tantivy not tantivity :)

francoismassot··on Show HN: Highly cost-efficient search engine in Rust
Hi HN!

After months of quiet work, we are releasing today our first version of Quickwit.

We wrote a blog post about how we improve cost efficiency of traditional search engines: https://quickwit.io/blog/quickwit-first-release

francoismassot··on Quickwit: A highly cost-efficient search engine in Rust
Thanks for your kind words! Hope you will enjoy using it :)
francoismassot··on Searching the web for under $1000/month
Thank you! We did not plan to regularly update the index. But as it takes only 24 hours to index 1B pages, the easiest way would be to reindex everything, upload it to S3 and update the metadata so the search engine will query the right segments.
francoismassot··on Searching the web for under $1000/month
Thanks! You are totally right. For the demo, we have even banned a few words like "the" because the inverted list contains almost all doc ids...
francoismassot··on Searching the web for under $1000/month
The main reason is that AWS S3 is widely used. We obviously want to make it work on HDFS, MinIO and other relevant storage systems.
francoismassot··on Searching the web for under $1000/month
thanks! I guess you had no luck and the server did not respond, we have a bunch of errors on the python server and it may come from here.

It's working now, you can try it and found that the result is "fast and easy": https://common-crawl.quickwit.io/?query=fast&partOfSpeech=AD...

← PreviousPage 4 of 4