fulmicoton

Quickwit is compatible with S3-compatible object storages yes. I don't remember the feedback for R2 specifically, but we have users running Quickwit on S3, GFS, Azure, MinIO, Garage, IBM, and all of the major chinese clouds.

> Secondly, do you see Quicwit being used for analytics, such as tracking daily visits or analyzing user retention?

Excellent question. Quickwit is very fast on ElasticSearch aggregations. We do not support cardinality-aggregation but it is scheduled for version 0.8.

Analyzing user retention and generally speaking running complex analytics, will not be possible any time soon. Maybe next year?

Zookeeper is not used by Quickwit at all. It is used by Kafka however.

Not yet. Only prefixes. Also you could probably cook something with an ngram tokenizer.

Is it for a field with a high cardinality? If you tell us more about your use case, maybe we can find a workaround.

Web crawl can be an ok use case actually.

The idea then would be to "reindex" the world. It might seem ludicrous, but to give you an idea, indexing CommonCrawl takes about a day with 8vCPUs.

Most users do not use Kafka/Zookeeper. The only external service for them is a S3 bucket. They then use the PushAPI. It is perfectly fine if you have only a couple of TB a day.

For the crazy large use cases, like the one described in the blog post, Kafka becomes necessary. At that scale, our users usually already have their data in Kafka or RedPanda and are actually happy to be able to get native integration: - their data does not need to be copied/replicated in a WAL "again" - we get exactly-once semantics

Also, in 0.8, we will be adding proper support for distributed ingest. The feature is actually already implemented and was originally scheduled for 0.7. but we preferred to test it more before actually shipping it.

Quickwit is FTS too though. I think the difference comes from the fact they stored stuff on EBS while Quickwit stores its index on S3.

You can use AGPL for commercial use. The limitation is that you will have to opensource any patch you make to it.

It is a indexing vs search trade-off.

Let's just consider IO, as it is the main effect here. With a columnar, you have to read all of the fields targeted by your query.

If this is log, let's assume

- 200B per line of logs

- 100TB of logs = 500 billions lines of logs

- 30 days of retention.

- a body text field taking 70% of your data.

- highly compressible (10x compression ratio). Note this will come with a higher cpu cost, but let's focus on an IO lower bound.

If you have 100TB of data, regardless of the query, you will have to read (and decompress, but let's not talk about cpu) 7TB worth of data.

Now with an inverted index? You will have to read the posting lists only. The posting lists are delta-encoded and bitpacked.

Assuming the probability of presence of a given term in a log line is p, the worst thing that can happen is having a token that is in 99% of the documents.

In that case, you will have to read 2.06 bits per documents. That's 128GB for the worst posting list.

If you are looking for a single keyword, in the worst possible case, you will have to read 54 times less data than with the columnar solution.

In practice, users search for several keywords, but also considerably less pathological than the example I just gave. Overall you will typically end up reading 20 to 100 times less data than with the grep solution.

I left CPU aside, but actual search engines are also much more CPU efficient.

But I said there was a trade-off... where is it? Well you had to pay a much higher cost at indexing.

Some search engine implementation makes it seem like indexing is more expensive than it should be. Quickwit/tantivy are especially efficient there. With a 4vCPUs VM, you can expect to index at 2TB/day. So in the example above, you will have to dedicate 8vCPUs for indexing. This is perfectly reasonable.

BUT if your retention is much much shorter (few days), indexing might not be worth it.

If your volume of data is small too, you probably do not need to care at all about efficiency.

I didn't know about Azure Premium blog storage! The pricing is very similar too.

I ran the benchmark at Quickwit. I confirm it works as intended. I was extremely excited about this feature, primarily interested in the decreased GET request cost, and secondly the lower latency.

Unfortunately the price model puts it in a place where it is the right technology only for some very rare places.

In a nutshell the key thing you need to know is: - The storage is 6.4x expensive than classic S3. - The GET requests are 2x cheaper (with additional cost for large requests). - Your data is replicated within a single region. - latency is single digit ms.

From a pure cost wise point of view, the realm where it makes sense to use it is there, but small, and often competes more with EBS than it competes with S3.

> The other is that it implies serialised processing - you can't process anything > in parallel as you have single index threshold that defines what has been and > what has yet not been processed.

Fortunately Kafka is partitioned. You cannot work in parallel along partitions.

Also, you can streamline your process. If you are running your data through operation (A, B, C). (C on batch N) can run at the same time as (B on batch N+1), and (A on batch N+2)

We do both at quickwit.

That's a good list.

It has information from 2021. ChatGPT presents Quickwit as follows:

As of my last knowledge update in September 2021, Quickwit is an open-source search engine infrastructure that is designed for building and deploying search solutions quickly and efficiently. It focuses on providing fast and scalable full-text search capabilities for applications and websites. Quickwit is built on top of the Rust programming language and leverages technologies like the tantivy search engine library.

Haha I regenerated the results something like last week, and the PISA engine is not in the list anymore :)

This is indeed correct. Yes, we have comparatively few actors. One thing that is not explained in the blog post however, is that we can run several indexing pipeline on the same machine. We have a bunch of tricks in our framework to run ~1000ish index pipeline on the same server. That's a target of around 10k actors in the most extreme case.

We also send very few messages: the smallest message we send is represent batches of log lines.

As a result, our framework does not really care about performances. On one thread, it can process ~2.5 millions messages on my laptop: this is well above our needs, but it is correct to say that our framework is very slow compared to others.

The trouble with spawn_blocking is that it runs on the tokio thread pool... By default the size of that thread pool is gigantic because it is meant to run blocking IO. For computation, you would want to have a number close to the number of threads of the CPU.

... And you can't have both.

It can be used to search other things than logs, but it has to be large datasets of Append only data. Emails, Chat, Web Crawl data, logs...

And for those who cannot wait, there is quickwit :-) https://quickwit.io

pure algorithm & perf stuff... - exponential unrolled linked list: I don't know what they are called, so I called them that way https://fulmicoton.com/posts/tantivy-stacker/. - radix heap. I actually had to use that data structure on a real use case. - radix tree. Actually super useful. - HashMap<String, V> where the string and the value are contiguous in memory. Weirdly I've never seen that one... The hashmap can stores the full hash in its hash table anyway, so false positive are super rare. It will have to check the string anyway... You can improve your locality by keeping the key and the value in the same place in RAM.

just practical stuff, in python, I often use a magic dict defined as follows.

``` from collections import defaultdict >>> def MagicDict(): ... return defaultdict(MagicDict)

```

then you can use your MagicDict as a weird json-like map.

``` c = MagicDict() c[3]["aaa"][2] = 5 ```

In the beginning of the 90s I'd say there was a minitel in maybe 30% of homes in France (maybe a computer in <5%).

That was the preferred to look up a phone number / address, and checking exam results.

This is correct yes. Rumor-mongering is more efficient both in term of detecting/propagating a node failure and in term of network overhead.

We stopped using SWIM because it is too hard to get right, and because scuttlebutt allows all nodes to expose a bunch of meta-information about themselves.

It took a lot of time and effort to Hashicorp to patch memberlist into what it is today.

Quickwit CEO here. For people interested, we are focusing on immutable data (logs of all kinds typically). https://github.com/quickwit-oss/quickwit

We make it much 10x cheaper and easier to scale and operate. We store the index on Amazon S3.

With inlining maybe?

I agree!

... But I do wish I had some knobs in stable to hint the compiler in rust.

Mark a condition as unpredictable, a branch as likely, a loop as likely to be long, etc.

Some of it exists as intrinsics but this is not accessible in stable rust.

Not really. Lingua Franca is not really related to French.

A normal search experience (displaying a 20 hits search page) requires num segments * (1 + num terms * 2) + 20 GET requests.

We have 180 segments for our commoncrawl index. So we can consider a generous upper bound of 1000 requests.

The GET request costs adds $0.0004 per commoncrawl search request. Storage costs us $5 per day, so the cost of GET request starts topping storage cost at >10k request per day.

Our search engine is meant for searching large datasets, with a low number of queries: Logs, SIEM, e-discovery, exotic big data datasets, etc. These use case have typically a low daily query rate.

For high request rate, (1 query per second) like e-commerce, entirely decoupling storage and compute is actually a bad idea. For low request rate (< 1000 per day), using S3 without caring about the GET request cost is perfectly fine. And in the middle, you might probably want to use another object model with a more favorable pricing model.

There are only 180 splits. For this demo we use a file.

For more serious stuff we use postgresql.

That's on my to-do list for next week. :)

Absolutely! I want to try that.. We are especially interested in testing the latency minio could offer.

It depends on the use case does not it.

Shared nothing is the best architecture for e-commerce search for instance.

But if you have one query every minutes or so for a 1TB dataset, it feels a bit silly to have a couple of servers dedicated to it doesn't it? Imagine this is the case for all big data search you can think of... Logs, emails, etc. This is a waste of CPU and RAM.