HNHacker News
TopNewBestAskShowJobs

francoismassot

831 karma · joined April 25, 2021

Co-founder at Quickwit

https://quickwit.io https://github.com/quickwit-oss/quickwit

submissionscomments
francoismassot··on I can't recommend serious use of an all-in-one local Grafana Loki setup
Good point.

Unlike index-free solutions like Loki or Parseable, Quickwit is built on top of a modern full-text search index (tantivy). At query time, Quickwit produces much faster results (all other things being equal: CPU, memory, etc.), especially when the volume of data to analyze is large or queries are complex (high cardinality values, aggregations). Quickwit also stores data in a columnar format, so it's also good at OLAP-style queries (no joins though).

This comes with a cost during ingestion; Quickwit is more resource hungry than Loki but can still ingest at 20MB/s to 40 MB/s on a commodity instance with 4CPU. Similarly, regarding storage footprint, Loki compresses logs better because it does not maintain those extra data structures. Still, a Quickwit index tends to be much smaller than an Elasticsearch index.

The next release of Quickwit (may) will be shortly followed by the publication of a benchmark against Elasticsearch/OpenSearch, and by another one later, against Loki. You'll be able to see for yourself.

francoismassot··on I can't recommend serious use of an all-in-one local Grafana Loki setup
Quickwit is an open source Loki alternative too.

Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :)

https://github.com/quickwit-oss/quickwit

(disclaimer: I'm one of the cofounders)

francoismassot··on Show HN: We built a ClickHouse-based logging service
I see, thanks for your answer.

> Do you implement your own storage with Quickwit or can it be backed by a cloud storage solution like S3?

I'm a cofounder of Quickwit and I did not use OpenReplay directly, I just know OpenReplay used ClickHouse + Quickwit for search. I don't know if this is a common setup and if it was just used for very specific use cases. You should ask them :)

francoismassot··on Show HN: We built a ClickHouse-based logging service
How does it compare to OpenReplay ?

> Before deciding on ClickHouse, we were planning to use OpenSearch

You should have tried Quickwit :)

Anyway, sounds like a great project, best of luck!

[1] https://github.com/openreplay/openreplay

[2] https://github.com/quickwit-oss/quickwit

francoismassot··on Supabase Logs: open-source logging server
thanks :)

I'm convinced that a combo supabase + quickwit can be quite powerful.

It's possible to make Quickwit support simple SQL queries like "SELECT xxx FROM yyy GROUP BY something" quite fast. If you are eager to explore a POC with quickwit, let's have a quick chat, here is my email francois [at] quickwit [dot] io.

francoismassot··on Supabase Logs: open-source logging server
Cool :)

We would need to support SQL though.

Just putting the link here: http://github.com/quickwit-oss/quickwit

Would love to have your feedback on ease of use/perf/whatever here/on twitter/discord (https://discord.quickwit.io/).

francoismassot··on Show HN: Quickwit 0.5 released, a new distributed tracing backend for Jaeger
Hi HN folks

This release marks a turning point in the project as we are adding a developer-facing feature from end to end: a new Jaeger backend to store and search your traces.

To play with it, here is the documentation: https://quickwit.io/docs/distributed-tracing/overview

Have fun!

francoismassot··on Elastic, Loki and SigNoz – A Perf Benchmark of Open-Source Logging Platforms
I'm working on such a benchmark. I'm scared of the difficulty to be not biased though.

(disclaimer: quickwit cofounder here)

francoismassot··on Elastic, Loki and SigNoz – A Perf Benchmark of Open-Source Logging Platforms
Here it is: http://github.com/quickwit-oss/quickwit

Quickwit targets big and append-only data use cases, log search and traces in particular.

francoismassot··on Elastic, Loki and SigNoz – A Perf Benchmark of Open-Source Logging Platforms
One thing that I would want to see in this kind of benchmarks is the performance of these engines when data is stored on an object storage.

The main reason for that is that the amount of logs, metrics, traces data can be huge...

I think Loki was made to work on object storage.

francoismassot··on Sonic: Fast, lightweight and schema-less search backend
You can have a look at Quickwit (https://quickwit.io), it's a search engine made for logs :). It's still pretty young and... there are way less features than in ES.

(disclaimer: I'm one of the cofounder)

francoismassot··on Sonic: Fast, lightweight and schema-less search backend
You can consider also lnx that is based on tantivy and is performing quite well (https://lnx.rs/).
francoismassot··on Sonic: Fast, lightweight and schema-less search backend
One of the reasons for not using Quickwit for user facing search is the latency: for example, you pay 70ms of latency when you make a request on AWS S3... and generally you expect latency below that figure. Decoupling compute and storage while keeping a very low latency may be then impossible unless ending up by caching all your data on disk :).

You can have a look at lnx (https://lnx.rs/) that is based on tantivy and is performing quite well. It's not yet distributed but the author Chillfish8 has some thoughts about how to do it.

francoismassot··on Postgres Full Text Search vs. the Rest
I'm curious also about your use case (disclaimer, I'm one of the cofounder of Quickwit). Do you want simple search with a basic UI? Or some dashboarding with grafana for example?

Thanks @qdequelen for pointing out Quickwit :)

francoismassot··on ZincSearch – lightweight alternative to Elasticsearch written in Go
Yes Splunk is well known of course, I restricted myself to OSS projects.
francoismassot··on ZincSearch – lightweight alternative to Elasticsearch written in Go
(disclaimer: I'm one of the cofounder of Quickwit.io, we are doing log search)

(edited to add vector search category and add manticore)

When looking at OSS search projects, it's nice to have a rough ideas of the different use cases, here is the list I have in mind:

- user facing search (think algolia): elasticsearch[1]/opensearch[2], meilisearch[3], typesense[4]

- enterprise search: elasticsearch/opensearch

- log search: elasticsearch/opensearch, loki[5], quickwit[6], zincsearch[7]

- vector search: Milvus[8], Qdrant[9], Vald[10] Weaviate[11]

Vespa.ai[12] is great but I don't really know in which category to put them, user facing and enterprise search seems to work well with them.

It's interesting to note that Elasticsearch and Opensearch are general purpose search engine, Solr as well. They are all powered by Lucene, the popular and performant search engine library.

Another general purpose search engine is Manticore search[13].

I would love to see some benchmarks by category :)

Note: I don't know well ZincSearch and put it in the log search as said on their front page.

[1]: https://github.com/elastic/elasticsearch

[2]: https://github.com/opensearch-project/OpenSearch

[3]: https://github.com/meilisearch/meilisearch/

[4]: https://github.com/typesense/typesense

[5]: https://github.com/grafana/loki

[6]: https://github.com/quickwit-oss/quickwit

[7]: https://github.com/zinclabs/zinc

[8]: https://github.com/milvus-io/milvus

[9]: https://github.com/qdrant/qdrant

[10]: https://github.com/vdaas/vald

[11]: https://github.com/semi-technologies/weaviate

[12]: https://github.com/vespa-engine/vespa

[13]: https://github.com/manticoresoftware/manticoresearch

francoismassot··on Writing your own search engine is hard (2004)
A search index is often made of smaller independent pieces often called segments. So you can download & process progressively the data locally and upload it to an object storage. And run queries on it. That's what we did here for this project: https://quickwit.io/blog/commoncrawl

Also an interesting blog post here: https://fulmicoton.com/posts/commoncrawl/

francoismassot··on Ask HN: Who is hiring? (June 2022)
Quickwit | Senior Software Engineer | Remote (Europe, Middle East, Asia) | Full-time

Quickwit is the next-gen open-source search engine for logs at petabyte-scale data - https://github.com/quickwit-oss/quickwit.

We are looking for an engineering generalist with extensive experience writing Rust code and leading technical architecture. Working with large data sets, dealing with complex distributed systems, and developing backend services should be second nature.

Our search engine is coded in Rust and the following features are on our roadmap:

- distributed indexing

- aggregations (min, max, count, sum, avg, ...) for search queries

- gossip protocol for cluster formation

We bring particular attention to high performance, high reliability, and code quality.

Learn more and apply at https://quickwit.io/jobs/distributed-software-engineer

francoismassot··on Husky, Datadog's Third-Generation Event Store
Nice article indeed, we ended up implementing the exact same architecture at Quickwit for... log search! :)

https://twitter.com/fulmicoton/status/1526776987553263616 https://github.com/quickwit-oss/quickwit

francoismassot··on Decentralized cluster membership in Rust
Indeed, thanks for spotting that. We are fixing this.
francoismassot··on Decentralized cluster membership in Rust
And it's open-source, MIT License.

Crate: https://crates.io/crates/chitchat

Github: https://github.com/quickwit-oss/chitchat

francoismassot··on Tell HN: OpenBB scraping GitHub emails for marketing spam
Received the same email.

Here is the GitHub link[0] to report abuse. I send this short text[2]. Generally I'm ok to be contacted by email and I think I'm quite tolerant regarding this as long as I don't receive a spam saying that I subscribed to a newsletter while I have not.

[0]: https://support.github.com/contact/report-abuse?category=rep...

[1]: As described on [HN](https://news.ycombinator.com/item?id=30900237), the owner of this [repository](https://github.com/OpenBB-finance/OpenBBTerminal) is spamming GitHub users that have starred their repository. I never subscribed to their newsletter.

francoismassot··on The User Manual for Founder Psychology
I founded 3 startups, Quickwit.io is the last one.

In my first 2 startups, resentment between founders and burnout made me do poor choices. I wish I could have read this manual before. I strongly encourage founders to do so.

Luckily for me, now I found 2 great cofounders at Quickwit so I'm not stressed at all. For now :)

francoismassot··on Silicon Valley billionaires believe in the free market, as long as they benefit
broken link
francoismassot··on Meilisearch raises a $5M Seed to change the world of user-facing search
Quickwit co-founder here :)

I pretty much agree with you qdequelen, Quickwit and Meilisearch are targeting 2 very different markets: user facing search for Meilisearch and what I would call "search analytics on large (almost) immutable datasets" for Quickwit. We want to handle deletes in 2022 in the form of "delete by query" but it will be quite a heavy operation and you won't be able to do that too often (few times per day typically). So yes, log market is one of our target, plugging Quickwit search to ClickHouse was also pretty fun.

And as concerning the "best possible developer experience", Meilisearch is very inspiring and we want to follow the same path :)

francoismassot··on Introducing the new star-history.com – the missing GitHub star history graph
Fan of start-history.com here, thanks for your work.

One feature is missing: search repository with a nice autocomplete would be very useful.

francoismassot··on Click and Swap, our alternative to Drag and Drop
Yes indeed, love that too, "Click and Drop" is nice.
francoismassot··on Click and Swap, our alternative to Drag and Drop
Not bad but I feel like a hover on the item would be nicer than having to click on the button. I guess the button is only for reordering right?
francoismassot··on Click and Swap, our alternative to Drag and Drop
I would love to see a demo to feel the UX. I'm not fond of drag and drop and really want to test the click and swap.
francoismassot··on Ask HN: Which web analytics software are you using in Europe?
Plausible seems good but never tried it: https://plausible.io/
← PreviousPage 3 of 4Next →