HNHacker News
TopNewBestAskShowJobs

francoismassot

831 karma · joined April 25, 2021

Co-founder at Quickwit

https://quickwit.io https://github.com/quickwit-oss/quickwit

submissionscomments
francoismassot··on Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog
We store the indexes on the object storage. We worked on the index data structure to optimize the query path.

For use cases where you have a lot of QPS, we recommend using Garage or MinIO or the feature to cache index data on the local disks (new in the 0.7).

I wrote a blog post that explains how we do that: https://quickwit.io/blog/quickwit-101

francoismassot··on Show HN: Quickwit – OSS Alternative to Datadog, Elasticsearch
thank you! From the beginning, we built Quickwit to work on object storage, so we worked hard to minimize the number of read requests to execute a search query. I wrote a blog post that explains the performance / costs you can obtain with Quickwit: https://quickwit.io/blog/quickwit-101
francoismassot··on Ask HN: What software do you use for IoT devices and server
On the storage part, quickwit, an OSS search engine, can also be an option (disclaimer: I work there).
francoismassot··on Ask HN: Apache-2.0 or LGPL-3.0?
Can you give the reasons why you chose the LGPL-3.0? Did you consider the AGPLv3 too?
francoismassot··on FOSDEM 2024 Schedule
I can't wait to be there, learn, and meet great people

This is my second year there, and here is our recipe at Quickwit:

- book an Airbnb for 1 week at Bruxelles, a spacious house is definitely better

- submit a few talks to share our feedback from the past year

- meet a LOT of people

- carefully choose a few talks to watch; the place is crowded, so you need to be in the room in advance

- organize an evening event with the fellows we appreciate a lot and who work in the same field

- enjoy teamwork during a few days in the big house

I hope to meet a lot of HN readers there; see you in 1 month or so!

francoismassot··on Pitfalls of Helm – Insights from 3 years with the leading K8s package manager
While I really enjoy helm when playing with k8s or kickstarting projects, I never feel "safe" when using it in the long run for updates/upgrades. "values.yaml" files and templating YAML files are too error-prone...
francoismassot··on S3 Express Is All You Need
We tested S3 Express for our search engine quickwit [0] a couple of weeks ago.

While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter.

I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they have a lot of requests.

[0] https://github.com/quickwit-oss/quickwit/

francoismassot··on OpenTelemetry at Scale: Using Kafka to handle bursty traffic
Yeah, I was only talking about exactly once semantic between Kafka and Quickwit.
francoismassot··on OpenTelemetry at Scale: Using Kafka to handle bursty traffic
Exactly-once semantic between Kafka and the observability engine.
francoismassot··on OpenTelemetry at Scale: Using Kafka to handle bursty traffic
Unfortunately no :/
francoismassot··on OpenTelemetry at Scale: Using Kafka to handle bursty traffic
Good point: first you're right, we do the ack on Kafka but it's not necessary. Second, this is not what I wanted to stress... and I should have not used the verb "acknowledge". What we do is upload the data on S3, then we commit partitions + positions in what we call the metastore. I can't edit my comment unfortunately.

> One downside is that you leak internals of other system (partitions).

True, but we generalized the concept of partitions for other datasources, pretty convenient to use it for distributing indexing tasks.

francoismassot··on OpenTelemetry at Scale: Using Kafka to handle bursty traffic
I really like this idea. And there is an OTEL exporter to AWS S3, still in alpha but I'm gonna test it soon: https://github.com/open-telemetry/opentelemetry-collector-co...
francoismassot··on OpenTelemetry at Scale: Using Kafka to handle bursty traffic
I heard several times that Kafka was put in front of elasticsearch clusters for handling traffic burst. You can also use Redpanda, Pulsar, NATS and other distributed queues.

One thing that is also very interesting with Kafka is that you can achieve exactly-once semantic without too much efforts: by keeping track of the positions of partitions in your own database and carefully acknowledging them when you are sure data is safely stored in your db. That's what we did with our engine Quickwit, so far it's the most efficient way to index data in it.

One obvious drawback with Kafka is that it's one more piece to maintain... and it's not a small one.

francoismassot··on Pg_bm25: Elastic-Quality Full Text Search Inside Postgres
Nice! I would be very interested by your benchmark, don't hesitate to jump in the quickwit discord server to talk about the results. https://discord.quickwit.io/
francoismassot··on Show HN: Run globally distributed full-stack apps on high-performance MicroVMs
I would need a very good bandwidth between the instances and the object storage. On AWS, we manage to get 1 GB/s for one instance. Is there one particular service that will perform particularly well? Also, I don't really want to pay for data transfer...
francoismassot··on Show HN: Run globally distributed full-stack apps on high-performance MicroVMs
Just tested it with my Quickwit engine :)

Easy setup and fast deployment, pretty happy so far.

I have one question though: I did not see some object storage feature, is it possible to combine Koyeb with some kind of object storage service? Maybe an external one? What's you recommendation on this?

francoismassot··on A compressed indexable bitset
Yes, just to confirm there was no off-the-shelf implementation in Rust.

We know the PISA project very well because... it is the fastest engine (though academic) in the tantivy benchmark: https://tantivy-search.github.io/bench/

But I did not know that they were using Partitioned Elias-Fano, maybe the author of the article fulmicoton knew that, I will check that.

francoismassot··on In praise of blowing up your life
> Disruptions can work even if you didn’t initiate them yourself. How about a coin toss? Participants in a survey conducted by Steven Levitt who made major life changes they’d been mulling over—proposing marriage, moving—based on the result of a coin toss were much happier, six months out, than those who didn’t.

I'm so skeptical on such a study.. Obtaining solid results on this kind of survey is so hard or even impossible...

I'm happy for this guy but I have no idea if what he says will work for someone else. The only good thing I see in this article is that... it's possible to blow things up for the best, and that's great.

francoismassot··on The Asymmetry of Open Source
Really nice guide to understand how to drive your OSS project, I run an OSS project and many things written in the article feels right. The positive point is that I see many users/customers that fully understands that if they want the project to last, they will need to contribute financially, it clearly makes your life easier.

One point I want to emphasize is to think long-term. If you want to build something great that is solid, it will be hard and it will take years. You are also building trust with your user base and this takes a lot of time too. So you need to ensure that you won't be burned out by your project, both for you and for your users.

On the negative side of things, I see several OSS projects that try to oversell (consciously or not//too much marketing) their project to speed up the growth, this makes all the more important the assessment of an OSS project before putting it in production.

francoismassot··on Show HN: Homelab Monitoring Setup with Grafana
Nice experiment.

Side note: it should be possible to tweak some config parameters to optimize the memory usage or cpu usage of quickwit. Ask us on the discord server next time :)

francoismassot··on Michel de Montaigne
excellent!
francoismassot··on Michel de Montaigne
I'm very happy to see Montaigne on HN, there must be some affinities with the community here and I like that :)

This is my favorite philosopher by far (I'm not saying that because I'm french!). Partly because his essays are not too "abstract" and this allows me to truly understand it, partly also because I found his essays still valid today.

One little and unimportant anecdote I liked: his parents forced themselves and his tutor to speak only in latin when he was a child. Later, Montaigne thanked his parents a lot when he saw how others were struggling with this language.

francoismassot··on Launch HN: Common Paper (YC W23) – SAFEs for Commercial Contracts
Yes, it's purely on-prem for now.

Will follow closely your development, don't hesitate to ping me if you need some inputs to work on an on-premise contract.

francoismassot··on Launch HN: Common Paper (YC W23) – SAFEs for Commercial Contracts
This is awesome.

I really like the fact that templates are available on GitHub: https://github.com/CommonPaper

I cofounded a company that develop and sells an OpenSource software and it's a huge pain to write the contract. It's also costly... as I don't trust myself to write a contract and ask a lawyer to do it...

I see that templates are more for cloud based offering but that would be great to extend this to on-premise contrat too. I would be happy to contribute.

francoismassot··on Whichlang – Fast, OSS for Language Detection in Rust
This blog post introduces the algorithm behind: https://quickwit.io/blog/whichlang-language-detection-librar...
francoismassot··on Learn Makefiles with the Tastiest Examples
I've got the same usage of Make and wanted to give Just a try. Which limitations did you identify with Just?
francoismassot··on Ask HN: Simple Logging / Stats Solution?
Where do you want to store the logs? What are your contraints on the durabibility?

You can use Quickwit (disclaimer: cofounder here), it ingests/indexes logs and has a very basic UI. Logs can be stored locally or on an object storage. However, you need to handle auth stuff yourself.

If a hosted alternative is ok for you, I would try Loki/Grafana though I did not test their SaaS.

francoismassot··on Efficient indexing with Quickwit Rust actor framework
You guessed it :)

It's not mature enough and it will require some work to migrate it as we would need to clean some Quickwit dependencies. It was just easier to add the MIT licence directly in Quickwit directory to allow anyone to copy/paste the code.

francoismassot··on Efficient indexing with Quickwit Rust actor framework
I don't know rustler limitations.

Let me just tell you the limitations on Quickwit side: the framework is very bound to Quickwit code base right now and uses tokio runtimes to run the different kind of tasks.

We put the MIT licence so people can copy/paste pieces of code.

francoismassot··on Vercel Service Markup
Not a Vercel user but when I see such markups, I would like to see alternatives and compare their markups.
← PreviousPage 2 of 4Next →