For use cases where you have a lot of QPS, we recommend using Garage or MinIO or the feature to cache index data on the local disks (new in the 0.7).
I wrote a blog post that explains how we do that: https://quickwit.io/blog/quickwit-101
831 karma · joined April 25, 2021
https://quickwit.io https://github.com/quickwit-oss/quickwit
For use cases where you have a lot of QPS, we recommend using Garage or MinIO or the feature to cache index data on the local disks (new in the 0.7).
I wrote a blog post that explains how we do that: https://quickwit.io/blog/quickwit-101
This is my second year there, and here is our recipe at Quickwit:
- book an Airbnb for 1 week at Bruxelles, a spacious house is definitely better
- submit a few talks to share our feedback from the past year
- meet a LOT of people
- carefully choose a few talks to watch; the place is crowded, so you need to be in the room in advance
- organize an evening event with the fellows we appreciate a lot and who work in the same field
- enjoy teamwork during a few days in the big house
I hope to meet a lot of HN readers there; see you in 1 month or so!
While this was really satisfying on the performance side, we were a bit disappointed by the price, and I mostly agree with the article on this matter.
I can see some very specific use cases where the pricing should be OK but currently, I would say most of our users will just stay on the classic S3 and add some local SSD caching if they have a lot of requests.
> One downside is that you leak internals of other system (partitions).
True, but we generalized the concept of partitions for other datasources, pretty convenient to use it for distributing indexing tasks.
One thing that is also very interesting with Kafka is that you can achieve exactly-once semantic without too much efforts: by keeping track of the positions of partitions in your own database and carefully acknowledging them when you are sure data is safely stored in your db. That's what we did with our engine Quickwit, so far it's the most efficient way to index data in it.
One obvious drawback with Kafka is that it's one more piece to maintain... and it's not a small one.
Easy setup and fast deployment, pretty happy so far.
I have one question though: I did not see some object storage feature, is it possible to combine Koyeb with some kind of object storage service? Maybe an external one? What's you recommendation on this?
We know the PISA project very well because... it is the fastest engine (though academic) in the tantivy benchmark: https://tantivy-search.github.io/bench/
But I did not know that they were using Partitioned Elias-Fano, maybe the author of the article fulmicoton knew that, I will check that.
I'm so skeptical on such a study.. Obtaining solid results on this kind of survey is so hard or even impossible...
I'm happy for this guy but I have no idea if what he says will work for someone else. The only good thing I see in this article is that... it's possible to blow things up for the best, and that's great.
One point I want to emphasize is to think long-term. If you want to build something great that is solid, it will be hard and it will take years. You are also building trust with your user base and this takes a lot of time too. So you need to ensure that you won't be burned out by your project, both for you and for your users.
On the negative side of things, I see several OSS projects that try to oversell (consciously or not//too much marketing) their project to speed up the growth, this makes all the more important the assessment of an OSS project before putting it in production.
Side note: it should be possible to tweak some config parameters to optimize the memory usage or cpu usage of quickwit. Ask us on the discord server next time :)
This is my favorite philosopher by far (I'm not saying that because I'm french!). Partly because his essays are not too "abstract" and this allows me to truly understand it, partly also because I found his essays still valid today.
One little and unimportant anecdote I liked: his parents forced themselves and his tutor to speak only in latin when he was a child. Later, Montaigne thanked his parents a lot when he saw how others were struggling with this language.
Will follow closely your development, don't hesitate to ping me if you need some inputs to work on an on-premise contract.
I really like the fact that templates are available on GitHub: https://github.com/CommonPaper
I cofounded a company that develop and sells an OpenSource software and it's a huge pain to write the contract. It's also costly... as I don't trust myself to write a contract and ask a lawyer to do it...
I see that templates are more for cloud based offering but that would be great to extend this to on-premise contrat too. I would be happy to contribute.
You can use Quickwit (disclaimer: cofounder here), it ingests/indexes logs and has a very basic UI. Logs can be stored locally or on an object storage. However, you need to handle auth stuff yourself.
If a hosted alternative is ok for you, I would try Loki/Grafana though I did not test their SaaS.
It's not mature enough and it will require some work to migrate it as we would need to clean some Quickwit dependencies. It was just easier to add the MIT licence directly in Quickwit directory to allow anyone to copy/paste the code.
Let me just tell you the limitations on Quickwit side: the framework is very bound to Quickwit code base right now and uses tokio runtimes to run the different kind of tasks.
We put the MIT licence so people can copy/paste pieces of code.