> Hm, I am interested, but I would love to use it as a rust lib and just have rust types instead of some json config...
Yes that's how you use tantivy normally, not sure which json config you mean.
tantivy-cli is more like a showcase, https://github.com/quickwit-oss/tantivy is the actual project.