About how long does it take to create the index, maybe compared to a zgrep? Could it be parallelized with something like `parallel --pipe`?
I haven't tried yet, but this seems like a great tool.
I haven't tried yet, but this seems like a great tool.
I run the indexer in parallel on multiple log files: in general there's no way to parallelize the indexing of a single file as decompressing a gzipped file requires all the bytes up to that point. Unless you already have a gzip checkpoint around somewhere, you need to sequentially scan: this is part of what zindex does to allow quick random access.
Thanks for the feedback :)