1,194 karma · joined March 1, 2014
Primary hacker of Compiler Explorer - https://godbolt.org
Mucks about with Javascript emulators - https://miracle.xania.org, https://bbc.xania.org/ and in C++ too https://spectrum.xania.org
Some OSS on https://github.com/mattgodbolt/ and https://github.com/compiler-explorer
https://rust.godbolt.org/ https://go.godbolt.org/ https://d.godbolt.org/
Rust, Go, and D language explorers are https://<lang>.godbolt.org, e.g. https://rust.godbolt.org/ and so on
Source: http://en.m.wikipedia.org/wiki/Financial_Information_eXchang...
I've raised an issue to capture this use case; feel free to chime in there! :) https://github.com/mattgodbolt/zindex/issues/12
The code and file format support multiple indices in the same zindex file but I haven't got a decent way to pass command line arguments to define them yet!
Watch this space: also feel free to ping me on email for more comments!
The trick is to store the gzip internal state as a checkpoint every few hundred MB. Then to decode data at a particular offset, you find the further internal state that's before your offset; reinitialize the compression internal state with that, and then scan on until you get to the offset you need.
The zindex file that's generated contains both these checkpoints and the index that lets one scan there.
There's an open issue I'd like to resolve to add support for indexing a particular field: would support for arguments like those the "cut" UNIX utility allow for your use case?
I run the indexer in parallel on multiple log files: in general there's no way to parallelize the indexing of a single file as decompressing a gzipped file requires all the bytes up to that point. Unless you already have a gzip checkpoint around somewhere, you need to sequentially scan: this is part of what zindex does to allow quick random access.
Thanks for the feedback :)