Elemelek is designed to sample subsets of instructions gathered/generated from various sources (and so with various quality/diversity) for LLM fine-tuning tasks.

Under the hood elemelek does the following:

-creates sqlite database to keep instructions / features in

-computes embeddings of instructions

-indexes the embeddings in HNSW index via usearch clusters the embeddings

-compute features of each instruction in dataset (basic text statistics + rerank score)

Once created it provides simple interface to filter and sample filtered data