Elemelek is designed to sample subsets of instructions gathered/generated from various sources (and so with various quality/diversity) for LLM fine-tuning tasks.
Under the hood elemelek does the following:
-creates sqlite database to keep instructions / features in
-computes embeddings of instructions
-indexes the embeddings in HNSW index via usearch clusters the embeddings
-compute features of each instruction in dataset (basic text statistics + rerank score)
Once created it provides simple interface to filter and sample filtered data