int8blog

hi, given AGI is already reached (?) we decided to publish the model kept in secret for multiple years now - we don't care anymore, it's so over anyways

Elemelek is designed to sample subsets of instructions gathered/generated from various sources (and so with various quality/diversity) for LLM fine-tuning tasks.

Under the hood elemelek does the following:

-creates sqlite database to keep instructions / features in

-computes embeddings of instructions

-indexes the embeddings in HNSW index via usearch clusters the embeddings

-compute features of each instruction in dataset (basic text statistics + rerank score)

Once created it provides simple interface to filter and sample filtered data

I spent some time gathering the knowledge around LLMs and most recent developments making them accessible on consumer hardware - created a blog post containing all my notes

finally finished my post about attention mechanism in NLP. There is 100 posts about it - let it be 101