> TinyChatEngine provides an off-line open-source large language model (LLM) that has been reduced in size.
But then they download the models from huggingface. I don’t understand how these are smaller? Or do they modify them locally?
Turns out the original source is actually somewhat informative. Including telling you how much hardware do you need. This blog post looks like your typical note you leave for yourself to annotate a bit of your shell history.
Of course the main problem is that I don't know enough about the subject to reason on it on my own.
From a performance point of view (quantized) integer parameters are going to run better on CPUs than floating point parameters.
Their optimized models are not downloaded from HF, but from dropbox. I have no idea why.