TinyLlama a 1.1B Llama model trained on 3T tokens reaches 1.0 release
huggingface.co
huggingface.co
Looks interesting but I'd like to see how well it does with my prompt.
For example, you can run LLM inference with txtai (https://github.com/neuml/txtai) in Python using the following 3 lines of code.
from txtai.pipeline import LLM
llm = LLM("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
llm("""<|im_start|>user What is the speed of light?<|im_end|> <|im_start|>assistant""")
I expect they'll be GGUF and AWQ versions of this model soon that will make inference possible on even smaller devices.