Great job porting the C++ code! Seems like the reasoning was to provide the code as a library to embed in a HTTP Server, cannot wait to see that happen and try it out.
Looking at how the inference runs, this shouldn't be a big problem, right? https://github.com/setzer22/llama-rs/blob/main/llama-rs/src/...