3,052 karma · joined April 10, 2013
There's a limit to how much RAM can be assigned to video, and you'd be constrained on what you can use while doing inference.
Maybe there will be lower quants which use less memory, but you'd be much better served with 96+GB
Not sure about ollama, but llama-server does have a transparent kv cache.
You can run it with
llama-server -hf ggml-org/gpt-oss-20b-GGUF -c 0 -fa --jinja --reasoning-format none
I moved away from ollama in favor of llama-server a couple of months ago and never missed anything, since I'm still using the same UI.
If you only need to stream media and serve files, sure.
As long as Google doesn't remove the ability to sideload apps, Android users are fine.
Something like the internet archive, but fully decentralized.
What amuses me even more is people thinking their code is too unique and precious, and that GitHub/Microsoft wants to steal it.
Been trying the 109b version on Groq and it seems less capable than Gemma 3 27b
I tried your version and when I ask it to create a tetris game in python, the resulting file has syntax errors. I see strange things like a space in the middle of a variable name/reference or weird spacing in the code output.
Other than that, the experience was completely different:
- The game worked on first try
- I iterated with the model making enhancements. The first version worked but didn't show scores, levels or next piece, so I asked it to implement those features. It then produced a new version which almost worked: The only problem was that levels were increasing whenever a piece fell, and I didn't notice any increase in falling speed.
- So I reported the problems with level tracking and falling speed and it produced a new version which crashed immediately. I pasted the error and it was able to fix it in the next version
- I kept iterating with the model, fixing issues until it finally produced a perfectly working tetris game which I played and eventually lost due to high falling speed.
- As a final request, I asked it to port the latest working version of the game to JS/HTML with the implementation self contained in a file. It produced a broken implementation, but I was able to fix it after tweaking it a little bit.
Gemma 3 27b on Google AI studio is easily one of the best LLMs I've used for coding.
Unfortuantely I can't seem to reproduce the same results in ollama/open webui, even when running the full fp16 version.
But ollama and openwebui performance is very bad, even when running the FP16 version. I also tried to mirror some of AI studio settings (temp 1 and top p 0.95) but couldn't get it to produce anything useful.
I suspect there's some bug in the ollama releases (possibly wrong conversation delimiters?). If this is fixed, I will definitely start using Gemma 3 27b as my main model.
I suspect the Ollama version might have wrong default settings, such as conversation delimiters. The experience of Gemma 3 in AI studio is completely different.
My prompt to Gemma 27b (q4) on open webui + ollama: "Can you create the game tetris in python?"
It immediately starts writing code. After the code is finished, I noticed something very strange, it starts a paragraph like this:
" Key improvements and explanations:
Clearer Code Structure: The code is now organized into a Tetris class, making it much more maintainable and readable. This is essential for any non-trivial game.Followed by a bunch of fixes/improvements, as if this was not the first iteration of the script.
I also notice a very obvious error: In the `if __name__ == '__main__':` block, it tries to instantiate a `Tetris` class, when the name of the class it created was "TetrisGame".
Nevertheless, I try to run it and paste the `NameError: name 'Tetris' is not defined` error along with stack trace specifying the line. Gemma then gives me this response:
"The error message "NameError: name 'Tetris' is not defined" means that the Python interpreter cannot find a class or function named Tetris. This usually happens when:"
Then continues with a generic explanation with how to fix this error in arbitrary programs. It seems like it completely ignored the code it just wrote.
I don't know about the OP tool, but open webui has its own document database which you can integrate with LLMs, and when answering questions it always cites the source with a link for you to verify
Can you elaborate on that?
No purpose. I think I added in the initial implementation and ended up not being required, but I forgot to remove
I then asked for Claude to write the docstrings and examples which increased by 500
People change. As you get older, you might find you no longer care that much about subjects you previously had very strong opinions about
I think the problem is thinking that you always need to use the best LLM. Consider this:
- When you don't need correct output (such as when writing a blog post, there's no right/wrong answer), "best" can be subjective.
- When you need correct output (such as when coding), you always need to review the result, no matter how good the model is.
IMO you can get 70% of the value of high end proprietary models by just using something like Llama 8b, which is runnable on most commodity hardware. That should increase to something like 80% - 90% when using bigger open models such as the newly released "mistral small 3"