400 karma · joined January 14, 2024
> python -m forge.proxy --backend-url http://localhost:8080 --port 8081
This is a good example because I've currently stuck with llama.cpp's UI. I can read your code (or throw Gemma at it =p ) but thought I'd ask anyway.
In this example, what is it exactly that your proxy is fortifying? The HTTP SSE requests? (Those would be `/chat/completions`.)
Not even remotely true. They regularly shut down products and services with impunity. If Gmail cost more than the data they directly or indirectly mine and sell from their users, Gmail wouldn't exist either.
For coding I don't need image support so I stuff the entire GPU with text-only mode. I don't have a workflow where I send LLMs off to generate thousands of lines of code but what little coding I did I did with Qwen3.6 and it was spectacular, as you likely suggest.
There are definitely differences in the eagerness to tool-call that you'll need to manage. And for all local models I've ever used, I've had to micromanage the tools provided by servers to eliminate any possibility that they reach for something wonky or confusing.
For someone who's been running local models for a long while, these are very very exciting times.
Love the joystick for mobile users.
But even if you disagree with me the point is that I can count on only one hand the number of times I went "oh man, I need a FastCGI middle end".
Again, just echoing the paper here. I don't know that I'm doing it justice.
And Brussels is not the capitol of the EU because the EU is not a country.
We actually have a /128 address only, and had to tweak several settings including enabling IPv6 masquerading (NAT).
I haven't the slightest clue why they didn't give us a block.
Good engineering is building the strongest bridge within budget and time.
Dead on.
Then again there are like 10 different ways to refer to soup in the various dialects.
Are you willfully ignoring people at bars, night clubs, supermarkets, etc?
It's obvious 99% of the time whether or not the conversation is in the wrong place and wrong time.