8,784 karma · joined August 13, 2009
[0] https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
However there is a bug somewhere between my instructions and the model - i think it is the jinja template, though i'm not sure as i haven't actually looked too deep into it - that in some cases make my last message be sent repeatedly to the model after each tool call, making the model think that i'm getting impatient or something :-P. When i notice it, i tell the model that this is some harness bug and it can ignore the repeated messages, which it does.
It seems to happen when i stop it while it is thinking after a tool call but i'm not certain. I try to stop it while some tool call is in process so it cancels the call as that seems to avoid the issue. Restarting llama-server also seems to make the issue go away so it might be a llama.cpp bug, but TBH i haven't found a clear repro case aside from it happening when i interrupt it while it is thinking. I think it is more common when the context is close to full, or at least i mainly remember it happening when i've been using a session for some time (i.e. not in fresh sessions, which does make reproducing it a bit harder).
In any case, the model itself seems to deal with all the above fine (and accepts the bug explanation for repeated messages), it is the rest of the stack that can show an issue with interrupting it (but even that is relatively rare).
llama-server --model Qwen3.8-27B-UD-Q4_K_XL.gguf --ctx-size 131072 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --host 0.0.0.0 --port 8080 --spec-type draft-dflash --spec-draft-model dflash-Qwen3.8-27B-Q4_0.gguf --spec-draft-n-max 3 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --mmproj mmproj-BF16.gguf --no-mmproj-offload --image-min-tokens 2048 --reasoning-preserve
It just barely fits in my RX 7900 XTX's 24GB of VRAM (TBH i'm not sure if dflash is much better than MTP - which is included as part of the GGUF - and MTP saves ~1GB of VRAM). I'm using Linux with Xorg/Window Maker as my DE so it uses no VRAM at all (Qt-based stuff - and i guess modern Gtk too - tend to use some VRAM for themselves, even opening Kate shows up in amdgpu_top :-P). The above gives me on average ~50 tokens/sec generation (drops to about 35 when the context is close to full) and ~700 tokens/sec prompt processing, which while not particularly speedy is fine for agentic stuff as i just give the AI a task and leave it in the background doing things. To save some VRAM i run the multimodal projector for vision on the CPU (the --no-mmproj-offload) and only one task is ever executed at the same time (--parallel 1). Also the kvcache is quantized to q8_0 - supposedly this makes the model a bit dumber, though the extra VRAM needed for the full f16 kvcache makes it more than ten times slower :-P (and AFAIK dense models like the 27B aren't as affected by it as MoE models are).
I'm using Pi[1] (a slightly older version that i've had remain offline and disabled the telemetry) for now (will replace it at some point since i'm not a fan of the ~500MB dependency tree, the almost-dark patterns for online access/telemetry and the increasingly worrying push for the devs' services, but i do like how it approaches compaction and that for the most part it doesn't need any handholding when some error happens - which is ideal for "let the AI do something and check it some time later").
It is running inside a container so it doesn't get any access to my real stuff (mainly to avoid the LLM making any mess - of which it sometimes does in the container, but well, it is contained :-P). In the container Pi gives the LLM some basic tools - read, write, edit and bash - and the last one is basically used to do most of the work (i've also made a bunch of tools the LLM can use via the CLI).
[0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...
[1] https://pi.dev/
Every role but the verifier is the same
underlying model, called in a fresh context
and coordinating through a shared workspace on disk:
<ws>/task.md - the problem statement
<ws>/plan.md - the manager’s overarching plan
<ws>/tasks.json - a debug mirror of the task list
<ws>/notes.md - curated ideas / findings / partial proofs
<ws>/solution.py - the current solution
My use of Qwen 3.8 27B (which is the only model i use almost since it came out) is really doing something very similar to the above where first a task is described, a `PLAN.md` file is made with the (current) plan and tasks are either in the plan file or in a separate issue tracker the LLM has access to via a CLI tool, a `STATUS.md` that tracks the current status and a `GOTCHAS.md` file for writing down various notes/tidbits/gotchas (TBH this is largely because Qwen 3.8 27B does want to note down stuff anyway so it mainly acts as an attractor for not writing its notes down in the other two places :-P).
There isn't any real harness driving this, i just give it a task, tell it how it'll be handled and off it goes. I guess i'm playing the "role" of the harness there as i tell it to not go and do everything in one go :-P but i never bother with fresh contexts, Pi's "rolling" approach to compaction (i.e. summarizing the older messages to make space while keeping the recent messages as-is) make it almost a non-issue.
I have considered making it a planning tool, though as some tool it can use via the CLI (it is running in a container), not a harness - but at the same time it feels like it can do the same thing with the `PLAN.md`/`STATUS.md`/`GOTCHAS.md` triplet. The only thing the tool would help with is to provide a UI for me to check what it's currently doing, but even then i usually just stop it and ask it directly :-P then tell it to continue.
Perhaps at some point i'll make some custom benchmark with stuff that are vaguely similar to what i work on and try a bunch of different approaches. It is something i want to do anyway to see how and if Qwen4 27B (whenever that comes it) is an improvement over 3.8.
My point wasn't that you can use agents for all feedback, but for the given case of testing your readme file's instructions (the article's title even mentions it is about the readme file) you can certainly use agents for that.
I read that paragraph not as "this will make your finances worse" but as "can you even afford to do this?"
It is much easier to go full kumbaya when your savings can last for the rest of your lifetime.
So what the author actually paid for was to interact with humans and the README checking was secondary - because, really, my own first thought was literally to ask an LLM check and try to follow the instructions in the README and pretty much any decent LLM (including several local ones) would be able to check if they're adequate and even suggest improvements (just don't let them write it for you :-P).
So i decided to share it with a couple friends and retro enthusiast communities on Discord. I took a video capture of my engine working, showing the 3d rendering, audio, video, etc with an Xbox->HDMI->USB capture chain. Then attached the file on Discord's input field and posted it (i have a tab with the web UI pretty much constantly open in Firefox). People weren't around at the time but some saw it today and commented while i was sleeping (with my computer off).
Last time i checked, this wouldn't be possible with IRC because:
1. There is no video (or any other file) host on IRC servers.
2. IRC discussions are ephemeral, you either need to be online all the time or setup some sort of proxy that keeps the discussions around while you or others are away (or do what i used to do 15+ years ago - run a terminal-based irc client in a screen session in a VPS :-P).
AFAIK #1 is still impossible and #2 is an optional extension (that i've never seen in real life). Meanwhile i can now connect to a new Discord community and not only not worry about losing any messages while i do other stuff (i often check what people have talk about in Discord since my previous "visit" to be up to date) but also search what people talked about previously - e.g. did anyone brought up some XYZ topic i want to talk about?
IME the above two (also add images and audio to the mix - in two of the communities i'm in there are people who post WIP music they make) are basically 90% of how i see people use Discord: they post something (very often media-based) and very often others see/respond to it in their own leisure much later (in some channels people even live on different timezones so they can't do otherwise).
TBH nowadays if i wanted to set up some community in a self-hosted manned and wanted to stick with open stuff, i'd probably use RocketChat or something like that (Godot uses a self-hosted RocketChat instance that while a bit more janky than Discord, it seems to provide pretty much the same features i mentioned above).
Then i noticed the horizontal scrollbar :-P
It is a direct image link, it wont ask your password.
Also, https is not available for free even if some solutions are available without asking money for it. In this case: the site is on a shared host so the only option is the paid one from the host (so it isn't free-as-in-money) while the alternative is having a VPS and installing something like let's encrypt (which is more expensive than the shared host and requires me to play sysadmin for it, so not only it isn't free-as-in-money but also free-as-in-time).
However, appearance-wise it looks very noisy/distorted with some crooked elements (especially window buttons) and missing visual parts (mainly bevel edges). I made a screenshot with comments for it[0].
I guess this was made with AI which isn't that great at making stuff pixel-level unless you explicitly told it to check reference images and be pixel level precise, so i'd recommend firing up a VM with WinXP in classic theme mode (or even better, 86Box running Windows 95 or Windows 98) and grabbing a bunch of screenshots.
Also i found some bugs:
1. You cannot click on non-visual controls (or disabled controls) to select them (e.g. timer). You can still use the rect selection to select them and drag them around by resizing via the handles but clicking on them selects the form.
2. Adding a control doesn't update the controls list in the editor unless you double click a control without a handler (i.e. the form, as long as you remove the Form_Load handler). Even after the controls are updated, if the event list has only one event, you cannot have the editor add by selecting it (i guess because it is already the selected item and you don't receive an event about selecting it).
I tried to make a simple pong game but it seems no key or mouse motion events are implemented yet so all i did was to make a bouncing OptionButton :-P.
[0] http://runtimeterror.com/pages/iv/images/5ffbf1494821c77a3fe...
Pi ended up hitting a compaction during a tool call and finished without issues, which is what i expected - in fact after giving the instruction i left to visit a relative since i expected it wouldn't need me to babysit it.
FWIW after it finished, i asked it the following:
---
Can you answer me the following questions about how Hax manages the context?
1. How does Hax handle running out of tokens? It does have some form of compaction, but if the compaction fails for some reason (e.g. the summary ends up needing more tokens) what does it do?
2. Can it handle cases where the context runs out of tokens during tool calls and if so, can it recover? How?
3. What happens with compaction if an LLM produces a few large responses or the LLM reads a few large files? Does the process ends up summarizing the entire (or all but one) conversation? Or is the entire conversation lost?
4. Are there any safeguards in place to avoid overwhelming the LLM? For example any file and tool output limits? If there is and any limits are reached, how does it handle them?
---
It went on and checked the code and, briefly the response (it was bigger but i don't want to repeat the entire thing):
1. It doesn't update the session on failure (last "good" session is kept), running out of context is treated like any other error without any automatic recovery and you're expected to fix it by hand (personally i'm not a fan of this).
2. Multiple tool calls are fine (there is a 85% threshold check to trigger compaction and a 50k tool result limit) but if a response and results jumps from below 85% to over 100% despite being under the 50k limit, it doesn't trigger any compaction and the next request is denied by the provider (i.e. llama-server). I have a feeling this is what i hit when i tried Hax with its own code.
3. There is no recovery from a user turn (prompt + response + tool result) that exceeds the window. FWIW this also seems to be the case with Pi.
4. It found a bunch of safeguards (tool output cap, caps in bytes and lines for the read tool, bash writes to a temp file and only a part of it is sent to the LLM, edit size cap, etc). AFAICT it is the same as Pi with one neat addition in that there is a default 2 minute timeout for bash calls (i've seen the LLM more than once run a command in Pi and end up stopping for 30+ minutes because the command wouldn't end).
For 3 i asked it a followup question: "About 3: AFAIK Pi (the harness you're on right now) does a "spit" summarization where the old messages are summarized up to a cutoff/split point and replaced with the summary while the newer messages after the cutoff/split point remain intact, which allow a mostly seamless transition between compactions. Does Hax do the same or something similar?"
The response was that, no, it doesn't, it summarizes the entire context. Which TBH is a bit of a dealbreaker for me since i often rely on this "seamless" continuity in my prompts and feels like the main reason why compactions feel like a non-issue with Pi.
Take the above with a grain of salt, i only checked the code for the compaction not using a split/cutoff point between older and recent messages, the rest are whatever Qwen 3.8 27B understood, but they do match my short empirical test. Also if my own understanding of the code is correct, it seems to be using the same system prompt for the summary as for regular/interactive use while AFAIK Pi uses a dedicated "you're an expert summarizer" (or something like that :-P) prompt. Not sure if it makes much or any difference, with LLMs being what they are, but TBH whatever Pi does works great IME.
On the other hand the idea of a self-contained native AI harness in C/C++ is enticing, especially one that doesn't have any network traffic outside of LLM-related stuff[0] and explicit user requests (Pi does try to autoupdate and has a separate opt-out telemetry beacon - both of which are disabled in different means, one via environment variable and another via a setting, which smells a bit like an dark pattern to me).
Anyway, this is the result of my findings about Hax. It is neat, but TBH the context handling is the main dealbreaker for me, especially since i'm often having the agent do something in the background (using a local LLM isn't exactly the speediest workflow) and do other stuff or leave the computer alone, so the last thing i want is to babysit the agent for errors. Pi's split summarization and context overflow handling seem to work much better.
For now i'll probably stick with Pi (i have autoupdates and telemetry disabled and i hope there isn't any other hidden snitch in place) and perhaps at some point i'll do the NIH thing and make yet another agent myself :-P
[0] well, it does attempt to autoconnect to a potentially running llama-server in localhost without being explicitly told to do so (Pi wants explicit configuration) but meh
I'm just running it now in its own source with Qwen 3.8 27B and llama-server and asked it to analyze the code itself. I'm mainly curious to see how it handles context compaction during tasks (what Pi does is almost seamless and AFAICT it isn't anything particularly fancy so i'd expect Hax to do something similar) and i guess asking it to analyze a whole C codebase would help trigger that with a 131,072 context. Unfortunately it seems to be missing some "context usage" indicator while it does stuff (it shows context usage in the prompt but not while working), but i guess if it does manage to analyze the C code properly, i can ask it to add that :-P and see how it fares (from my use of Pi i'm positive Qwen 3.8 27B can do all that stuff, so it'd mainly be up to the harness).
EDIT: also i wonder if it works nicely if it is possible to convert Pi transcripts to Hax - i have a few "in progress" and i'd like to continue where i left from, though while both seem to use JSONL for the transcripts i'm not sure if they're compatible
EDIT2: hrm, it tried to use more than available tokens during a compaction and stopped there expecting me to increase the limit (i can, but what if i couldn't?) and restart the llama-server. Pi sometimes does hit it but it manages to recover by itself without requiring any input by me (or to increase llama-server's limit).
You can already mix Free Pascal with C/C++ code if you use the same linker and libraries - i did manage to get FLTK statically linked with an initial backend for Lazarus in fact - this shot[0] shows a form and a few buttons from a self-contained binary on Linux (it links against FLTK statically and X11/etc dynamically).
But it is a PITA to get working for C++ libraries specifically (and you need a C intermediate for FPC to use). Also i couldn't get it to work for Win32, only Linux.
[0] http://runtimeterror.com/pages/iv/images/8a6ff400ed9b0d424be...
Also, since i mentioned Lazarus, if it can compile itself to Free Pascal, i wonder if it'd be useful for adding C++ support to Lazarus itself so that Free Pascal and C/C++ can be mixed in a project to make self-contained executables for desktop applications. It'd most likely need much more work than that just the compilation to make it a first class citizen like Free Pascal itself is for Lazarus/LCL (e.g. things like the object inspector and code tools being able to understand C++ well enough so that refactoring and stuff like doubleclicking on a button in a form automatically declaring and defining the handler and moving the editor cursor to the newly defined handler's code body), but maybe it could be used as a starting point.