kloud

Calling open-weights as open-source in marketing materials is the usual misrepresentation. But now with the restriction on commercial use (which is against opensource definition) it is not even open-weights, technically it would be more accurate to call it weights-available.

The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.

Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.

GitHub is really two separate products: a code forge for teams and a social coding platform to collaborate on open source.

When I use the social platform, it is mostly as a read-only website. I am occasionally commenting, opening issues, PRs etc. The slowness and annoyances do not matter that much.

But I live in the team tool, where the annoyances and paper cuts are painful. Besides poor craftsmanship, there is often tension between the simpler design for social which sacrifices usability for pro use cases.

The social platform, which has huge network effects and is really hard to replace, is basically a loss leader for marketing the team product. I wonder if it would be viable for another company to create a team tool that stands on its own by outclassing Github like Linear outclasses Jira? That should be easier to replace. And Microsoft could keep running the social platform for the time being, where the adoption of alternatives is much slower and requires a much longer time frame.

This is quite puzzling. Back in the day I saw the breakthrough with Opus 4.5, and since then was very lightweight on CLAUDE.md, trying to declare intent, rather than prescribe imperative instructions. If it was off the mark, I would see clear mismatch, which gave me opportunity to correct, usually to clear and strip context.

Starting with Fable 5, if it goes off the rails, it is more difficult to correct it, because it is overall wrong, but covers its tracks with plausible sounding arguments, so it is hard to pin point and correct.

Now, this article points out techniques that were useful to rely on models more, but those peaked at 4.6. Now according to benchmarks Opus 5 is on the frontier. But when when it has looser reigns, it ends up gaslighting me even more with abstract word soup than any model before.

Gotcha, yeah if a prompt does not produce the result you want, you need to add more details to it, depending on the scope it can be a lot. When it includes all sufficient details it becomes a spec.

You can reference other specs in a prompt. Video codecs are well specified with hundreds of pages of specs, they are probably in some form in the weights already, but you could also reference them and hash them as inputs directly to create a functional alternative to ffmpeg.

This resonates, I can imagine the development process as a cascade: we get business goals and user feedback, product is defined which can be pretty fuzzy, designers define UI, engineering works data models and tech constraints. Inputs at the top can be fuzzy, but each node in the graph applies some intelligence and clarifies it.

Related to the friction you mention, since the flow graph of the context is not preserved when it gets transformed into code, I find LLMs getting confused and making worse changes, because the explicit decisions with good reasons are getting mixed with implicit decisions from LLM surfaced during designing/planning. Sometimes when things change the code should change radically to match the new reality, but LLMs often "play it safe" by accumulating code and adding to tech debt. With a flow graph when change is made in the upper levels, it could propagate and allow decisions in lower levels to be revisited.

I agree, another issue I have with existing SDD frameworks I've seen like spec-kit or openspec, is they seem to miss the mark by coupling specification (what the software should be) and execution (how to get there). If the description of execution is needed, it means the spec is weak. As a rule of thumb if a framework defines execution concepts like tasks or tickets, that is a red flag for me.

As a counter argument for the compiler perspective: what Cursor did is basically transpiling a project to another language, which is a form of compilation.

If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.

Specs might become one solution for coping with the need to review increased volume of code. A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code. The industry made the transition to higher-level once, paradigm is changing so it might happen again.

The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, review and edit the spec which should be tighter and much more compressed, then deterministically compile to code. Of course we don't want to be spec-first only, that would be going back waterfall, but doing iterations back and forth.

Now, Claude is not a compiler because it is closed and non-deterministic (they do opaque processing on server, hiding reasoning tokens), but LLMs might be. We refer to a piece of code from npm/pip by name to get some code by downloading it. We then have lockfiles with hashes to ensure integrity. Currently we are vibing it, but in the future we could refer to a piece of code by prompt/spec and getting the code by inferring it. To ensure integrity, the lockfile would be hashes of open weights and inference code (and ironing out implementation details like non-determinism due to GPU scheduling, etc.).

Would be interesting to see coding performance on SWE benchmarks.

Fusion of frontier models beating Fable, or cheaper models matching Fable performance at half the cost. Great announcement timing.

What is missing in the article is the reasoning/effort levels, so it is not ruled out the results differ just due to different reasoning budgets.

I would also be interested in seeing coding performance on SWE benchmarks.

Fair point, given it is more unconventional design, it is a riskier way in order to get something that works.

Cool project, I like the idea of having tmux-compatible CLI. I used Zellij to get better UX, but many agent tools integrate with tmux. This way agent tools can still integrate tmux as a defacto standard for programmatic interface while having better interface for users.

But I wonder if tmux/rmux design is suboptimal since it couples session persistence and window management together. Do you have an opinion the design which separates the responsibilities? An example that pioneered this approach is abduco, and libghostty-based zmx being a modern implementation.

Reviewing large volume of code is a problem. In the pre-LLM era, as a workaround to occasionally review large PRs, I used to checkout the PR, reset commits, and stage code as I would review. In the first pass I would stage the trivial changes, leaving the "meat" of the PR that would need deeper thinking for later passes.

With the increased volume of code with agentic coding, what was once occasional is now a daily occurrence. I would like to see new kinds of code review to deal with larger volume of code. The current Github review interface does not scale well. And I am not sure Microsoft has organizational capacity to come up creative UI/UX solutions to this problem.

Exactly this, existing code review tools became insufficient with the increase of volume of code, I would like to see more innovation here.

One idea that comes to mind to make review easier would be to re-create commits following Kent Beck's SB Changes concept - splitting structure changes (tidying/refactoring) and behavior changes (features). The structure changes could then be quickly skimmed (especially with good coverage) and it should save focus for review of the behavior changes.

The challenge is that it is not the same as just committing the hunks in different order. But maybe a skill with basic agent loop could work with capabilities of models nowadays.

Good to know, that makes it even better. I still find Opus 4.5 to be the best model currently. But if next generation of GPT/Gemini close the gap that will cross the inflection point for me and make 3rd party harnesses viable. Or if they jump ahead, that should put more pressure on the Flicker Company to fix the flicker or relax the subscriptions.

The OpenClaw/pi-agent situation seems similar to ollama/llama-cpp, where the former gets all the hype, while the latter is actually the more impressive part.

This is great work, I am looking forward how it evolves in the future. So far Claude Code seems best despite its bugs given the generous subscription, but when the market corrects and the prices will get closer to API prices, then probably the pay-per-token premium with optimized experience will be a better deal than to suffer Claude Code glitches and paper cuts.

The realization is that at the end agent framework kit that is customizable and can be recursively improved by agents is going to be better than a rigid proprietary client app.

This is awesome! I was thinking it would be neat to have something like abduco but on a more reliable foundation, like libghostty-vt.

For my agent management scripts I use zellij since it is more ergonomic than tmux. Abduco sounded good in principle, but implementation is too limited. However, zellij is quite huge in the order of tens of thousands LOC and I am using only small part of it. It looks like zmx might implement just the right amount of features for this use case, I am going to try it. It is always nice to achieve same functionality with leaner tools.

Do you also think about dvtm part alternative? I wonder if once libghostty proper gets finished it would open possibility to level up textual multiplexing and unlock some cool features with graphical UIs.

Great thought provoking article! Indeed, typing commands on the command line feels primitive like typing code into interactive interpreters (python, irb, etc.). Those are primitive REPLs.

With lisp REPLs one types in the IDE/editor having full highlighting, completions and code intelligence. Then code is sent to REPL process for evaluation. For example Clojure has great REPL tooling.

A variation of REPL is the REBL (Read-Eval-Browse Loop) concept, where instead of the output being simply printed as text, it is treated as values that can be visualized and browsed using graphical viewers.

Existing editors can already cover the runbooks use case pretty well. Those can be just markdown files with key bindings to send code blocks to shell process for evaluation. It works great with instructions in markdown READMEs.

The main missing feature editor-centric command like workflow I can imagine is the history search. It could be interesting to see if it would be enough to add shell history as a completion source. Or perhaps have shell LSP server to provide history and other completions that could work across editors?

Also in the context of LLMs I think model weights themselves could be considered an untrusted input, because who knows what was in the training dataset. Even an innocent looking prompt could potentially trigger a harmful outcome.

In that regard it reminds me of the CAP theorem, which also has three parts. However, in practice partitioning in distributed systems is given, so the choice is just between availability or consistency.

So in the case of lethal trifecta it is either private data or external communication, but the leg between these two will always have some risk.

Non-determinism is just a limitation of current implementations, but it is not a fundamental property: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

The strength of Lisps is in ability to define DSLs and then concisely express solutions for problems in that domain. Arguably no other programming language was able to exceed or even match that power until now.

The math behind transformers is deterministic, so LLMs could be treated as compilers (putting aside intentionally adding temperature and non-determinism due to current internal GPU scheduling). In the future I imagine we could be able to declare a dependency on a model, hash its weights in a lockfile and the prompt/spec itself will be the code, which corresponds to that insight.

That linear trend line does not seem to fit very well, I say we are looking at the beginning of a hockey stick :)

Stopped dual-booting for games and formatted the partition some time after Windows 7 EOL. Thank you Wine contributors, Valve and lord Gaben.

> It seems to me that LISP will probably be superseded for many purposes by a language that does to LISP what LISP does to machine language. Namely it will be a higher level language than LISP that, like LISP and machine language, can refer to its own programs. (However, a higher level language than LISP might have such a large declarative component that its texts may not correspond to programs. If what replaces the interpreter is smart enough, then the text written by a user will be more like a declarative description of the facts about a goal and the means available for attaining it than a program per se).

Pretty accurate foresight in 1980, in the "Mysteries and other Matters" section McCarthy predicting declarative textual description replacing lisp as a higher-level programming language, basically describing todays LLMs and agentic coding.

Although niche, things are pretty lively in the community. Among other things this year great progress was made on Jank, the native LLVM-based implementation with seamless low-level C++ interop. As part of that work a test suite is being created [0] and now includes runners for all of the major implementations to compare compatibility, next best thing besides a formal specification.

[0] https://github.com/jank-lang/clojure-test-suite

That sounds super cool, let me add another voice of encouragement, please do publish it.

Great work! I was just thinking the other day how an interface like this would be useful, it seems strange we don't see more UI attempts beyond basic linear chat.

I find most need for managing context for problem solving. I describe a problem, LLM gives me 5 possible solutions. From those I immediately see 2 of them won't be viable, so I can prune the search. Then it is best to explore the others separately without polluting the context with non-viable solutions.

I saw this problem solving approach described as Tree-based problem management [0]. Often when solving problems there can be some nested problem which can prove to be a blocker and cut off whole branch, so it is effective to explore these first. Another cool attempt was thorny.io [1] (I didn't get to try it, and it is now unfortunately defunct) in which you could mark nodes with metadata like pro/con. Higher nodes would aggregate these which could guide you and give you prioritization which branch to explore next.

Also graph rendering looks cooler, but outliners seem to be more space efficient. I use Logseq, where I apply this tree-based problem solving, but have to copy the context and response back-and-forth manually. Having an outliner view as an alternative for power users would be neat.

[0] https://wp.josh.com/2018/02/11/idea-dump-2018/#:~:text=Tree-... [1] https://web.archive.org/web/20240820171443/http://thorny.io/

Very cool! I created something similar for Casio F-91W. It also uses XState. The benefit of specifying a machine in XState, is that one can embed Stately editor to visualize the states and their transitions.

https://github.com/dundalek/casio-f91w-fsm

When a complex system cannot be meaningfully reduced, another approach might be trying to reduce scope.

Current areas include managing services on a server, managing a single-user laptop, and enterprise features for fleet of devices/users.

There is some overlap at the core where sharing code is useful, but it feels way more complexity than needed gets shipped to my laptop. I wonder how much could be shaved off when focusing only on a single scenario.