525 karma · joined June 24, 2020
Look at https://github.com/SWE-agent/mini-swe-agent for proof
Most commercial software is nowadays integrated into the real world in ways that can't be replicated by code alone, software which isn't like this can be easily replaced yes, but that kind of software already had free alternatives.
This is inevitable, you can't rely on user licenses as a growth metric
We already know that larger models hallucinate less since they can store more information, are there any smaller models which hallucinate less
Llms need a carefully designed interface which exposes tools at the intent level, most APIs are too low level for llms to perform user actions in a single call.
Semantic Layer needs proper language and tooling support which Malloy provides.
Semantic Layer is about decomposing views into dimensions and aggregates, then letting downstream apps/users compose their own views on top without having to redefine/re-calculate business level metrics.
This makes data analyis more flexible than sql views which are hardcoded on particular groupings.
Your idea of simulating agent interactions is interesting, but I want to know how are you actually evaluating simulation runs?
There are many ways to mix and match client(js) and server (php) in a composable way without react. Components can be created on the server side as well (mixing html+js+php). Only thing lacking is DX and type checking.
Here's my opinion test your code with all its side effects and infrastructure.
To do that fast and deterministically, switch to faster lighter weight versions of your production dependencies when running tests.
For example use SQLite or PgLite for databases, use in memory file system, use Mailpit for emailing etc.
Simulate your infrastructure dependencies, inject faults into them, test the real deal, SQL, file system semantics, protocols.
Don't leave the hard part of your testing to Q/A and users.
But I can see how this approach allows for parallelism as well, I especially like the fact that you only get one failure in case one of the steps fail
See it's demo:
https://manzt.github.io/quak/?source=https://pub-2fc10ef6724...