You sir are a legend to me :D i already can tell i will spend to much time in this in the future xD sc 4 ever
394 karma · joined October 7, 2019
I think its important that people don't just look at whats implemented rn, but rather just try out new things , not just "adjust" the existing ones. Thats how we got better progress, not by trying to build the 1000s variant of context compression but rather reconsider how can context be build to not have this issue (simple example).
Don't really understand why other agents to give those insights but well found my solution.
One side is that i started developing it with godot, and the game has well alot of menu stuff going on (without wanting to go to much into detail but you are to a certain degree able to configure units outside of a match into "blueprint" like system). This whole editor / management etc stuff got quite big and works really well.
Means, when i hit the gdscript limits, i did not want to port all of the existing stuff that already works perfectly fine with godot.
And ye since the c++ only runs math simulations but i still use the godot renderer (which so far seems to work out perfectly fine) i was okay with that mixed solution (for now).
The other reason is that in my well closer circle i got quite some people that code c++ on a daily basis and i don't know (at least i think so) anyone doin C# actively, so i thought if i would get stuck i have people i can easily ask about my c++ problems.
So it wasn't a "i think c++ is better than c#" decision rather a what do i know and what resources i have available easily.
Than started to move alot of heavy logic to c++ simulation : tedious indeed but the results speak for themself.
Basically allows me to use godot for things like menus dialogues and similar stuff while running the true heavy work in a c++ simulation.
While you might think "yey another coding agent" i went down some very different paths than other open source coding agents do right now. Ill not bore you with a ton of detailed breakdowns and rather list the main points that are relevant to understand why/how i build it.
Premise:
-The agent is meant to run against selfhosted environments first like Ollama etc with a focus on non cloud sized models, it should work properly on something like a 27b model already
-The agent with full intention trades execution time vs reasoning and result quality
So what does it do different:
1. The agent, instead of trying to let the model solve a whole task in 3-5 inferences, rather breaks down the "thinking process" in smaller chunks, basically decomposing the task into smaller tasks. While this is not a completely new concept, the agent will break it down to really simple single step variants which even for a smaller model can properly be solved. This involves not just file editings but all points along the way from interpreting user message to planing a task to defining acceptance criteria etc.
2. The agent is build in a way that it, apart from some fixed steps such as detect user intent at the beginning and synthesize response at the end, it will by itself decide what is the next proper step to execute/do based on the context i provide. While doing so the agent also revalidates its current execution and if it spots that, given the information i collected since it planned and started running, the current plan is not detailed enough or is missing out due to something he didnt "know" when he was planing, the agent will dynamicly mutate the coming plan stats, as in restructure/decompose/etc in order to have a properly layed out route to fulfill the task.
3. The agent has a rather complex system of how his context is composed which is combined by things like the chat history, previous step results, a backlog of what "single steps" it has done (updated file x , read range from file y etc) and in that it is strongly reason driven, so it will for most of the things it does also provide a small reason explaination for why it did that. Than the agent also has a working memory that contains facts and decisions it made along execution. There is more but thats just some thingsd to mention. The whole memory system and context is rather complex (tho not complicated). When i release it ill properly break down how all this works in the docs.
4. The agent provides full observability in the ui. At every point of execution you can basically in detail see whats in memory, whats the current composed context, whats the plan and what plan steps depend on what previous steps, you can look at artefacts it build and also you have basically a complete audit log of all single "actions" it has executed and can look into their details.
5. The agent allows to while in an execution send steering messages, meaning the message will be, using relevant current contextual information about its execution, send as inference with a prompt askind the agent to interpret on if its relevant, and how does it impact the current execution/plan. It than can decide either just do alter what it currently does, or even do a plan mutation in order to accomodate the users steering information.
6. The agent within its capabilities has a lot of error correction/self reflection logic. From dynamicly fixing json including a fallback to let LLM fix a response json, upon to if executions fail to much it will itself do a hypothesis on why this is happening, formulate a critique and with that reevaluate if it wants to try the steps again with the additional information, or if it wants to mutate the plan to accomodate the just spotted problems.
Theres quite some more i could list but i guess thats enaugh for now. The big trade off as mentioned earlier is execution time. In comparsion to cloud native agents, which will do giant editings and reasonings in just a hand full of inferences, my agent uses a ton of inferences. The big difference is that this allows (a) the agent to more precisly focus on specific tasks rather than overloading it with to much at once and (b) alot better spot problems/mistakes itself and adjust its execution plan to accomodate those without the user having to reprompt the agent 10 times until something is actually correctly solved.
Apart from the pure coding capability, the very same actions that allow for higher quality reasoning and coding at the same time allow the agent also be a good analysis and discussion partner.
So : what is it than used for if inferences and executions can take very long? Well for me its in terms of coding capabilities a hands off agent. Its meant to be informed once at the beginning with relevant details, and than should be fully capable (as long its in the capabilities for sure) to fulfill the task without the user having to permanently keep an eye open. I want to free time for myself and not change my time from coding to prompting.
Ill wrap it up here and say : the Agents name is "Loa" and i probably will post it on hackernews as soon i got a relatively stable beta to release. Im close to being fine with a beta release but i want to test some more runs before i publish.
Cool idea tho to make it as accessible (:
Basically a 2d rts, fully orthogonal view, plays in space with spaceship combat.
You start with a deployable base, kinda like the old C&C, which you can place around a minable planet. Than from this you can place further structures arround that planet including the typicals, mining (2 types) of resources, researching stuff, and building ships.
The 2 things i focus the most on in this game on are
1. Responsiveness. Im a strong fan of starcraft 2 (and brood war) and have spend tons of time playing sc2. I love this super responsive controles and its the only thing i will settle for in terms of controles
2. I love the game starsector (yes no rts) for the way its ships are designed. You have a base hull that has some stats like hp / speed / etc - than this thing has basically mountpoints where you can fit weapons into. I always loved this system. So what i build is a game where you can (outside of battle) define your desired ship blueprints combined of a hull, weapons and "inbuilds" (as in upgrades). You can than just build those ships ingame.
So ye made quite progress so far , not sure tho what will be the first mode i focus on once i get a playable (as in demo) state.
Btw using godot engine as base, but some logic is running as c++ gd extensions (rts is heavy calcs , gdscript is just not fast enaugh (or im just not a gamdev ¯\_(ツ)_/¯)
Here a screenshot of the ship blueprint editor for those who are curious - https://i.postimg.cc/bq5JB9J7/editorexample.jpg
Also asked the lawyer in consulted about it... i would be a heavy gamble
But sure location itself also plays a role no question.
The problem is that they are formulated in a way that it is super easy to have your software being possible "dual use" and that a judge has to decide if its fine or not. Making it worse it also states your "intention" which well is impossible to proof - if the judge says he doesn't believe your intentions are only good, you can literally get massively sued.
So ye i could move to another country and than publish it - apart from that i can let it rot on my hdd (which is prolly what will happen).
Edit: Additionally mentioned, it is not just the publishing in germany, even the facilitating already which is why i don't even have an article about it (any more).
What an irony. I cant publish a attack surface mapping / pentesting tool i wrote which runs fully deterministic and really controlable due to "dual use" legal problems - but llm driven tools hit public space......
sorry for the rant....
That said - i was obviously wrong.
I don't really know what else to say than : this breaks a fundamental foundation of privacy. Sure i was never 100% aligned with EU level decisions (neither with my residing country one's) but that's just... wrong on so many levels... how can i even still argue to people that EU and democracy are good if this is what its heading for?
Usually i try to explain stuff with common grounds, the bigger picture or whatever is fitting. You can't have everyone happy on every decision.
But at this point im lost of arguments. They just betrayed every citizen from my pov.
So whats my point of this comment? Maybe i need to vent in an environment where i think people might get how bad this actually is, since people IRL around me don't seem to remotly understand the impact....
This is one of the major issues that made the reddit variant a rather big nightmare for mods.
Sure you than can do it anonymous and so on but point is : its not like every actor that gets notified will react thankful to it. Some even just ignore it.
Tho i wanted to open source the tool (spend ~10 years developing it) and thats just not an option.
Don't wanne self advertise here , but for the sake of better understanding if you want to know the details you can read them here: https://blog.laughingman.dev/article/Ishikawa_10_years_of_bu...
This laws, while i wanne say have a good intention, just do the opposite...
I myself, residing in germany, developed a recon/vuln/scanning tool that im legally forbidden to publish cuz of the laws you just mentioned.
I have no public sources yet (will come at some point) but ill try to break it down into some simple points. After all: this is a research project.
Project: DeepThought
So instead of going for the path to take bigger and bigger models to solve more complex questions, i going another direction. My idea is to use LLM's in a way like an "inner monologue" to replicate a thought chain. Basically create thinking steps that can be dynamically chained.
Additionally, the project contains a 3 layer memory system which is parted into:
1. Frontbrain (this data composes the context for inference, its a set of "hot nodes" which have a temprature that per turn of conversation will cool down a bit, while if they are used in a "thinking process" get warmed up a big again. The idea is to have the context for the inference to only get the currently relevant information, while dropping of things that lost relevance. This should prevent context overflow
2. STM : Basically a session memory. This will keep all information from the current session even if they got to cold and dropped out of Frontbrain
3. LTS : LTS is always query'able for the thought process to retrieve information/structures, but only at the sessions end information is propagated from the STM to LTS. This makes identification of "unique" entities alot easier and has some other advantages.
So when you type something into the DeepThought engine, it will extract all information from your input and convert it into a kinda 2 type structure 1. A bitemporal hypergraph composed of Entities and Hyperatoms. While entities i think are kinda easy to grasp, hyperatoms can either represent "properties" (in form of facts) or relations to other entities. This allows to create a graph structure typed information network containing the relevant information
2. Frame summaries. Since only having a structured graph as just described looses a lot of processual/logical information which are relevant especially in more complex contexts, i also create basically short summary texts that are linked to entities.
This structures allow me to use dynamic graph traversal for searching for data, while also retrieving the related Frame summaries that are a more native variant for an LLM to understand logics and relations.
This is a very very superifical explaination because to go into detail would take quite prolly multiplage pages of info.
Important: Im running this on a local 5090 and it is NOT friendly in terms of amount of inferences (which is fine for me). I try to mimic a thought process not build a fast shipping product. Quality > quantity. If you would run DeepThought on any online inference provider your broke in 1 day.
So, rn i focus on the ingestion and retrieval logics to make storing and retrieving as good as possible with my hw options.
While the ingestions already involves multiple steps in which the "llm" basically works as judge to decide where to traverse in the graph, where to go into recursion and similar, this will become very relevant as soon ill start implementing "task execution" as capability.
If i solved those the next point is to reduce everything that i need in terms of thinking steps in what i would call "thinking primitives". The idea with those is, that i dont want a hardcoded thinking process, but it rather also want to have the thinking process in form of a graph structure. This would allow me to compose the process in form of data in the hyepergraph, which would in return allow me to enable the system to refactor/enhance its own thought processes.
So ye thats what im working on rn, very early concept/alpha phase.
When I referred to "such regions," I was personally referring to a combination of factors: infrastructure, supply chain consistency, reliability, and the general political situation. In this case, I would argue that poor infrastructure impacts transport and storage control when it comes to shipments. Supply chain consistency (even with organizations like UNICEF) is often not guaranteed; local partners change frequently, often influenced by the local situation, making it nearly impossible in some regions to maintain trusted chains. Reliability suffers because of these factors—when it is hard to maintain trusted partners, the problem persists. As for the political situation, I don’t believe I need to elaborate further.
So, when I say "such regions," I mean areas that fit this basic pattern. While not a perfect comparison, a notable example of this is when food supplies sent for civilians are intercepted by local armed groups. The supplies might reach the target location, but they do not always feed the people they were intended for. As you work in this area, you likely know this is not an isolated occurrence.
I am also not from the US, and I cannot speak specifically to the Sudanese American Medical Association. If they are truly creating change, that is a great thing, and everyone is free to donate to them. You will not see me advocating against donating to them.
Regarding your question on why I think you would send aid even if diversion was likely: I don't believe you would willingly fund "warlords." Rather, I believe that in high-risk regions, the intent of the donor doesn't always control the reality on the ground. My skepticism isn't a critique of your virtue or your specific organization, but a reaction to a historical pattern of aid diversion in volatile zones. You do this work because you believe the collected money will reach its destination and will not be abused, and I respect that you follow your beliefs for the "greater good."
You seem to be a good person doing important work, and to do that, you need to believe in the efficacy of your mission.
Lets go for the optimistic scenario in which UNICEF will only take a very small portion for the "processing" and really deliver lets say food and medical supplies to the region. Those warloard will simply come and take it away from those citizens and provide to their armies. Theres nothing those citizens can do against it.
Do i wish it would be different? Absolutely. But sadly the world doesn't work as i would wish it to.