356 karma · joined June 3, 2014
Java/Kotlin/JS/TypeScript/Node.js/Google Cloud developer in Perth, Australia
https://apporchestra.com
daniel.campagnoli@trafficguard.ai https://www.linkedin.com/in/danielcampagnoli/
meet.hn/city/au-Perth
Socials: - linkedin.com/in/danielcampagnoli
Interests: AI/ML, DevOps, Cybersecurity, Outdoor Activities, Open Source
---
The other day I was looking into adding Trusted Types in the Content-Security-Policy header, which was something new to me. In my chat with Claude I asked:
"Lets brainstorm 10 a list of ideas closely related to this so we can think of anything we might be missing on the topic to consider."
And that provided a good list of items to review to consider and expand out the sphere of thinking for the LLM.
It is an infuriatingly hard problem to have the LLM produce excellent results every single time, and have it just do everything and want it to read our mind and all the knowledge and context of a task. I think we'll make some good progress over the next few years as agentic workflows are built out to mimic out thought processes, and the cost/capability of the LLMs keeps improving.
I prefer DeepSeek's new context caching implemention being completely automatic and code free. https://platform.deepseek.com/api-docs/news/news0802/ And without a 25% premium
As for LLM costs that really depends what you're trying to do when it. Fortunately that cost is always coming down. When I was first building it with Claude Opus the costs did add up, but 100 days later we have 3.5 Sonnet at a fraction of the cost.
The Aider benchmarks are good to see how different LLMs perform for coding/patch generation. Sonnet 3.5 is best if it's in the budget. DeepSeek coder v2 gives the best bang for buck https://aider.chat/2024/07/25/new-models.html
The CLI scripts under src/cli would be the best examples currently to have a look at for running an autonomous agent, and the fixed workflows (e.g code.ts)
<files>${fileContents}</files>
<requirements>${requirements}</requirements>
You are a senior software engineer. Your task is to review the provided user requirements against the code provided and produce an implementation design specification to give to a developer to implement the changes in the files.
Do not provide any details of verification commands etc as the CI/CD build will run integration tests. Only detail the changes required in the files for the pull request.
Check if any of the requirements have already been correctly implemented in the code as to not duplicate work.
Look at the existing style of the code when producing the requirements.
Then there is a compile/lint/test loops which feeds back in the error messages, and in the case of compile errors the diff since the last compiling commit. Aider added some similar functionality recently.Then finally there's a review step which asks:
Do the changes in the diff satisfy the requirements, and explain why? Are there any redundant changes in the diff? Was any code removed in the changes which should not have been? Review the style of the code changes in the diff carefully against the original code. Do the changes follow all the style conventions of the original code?
This helps catch issues that Aider inadvertently introduced, or missed.I have some ideas around implementing workflows that mimic what we do. For example if you have a tricky bug, add a .only to the relevant describe/it tests (or create tests if they dont exist) add lots of logging and assertions to pinpoint the fix required, then undo the .only and extra logging. Thats whats going to enable higher overall success rates, which you can see the progress in the SWE-bench lite leaderboard as simple RAG implementations had up to ~4% success rate with Opus, while the agentic solutions are reaching 43% pass rate on the full suite.
Initially the coding and software dev agents were called workflows, but to make it more agenty I was ok with it being called an agent if the result of an llm call affected the control flow
If you think you might want to feel free to get in touch
I see a bit of reuse of words in other various llm related projects.
Langchain/langfuse/langflow
Llama/ollama/llamaindex
so I hadn't been too worried about it when became aware of them.
That's what Show HN is for, getting feedback, and a name changed now would be easy before I post it around more.
There was 100 days in between Claude 3.0 Opus and Claude 3.5 Sonnet being released which gave us similar capability at a 80% price reduction. When I was using Opus I was thinking this is nice, but the cost does add up. Having Sonnet 3.5 so soon after was a nice surprise.
One more round of 80% price cuts after that combined with building out the multi-step agentic workflows should provide some decent capabilities!
The way to get better results is with agentic workflows that breakdown the task into smaller steps that the models can iteratively come to a correct result. One important step I added to mine is a review step (in the reviewChanges.ts file) in my workflow at https://github.com/TrafficGuard/nous/blob/main/src/swe/codeE...
This gets the diff and asks questions like:
- Are there any redundant changes in the diff? - Was any code removed in the changes which should not have been? - Review the style of the code changes in the diff carefully against the original code.
Maybe try using that, or the package that I use which does the actual code edits called Aider https://aider.chat/
Gemini has two aces up its sleeve now with the long context and now the context caching for 75% reduced input token cost. I was looking at the "Improved Facuality and Reasoning in Language Models through Multi-agent debate" paper the other days, and thought Gemini would have a big cost advantage implementing this technique with the context caching. If only Google could get their model up to the level of Anthropic.
And in Google Vertex its free at the moment!
Agentic Coding Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude 3 Opus. Our evaluation tests a model’s ability to understand an open source codebase and implement a pull request, such as a bug fix or new feature, given a natural language description of the desired improvement. For each problem, the model is evaluated based on whether all the tests of the codebase pass for the completed code submission. The tests are not visible to the model, and include tests of the bug fix or new feature. To ensure the evaluation mimics real world software engineering, we based the problems on real pull requests submitted to open source codebases. The changes involve searching, viewing, and editing multiple files (typically three or four, as many as twenty). The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation. We run these tests in a secure sandboxed environment without access to the internet.
% of problems which pass all tests Claude 3.5 Sonnet 64% Claude 3 Opus 38% Claude 3 Sonnet 21% Claude 3 Haiku 17%
I remember having moments looking at the plans Opus generated and being impressed with its capabilities.
The slow speed of requests I could deal with, but the costs could quickly add up in workflows and the autonomous agent control loop. When GPT4o came out at half the price it made Opus quite pricey in comparison. I'd often thought if I could just have Opus capabilities at a fraction of the price, so its a nice surprise to have it here sooner that I imagined!
The agent platform isn't officially launched yet, but its up at https://github.com/trafficguard/nous
I never liked the Langchain API when I looked at the examples so I built it from scratch. It has an autonomous agent with a custom XML-based function calling, memory and call history. The software engineer agentic workflow I initially dog-fooded with a prompt like "Complete Jira XYZ-123". So it gets the Jira description, finds the appropriate terraform project in GitLab, clones, edits (delegating to Aider), creates a MR and messages on Slack. It also has a UI for running agents, human-in-the-loop interactions etc.
The difficulty of such a task is two-fold. First, the search space is large. For example, on average, each query entity connects with more than 50 facts, and each bridge entity in the ground truth proof connects with more than 900 facts. Second, there are no surface form clues to exploit and bias the search towards the ground truth proof, unlike most conventional QA benchmarks where the proof steps are transparent from the query.
To test LLMs based on non-parametric memory, we translate the facts into natural language by simple templates (Appendix F). Facts/queries for each attribute are grouped/tested separately. We test both the vanilla setup where all facts (28.2K on average) are loaded into the LLM context, and the retrieval-augmented setup (5.4K facts retrieved on average) where the two-hop neighborhoods of the two query entities are retrieved, which includes enough facts to deduce the answer. We also try both standard prompting where the model answers directly, and chain-of-thought (CoT) prompting where the model is prompted to verbalize the reasoning. We test GPT-4-Turbo and Gemini-Pro-1.5, where for GPT-4-Turbo we only test the retrieval-augmented setup due to context length limit.
Table 1:Results on the complex reasoning task. Direct/CoT: predict the answer directly/verbalize the reasoning steps. “+R”: retrieval augmentation.
GPT-4-Turbo Gemini-Pro-1.5 Grokked Transformer
Direct+R CoT+R Direct CoT Direct+R. CoT+R
Accuracy (%) 33.3 31.3 28.7 11.3 37.3 12.0 99.3Before Devon was released I started building a AI Software Engineer after reading the Google "Self-Discover Reasoning Structures" paper. I was always put off looking at the LangChain API so decided to quickly build a simple API that fit my design style. Once a repo is checked out, and its decided what files to edit, I delegate the code editing step to Aider. The runAgent loop updates the system prompt with the tool definitions which are auto-generated. The available tools can be updated at runtime. The system prompt tells the agents to respond in a particular format which is parsed for the next function call. The code ends up looking like:
export async function main() {
initWorkflowContext(workflowLLMs);
const systemPrompt = readFileSync('ai-system', 'utf-8');
const userPrompt = readFileSync('ai-in', 'utf-8'); //'Complete the JIRA issue: ABC-123'
const tools = new Toolbox();
tools.addTool('Jira', new Jira());
tools.addTool('GoogleCloud', new GoogleCloud());
tools.addTool('UtilFunctions', new UtilFunctions());
tools.addTool('FileSystem', getFileSystem());
tools.addTool('GitLabServer',new GitLabServer();
tools.addTool('CodeEditor', new CodeEditor());
tools.addTool('TypescriptTools', new TypescriptTools());
await runAgent(tools, userPrompt, systemPrompt);
}
@funcClass(__filename)
export class Jira {
/**
* Gets the description of a JIRA issue
* @param {string} issueId the issue id (e.g XYZ-123)
* @returns {Promise<string>} the issue description
*/
@func
@cacheRetry({scope: 'global', ttlSeconds: 60*10, retryable: isAxiosErrorRetryable })
async getJiraDescription(issueId: string): Promise<string> {
const response = await this.instance.get(`/issue/${issueId}`);
return response.data.fields.description;
}
}
New tools/functions can be added by simply adding the @func decorator to a class method. The coding use case is just the beginning of what it could be used for.I'm busy finishing up a few pieces and then I'll put it out as open source shortly!
So far I've been able to make a little workflow that can complete real simple infrastructure requests in JIRA. Pick the right repository, make the changes, compile, and push up the merge request.