HNHacker News
TopNewBestAskShowJobs

campers

356 karma · joined June 3, 2014

DevOps/SRE lead for TrafficGuard - Ad fraud protection platform with clients such as Disney Streaming, Gojek, BetFred, Bet365, Singtel. https://www.trafficguard.ai

Java/Kotlin/JS/TypeScript/Node.js/Google Cloud developer in Perth, Australia

https://apporchestra.com

daniel.campagnoli@trafficguard.ai https://www.linkedin.com/in/danielcampagnoli/

meet.hn/city/au-Perth

Socials: - linkedin.com/in/danielcampagnoli

Interests: AI/ML, DevOps, Cybersecurity, Outdoor Activities, Open Source

---

submissionscomments
campers··on Zed AI
That was a part of the reasoning of open sourcing my AI assistant/software dev project. Companies like Google have strict procedures around access to customer data. The same can't always be said about a startup racing to not run out of cash.
campers··on Former Google CEO Eric Schmidt's Leaked Stanford Talk
Agreed, such a critical piece of infrastructure, the mind boggles. I'll have to see what I can pry out of our Google account team about it.
campers··on Former Google CEO Eric Schmidt's Leaked Stanford Talk
Another way to look at it is everyone's productivity will be expected to increase to match the productivity increases by the competition who are also using AI. If you don't skill up on how to effectively use AI, then we'll find someone else who has.
campers··on Former Google CEO Eric Schmidt's Leaked Stanford Talk
There's definitely room to build specification builder agents, that have access to documentation and previous specifications.

The other day I was looking into adding Trusted Types in the Content-Security-Policy header, which was something new to me. In my chat with Claude I asked:

"Lets brainstorm 10 a list of ideas closely related to this so we can think of anything we might be missing on the topic to consider."

And that provided a good list of items to review to consider and expand out the sphere of thinking for the LLM.

It is an infuriatingly hard problem to have the LLM produce excellent results every single time, and have it just do everything and want it to read our mind and all the knowledge and context of a task. I think we'll make some good progress over the next few years as agentic workflows are built out to mimic out thought processes, and the cost/capability of the LLMs keeps improving.

campers··on Hermes 3: The First Fine-Tuned Llama 3.1 405B Model
Ask it to output the occurrences as items in a JSON array, then count the array items in code
campers··on Prompt Caching with Claude
Nice to see Anthropic providing this while Claude is my go to model.

I prefer DeepSeek's new context caching implemention being completely automatic and code free. https://platform.deepseek.com/api-docs/news/news0802/ And without a 25% premium

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
Actually when I was bouncing ideas off Claude it was suggested with the alternative spelling of noos. Then I can keep the concept and only have have one letter to change
campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
To have it deployed costs nothing to run with the Cloud Run and Firestore free tier.

As for LLM costs that really depends what you're trying to do when it. Fortunately that cost is always coming down. When I was first building it with Claude Opus the costs did add up, but 100 days later we have 3.5 Sonnet at a fraction of the cost.

The Aider benchmarks are good to see how different LLMs perform for coding/patch generation. Sonnet 3.5 is best if it's in the budget. DeepSeek coder v2 gives the best bang for buck https://aider.chat/2024/07/25/new-models.html

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
Updating the Dockerfile and docker-compose.yml was the last change I made so glad to hear that worked for you! What change did you make to the docker compose file?

The CLI scripts under src/cli would be the best examples currently to have a look at for running an autonomous agent, and the fixed workflows (e.g code.ts)

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
I have a few steps so far in the code editing at https://github.com/TrafficGuard/nous/blob/main/src/swe/codeE... There is a first pass I initially created when I was re-running a partially completed task and it would sometimes duplicate what already had been done. This helps Aider focus on what to do.

  <files>${fileContents}</files>
  <requirements>${requirements}</requirements>
  You are a senior software engineer. Your task is to review the provided user requirements against the code provided and produce an implementation design specification to give to a developer to implement the changes in the files.
  Do not provide any details of verification commands etc as the CI/CD build will run integration tests. Only detail the changes required in the files for the pull request.
  Check if any of the requirements have already been correctly implemented in the code as to not duplicate work.
  Look at the existing style of the code when producing the requirements.
Then there is a compile/lint/test loops which feeds back in the error messages, and in the case of compile errors the diff since the last compiling commit. Aider added some similar functionality recently.

Then finally there's a review step which asks:

  Do the changes in the diff satisfy the requirements, and explain why? Are there any redundant changes in the diff? Was any code removed in the changes which should not have been? Review the style of the code changes in the diff carefully against the original code.  Do the changes follow all the style conventions of the original code?
This helps catch issues that Aider inadvertently introduced, or missed.

I have some ideas around implementing workflows that mimic what we do. For example if you have a tricky bug, add a .only to the relevant describe/it tests (or create tests if they dont exist) add lots of logging and assertions to pinpoint the fix required, then undo the .only and extra logging. Thats whats going to enable higher overall success rates, which you can see the progress in the SWE-bench lite leaderboard as simple RAG implementations had up to ~4% success rate with Opus, while the agentic solutions are reaching 43% pass rate on the full suite.

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
Good question, at first I only called the fully autonomous agents as agents, as to me that's what having agency is. I didn't like when other projects had "multi-agent" when it's just a bunch of llm calls.

Initially the coding and software dev agents were called workflows, but to make it more agenty I was ok with it being called an agent if the result of an llm call affected the control flow

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
Thanks! I've spent a lot more time on the computer than I would like over the last few months building it.

If you think you might want to feel free to get in touch

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
Thanks! I posted a reply to another comment about the name clash. I thought I could add another weird to differentiate, but Nous Agents doesn't really roll off the tongue. New name ideas welcome!
campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
When I first picked the name, after a chat with Claude, I hadn't come across Nous Research back then, and they didn't show up Googling for just nous.

I see a bit of reuse of words in other various llm related projects.

Langchain/langfuse/langflow

Llama/ollama/llamaindex

so I hadn't been too worried about it when became aware of them.

That's what Show HN is for, getting feedback, and a name changed now would be easy before I post it around more.

campers··on Show HN: Nous – Open-Source Agent Framework with Autonomous, SWE Agents, WebUI
I can't take credit for that particular screen, it's the Trace UI in Google Cloud. I did look at LangSmith for tracing, but for now I wanted to stick with standard OpenTelemetry tracing, so you could export the spans to Honeycomb etc
campers··on Structured Outputs in the API
Thats ok from the perspective of it making room for a more capable and expensive GPT5 model to compete with Opus 3.5 when that arrives this year. The significant price drops for a small loss in quality is a reasonable tradeoff. Then GTP4o becomes the mid tier and GTP4o-mini the low tier.

There was 100 days in between Claude 3.0 Opus and Claude 3.5 Sonnet being released which gave us similar capability at a 80% price reduction. When I was using Opus I was thinking this is nice, but the cost does add up. Having Sonnet 3.5 so soon after was a nice surprise.

One more round of 80% price cuts after that combined with building out the multi-step agentic workflows should provide some decent capabilities!

campers··on Structured Outputs in the API
It is difficult to get the AI models to get everything right every time. I noticed too that it would sometimes remove comments etc when re-writing code.

The way to get better results is with agentic workflows that breakdown the task into smaller steps that the models can iteratively come to a correct result. One important step I added to mine is a review step (in the reviewChanges.ts file) in my workflow at https://github.com/TrafficGuard/nous/blob/main/src/swe/codeE...

This gets the diff and asks questions like:

- Are there any redundant changes in the diff? - Was any code removed in the changes which should not have been? - Review the style of the code changes in the diff carefully against the original code.

Maybe try using that, or the package that I use which does the actual code edits called Aider https://aider.chat/

campers··on Superconducting Microprocessors? Turns Out They're Ultra-Efficient (2021)
Imec working on this approach too https://spectrum.ieee.org/superconducting-computer
campers··on Large Enough
It seems to be the way with these releases, sticking with Claude, at least for the 'hard' tasks. In my agent platform I have LLMs assigned for easy/medium/hard categorised tasks, which was somewhat inspired from the Claude 3 release with Haiku/Sonnet/Opus. GPT4-mini has bumped Haiku for the easy category for now. Sonnet 3.5 bumped Opus for the hard category, so I could possibly downgrade the medium tasks from Sonnet 3.5 to Mistral Large 2 if the price is right on the platforms with only 123b params compared to 405b. I was surprised how much Llama3 405b was on together.ai $5/mil for input/output! I'll stick to Sonnet 3.5. Then I was also surprised how much cheaper Fireworks was at $3/mil

Gemini has two aces up its sleeve now with the long context and now the context caching for 75% reduced input token cost. I was looking at the "Improved Facuality and Reasoning in Language Models through Multi-agent debate" paper the other days, and thought Gemini would have a big cost advantage implementing this technique with the context caching. If only Google could get their model up to the level of Anthropic.

campers··on Show HN: We made glhf.chat – run almost any open-source LLM, including 405B
I noticed Fireworks.ai is only $3/million compare to $5 for together.ai for Llama3 405b

And in Google Vertex its free at the moment!

campers··on AI's $600B Question
Etched is another dedicated inference hardware company that recently announced their product. It only works for transformer based models, but is ~20x faster than a H100
campers··on Show HN: Improve LLM Performance by Maximizing Iterative Development
It feels a step backwards developing with a such a minimal type system. Seeing so many projects in Python was one reason we're open sourcing our LLM/agentic framework, to give more options for the TypeScript community. We haven't launched it but the code it up at https://github.com/trafficguard/nous
campers··on Claude 3.5 Sonnet
Another one is https://www.github.com/trafficguard/nous which provides a software dev agent that can find/clone repos, create a branch, search in a repo, then delegates to Aider for the code editing in the edit/compile/lint/test loop, and raise a merge request.
campers··on Claude 3.5 Sonnet
From the Anthropic model guide:

Agentic Coding Claude 3.5 Sonnet solves 64% of problems on an internal agentic coding evaluation, compared to 38% for Claude 3 Opus. Our evaluation tests a model’s ability to understand an open source codebase and implement a pull request, such as a bug fix or new feature, given a natural language description of the desired improvement. For each problem, the model is evaluated based on whether all the tests of the codebase pass for the completed code submission. The tests are not visible to the model, and include tests of the bug fix or new feature. To ensure the evaluation mimics real world software engineering, we based the problems on real pull requests submitted to open source codebases. The changes involve searching, viewing, and editing multiple files (typically three or four, as many as twenty). The model is allowed to write and run code in an agentic loop and iteratively self-correct during evaluation. We run these tests in a secure sandboxed environment without access to the internet.

% of problems which pass all tests Claude 3.5 Sonnet 64% Claude 3 Opus 38% Claude 3 Sonnet 21% Claude 3 Haiku 17%

campers··on Claude 3.5 Sonnet
I'm excited to test this out! I've been building an open source TypeScript agentic AI platform for work (DevOps related with an autonomous agent and software engineer workflow). The Claude 3 models had an influence on the design with their tuning on using XML and three levels of capabilities, and have been my preferred models to use.

I remember having moments looking at the plans Opus generated and being impressed with its capabilities.

The slow speed of requests I could deal with, but the costs could quickly add up in workflows and the autonomous agent control loop. When GPT4o came out at half the price it made Opus quite pricey in comparison. I'd often thought if I could just have Opus capabilities at a fraction of the price, so its a nice surprise to have it here sooner that I imagined!

The agent platform isn't officially launched yet, but its up at https://github.com/trafficguard/nous

I never liked the Langchain API when I looked at the examples so I built it from scratch. It has an autonomous agent with a custom XML-based function calling, memory and call history. The software engineer agentic workflow I initially dog-fooded with a prompt like "Complete Jira XYZ-123". So it gets the Jira description, finds the appropriate terraform project in GitLab, clones, edits (delegating to Aider), creates a MR and messages on Slack. It also has a UI for running agents, human-in-the-loop interactions etc.

campers··on Grokked Transformers Are Implicit Reasoners
This is the interesting result where their GPT-2 sized transformer blows away GPT4 and Gemini 1.5 in connecting together facts

  The difficulty of such a task is two-fold. First, the search space is large. For example, on average, each query entity connects with more than 50 facts, and each bridge entity in the ground truth proof connects with more than 900 facts. Second, there are no surface form clues to exploit and bias the search towards the ground truth proof, unlike most conventional QA benchmarks where the proof steps are transparent from the query.

  To test LLMs based on non-parametric memory, we translate the facts into natural language by simple templates (Appendix F). Facts/queries for each attribute are grouped/tested separately. We test both the vanilla setup where all facts (28.2K on average) are loaded into the LLM context, and the retrieval-augmented setup (5.4K facts retrieved on average) where the two-hop neighborhoods of the two query entities are retrieved, which includes enough facts to deduce the answer. We also try both standard prompting where the model answers directly, and chain-of-thought (CoT) prompting where the model is prompted to verbalize the reasoning. We test GPT-4-Turbo and Gemini-Pro-1.5, where for GPT-4-Turbo we only test the retrieval-augmented setup due to context length limit.

  Table 1:Results on the complex reasoning task. Direct/CoT: predict the answer directly/verbalize the reasoning steps. “+R”: retrieval augmentation.

                      GPT-4-Turbo      Gemini-Pro-1.5        Grokked Transformer
               Direct+R CoT+R  Direct CoT  Direct+R. CoT+R
  Accuracy (%)  33.3     31.3   28.7  11.3  37.3     12.0       99.3
campers··on Tool Use (function calling)
That looks like a nice concise API too. Naming is always tricky, I like the toolbox name, but then should I rename the @func decorator to @tool? It seems like function is the more common name for it, which also overloads with the JavaScript function keyword.
campers··on Tool Use (function calling)
I've added contact details to my profile for the moment, drop me an email
campers··on Tool Use (function calling)
I'm not sure if I'll migrate my existing function calling code I've been using with Claude to this... I've been using a hand rolled cross-platform way of calling functions for hard coded workflows and autonomous agents across GPT, Claude and Gemini. It works for any sufficiently capable LLM model. And with a much more pleasant, ergonomic programming model which doesn't require defining the function definition again separately to the implementation.

Before Devon was released I started building a AI Software Engineer after reading the Google "Self-Discover Reasoning Structures" paper. I was always put off looking at the LangChain API so decided to quickly build a simple API that fit my design style. Once a repo is checked out, and its decided what files to edit, I delegate the code editing step to Aider. The runAgent loop updates the system prompt with the tool definitions which are auto-generated. The available tools can be updated at runtime. The system prompt tells the agents to respond in a particular format which is parsed for the next function call. The code ends up looking like:

  export async function main() {
 
   initWorkflowContext(workflowLLMs);

   const systemPrompt = readFileSync('ai-system', 'utf-8');
   const userPrompt = readFileSync('ai-in', 'utf-8'); //'Complete the JIRA issue: ABC-123'

   const tools = new Toolbox();
   tools.addTool('Jira', new Jira());
   tools.addTool('GoogleCloud', new GoogleCloud());
   tools.addTool('UtilFunctions', new UtilFunctions());
   tools.addTool('FileSystem', getFileSystem());
   tools.addTool('GitLabServer',new GitLabServer();
   tools.addTool('CodeEditor', new CodeEditor());
   tools.addTool('TypescriptTools', new TypescriptTools());

   await runAgent(tools, userPrompt, systemPrompt);
  }



  @funcClass(__filename)
  export class Jira {

   /**
    * Gets the description of a JIRA issue
    * @param {string} issueId the issue id (e.g XYZ-123)
    * @returns {Promise<string>} the issue description
    */
   @func
   @cacheRetry({scope: 'global', ttlSeconds: 60*10, retryable: isAxiosErrorRetryable })
   async getJiraDescription(issueId: string): Promise<string> {
     const response = await this.instance.get(`/issue/${issueId}`);
     return response.data.fields.description;

   }
  }
New tools/functions can be added by simply adding the @func decorator to a class method. The coding use case is just the beginning of what it could be used for.

I'm busy finishing up a few pieces and then I'll put it out as open source shortly!

campers··on Show HN: Fructose – LLM calls as strongly typed functions
Nice, I've just started building something similar in TypeScript. I wasn't a big fan of the Langchain model. I wanted to develop with normal functions in an imperative manner so the code is very easy to read. I'm also using decorators to add the required functionality to workflow steps so I can support retries, and build something like LangSmith on top too.

So far I've been able to make a little workflow that can complete real simple infrastructure requests in JIRA. Pick the right repository, make the changes, compile, and push up the merge request.

← PreviousPage 2 of 5Next →