111 karma · joined April 1, 2020
How do your agents decide a suspected issue is a validated vulnerability, and what measured false-positive/false-negative rates can you share?
How is customer code and data isolated and encrypted throughout reconnaissance, exploitation, and patch generation (e.g., single-tenant VPC, data-retention policy)?
Do the agents ever apply patches automatically, or is human review required—and how does the workflow integrate with CI/CD to prevent regressions?
Ty!
I also like that you're transparent about how you might eventually introduce additional revenue streams like margin lending or maybe even PFOF. Knowing that upfront is better than a sudden terms-of-service surprise down the road. Still, I'd hope you'll consider giving users some say over how their shares are handled — like opting out of lending — so your incentives stay aligned over the long run.
Congrats on hitting $10M AUM. I'm rooting for more low-fee alternatives that keep the user in the loop!
Building product, talking to customers and making something people want!
It should be pretty straightforward at first glance!
We're working on a fix right now!
Sometimes, I even ‘talk’ into Cursor’s chat window instead of typing. The only downside? It can get a bit annoying for others when you're talking to yourself all day.
Making current voice agents reliable is incredibly time-consuming and complex. This challenge has kept many teams from pushing their agents into production. Those who do launch often release a very limited, basic version to minimize risk. We frequently talk to teams in both camps.
As a result, there aren't many 'killer' voice products on the market right now. But as models improve, we'll see more voice-centric companies emerge.
Teams are already calling their agents by hand and keeping track of experiment runs in a spreadsheet. We're just automating the workflow and making it easier to run experiments!
I'll reach out async!
Regarding text-based evals — part of testing voice agents involves assessing their core reasoning logic. To do that, we bypass the voice layer and simulate conversations via text. So yes, the core simulation engine is reusable for both conversational text and voice interactions.
We're also excited about shipping the ability to replay a simulated conversation inspired by a real user!
We've discovered that it's often faster for non-technical domain experts to iterate on prompts in a structured, eval-driven way, rather than relying on engineers to translate business requirements into prompts.
While storing prompts in code offers version control benefits, it can hinder collaboration. On the other hand, using a pure CMS for prompts enhances collaboration but sacrifices some modern software development practices.
We're working towards a solution that bridges this gap, combining the best of both approaches. We're not there yet, but we have a clear roadmap to achieve this vision!
Few examples:
1) Customer service: Simulating challenging customer interactions could help reps develop patience and problem-solving skills.
2) Emergency responders: Creating realistic crisis scenarios (like 911 calls) that could improve decision-making under pressure.
3) Healthcare: Virtual patients with complex symptoms could speed up the learning rate for med students.
4) Conflict resolution: Practicing with difficult personalities could aid mediators and negotiators.
5) Sales: AI-simulated tough customers could help salespeople refine their pitches and objection-handling skills in a low-stakes environment.
Thoughts?
Testing for edge cases is especially important because getting an order wrong can cause health hazards, long line-ups, and churn!
I'm happy to spin up some scenarios that are more relevant for you instead of our stock demo personas :)
Feel free to email me at sumanyu@hamming.ai
Our current product draws inspiration from Hamming distance because we're comparing the `distance` between current LLM output vs. desired LLM output.
TLDR - GPT-3.5-Turbo showed lower accuracy on the BICS benchmark than the BABILONG benchmark at the same context length and target depth, indicating that LLMs struggle more on code-based tasks than text-based tasks at long context length. - The hype is real. GPT-4o showed the best performance, closely followed by GPT-4-Turbo. The GPT-4-Series especially performed well at long context lengths compared to other models. - Llama3-70B reached GPT 3.5-Turbo levels (yay open-source!) - Gemini 1.0 Pro was bad across the board (super surprising)
Link to repo: https://github.com/HammingHQ/bug-in-the-code-stack See full results here: https://hamming.ai/blog/bug-in-the-codestack
GPT-3.5-Turbo showed lower accuracy on the BICS benchmark than the BABILONG benchmark at the same context length and target depth, indicating that LLMs struggle more on code-based tasks than text-based tasks at long context length.
GPT-4o showed the best performance, closely followed by GPT-4-Turbo. The GPT-4-Series especially performed well at long context lengths compared to other models. Gemini-1.0-pro performed the worst, surprisingly worse than Llama3-70B.
Generally, longer context length resulted in lower accuracy. However, there were some exceptions to this. Models react differently to the placement of the bug within the source code. GPT-3.5-Turbo and Claude 3 Opus were the most sensitive, and GPT-4-Series was the least sensitive. Generally, less sensitivity means a more robust model.
This benchmark has lots of limitations. I would love your feedback & suggestions on how we can make this benchmark more useful!
Link to results: https://hamming.ai/blog/bug-in-the-codestack Repo: https://github.com/HammingHQ/bug-in-the-code-stack