oh yeah, we did small-ish sft run to teach the model our trace compression style and so on. the problem with it is that you can't really teach good analysis this way, so RL was the core focus.
227 karma · joined December 21, 2023
flow-1 actually predicts how critical each finding is
I used GSPO because it worked really well for very long prompts (we're dealing with traces analyses so prompts are very very long, and by prompts I mean what goes into LLM call).
We RL-ed flow-1 in our harness, which made it very efficient at using tools it had. Our harness as I said in the post is basically coding agent, it has grep, read and couple of other aux tools. I spent a loooot of time and money on figuring out the right RL recipe and especially the reward. It was really hard to find the right balance between exploration and exploitation.
Some RL runs are still running, because the prod checkpoint as good as it is, thinks a lot. It produces a lot of thinking tokens, which affects the throughput, especially on the large, complex traces to be analyzed. So the next big challenge for me is to make it efficient thinker - i.e. reduce thinking tokens, without losing quality.
I assume you mean tracing voice agents in general. If so, it depends on the where do you want to trace things. Right now, Laminar, unfortunately, doesn't trace voice SDKs like OpenAI's realtime, but we're very actively looking into it.
However, if somewhere in your pipeline you have transcripts of your voice agents, then it shouldn't be that different from tracing any other agent. In this case, Laminar and Signals work perfectly.
We also trained and benchmarked flow-1 on a lot of multi-turn voice agent like conversations.
- SOTA on webvoyager
- browser agent observability
- fast and reliable
- CLI for easier interaction
- available as a serverless API
For the CLI and custom models, you can clone the repo, then go to the cli.py and manually add your model there. I will work on proper support of custom models.
pip install lmnr-index playwright install chromium index run
Also try experimenting with different models. So far, Gemini 2.5 Pro is the best in terms of quality/speed. Claude 3.7 is also pretty good.
- any task that requires UI interaction, button clicking, filter selection, form filling and so on. Just prompt it, it's surprisingly very robust and self-healing.
- complex long-running task that require extensive context - e.g. researching one topic and then creating spreadsheet, creating a presentation for a topic and so on.
Essentially, any task that can be done within a browser environment that previously required flacky hardcoded predefined scripts. Also, website testing is a great example.
here's a demo of CLI https://x.com/skull8888888888/status/1914728292193628330