Hacker News Reader

AbhinavXon Stealing Reasoning Traces from Proprietary LLM APIs

Not surprised. On many benchmarks (i.e tau), we have seen the same thing. Probably lots of training on every publicly available benchmark

AbhinavXon Can Claude Code in a loop improve an enterprise AI agent with $10,745 of budget?

>An interesting observation we made was that even though the 3 optimizers were given tens of thousands of dollars in compute, they found the best solution very early on

Honestly, this confirms some bias I already have from coding — models can get to the best solution pretty fast, but if the solution is something obscure, they usually don’t find it.

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

Awesome--let us know what you think!

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

we're working on that right now, we'd love to hear your opinions(if you're interested you can send us an email at team@lucidic.ai).

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

LLM as a judge for agent usually has context overload and even if you have a really good prompt for your evaluation, LLMs hallucinate because there is just too much information to ingest. So we created an agentic pipeline to basically do evaluations on rubrics which have better results and dont miss intricacies due to the overloaded context.

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

We're new to the open source scene so we don't have anything published yet but plan to in the future. A basic overview of the way we do clustering is we condense stateful information -> create a state embedding -> create tags -> cluster based on distance of tags + embeddings.

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

Colloquially, AI agents are just while loops with LLM calls and tool calls. More specifically, what distinguishes an agent from LLM pipelines is that its next step is determined dynamically (based on the output of the previous one) so the execution path isn’t fixed. The boundary between complex LLM chaining and agents is pretty fuzzy, but we support both.

Haha also our whole backend is in Django :)

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

Langfuse and Helicone work well for traditional LLM operations, but AI agents are different. We discovered that AI agents require fundamentally different tooling, here are some examples.

First, while LLMs simply respond to prompts, agents often get stuck in behavioral loops where they repeat the same actions; to address this, we built a graph visualization that automatically detects when an agent reaches the same state multiple times and groups these occurrences together, making loops immediately visible.

Second, our evaluations are much more tailored for AI Agents. LLM ops evaluations usually occur at a per prompt level (i.e hallucination, qa-correctness) which makes sense for those use cases, but agent evaluations are usually per session or run. What this means is that usually a single prompt in isolation didn’t cause an issue but some downstream memory issue or previous action caused this current tool to fail. So, we spent a lot of time creating a way for you to create a rubric. Then, to evaluate the rubric (so that there isn’t context overload) we created an agentic pipeline which has tools like viewing rubric examples, ability to zoom “in and out” of a session (to prevent context overload), referencing previous examples, etc.

Third, time traveling and clustering of similar responses. LLM debugging is straightforward because prompts are stateless and are independent from one another, but agents maintain complex state through tools, context, and memory management; we solved this by creating “time travel” functionality that captures the complete agent state at any point, allowing developers to modify variables like context or tool availability and replay from that exact moment and then simulate that 20-30 times and group together similar responses (with our clustering alg).

Fourth, agents exhibit far more non-deterministic behavior than LLMs because a single tool call can completely change their trajectory; to handle this complexity, we developed workflow trajectory clustering that groups similar execution paths together, helping developers identify patterns and edge cases that would be impossible to spot in traditional LLM systems.

AbhinavXon Launch HN: Lucidic (YC W25) – Debug, test, and evaluate AI agents in production

the way it is integrated (its explained more in the docs) is by installing the python/typescript sdk and writing "lai.init()" at the top of your code. Then we capture all LLM calls and tools with integrated providers (similar to LLM ops platforms). If you want to manually add more information you can add decorators, lai.create_step/create_event "logs", etc.

We then take all this information you give us and try to transform it i.e group together similar nodes, run an agent to evaluate a session, or to find root cause of a session failure in the backend.