adinagoerres

omg I got it right!!!

awesome to hear!

couldn't agree more

setup guide is here: https://docs.superglue.cloud/mcp/using-the-mcp best used for granular access management to integrations and data, in agent applications

Yes, we've seen this work best with clear instructions around Salesforce, Attio etc. superglue does come with a number of observability and monitoring features, but of course this can be extended for a specific setup.

Adina here, one of superglue's creators. I'm curious to hear folks opinions on this. Also to clarify how agents are able to call APIs via superglue: the first step is to set up auth and systems on superglue so it can process and extract documentation and any other context for calling the APIs, which is then passed on to the agents

How well can agents work with APIs they’ve never seen before? We tested 41 APIs across 8 different LLMs to find out.

API execution is great for benchmarking, because it tests core qualities and limitations of LLMs:_the depth of the data they were trained on, their stateless architecture, context dependency, and reasoning.

Today we're releasing v2 of API-Bench:_a benchmark that tests how well LLMs can execute against APIs. Here are the results: https://superglue.ai/benchmark_v2

Tl;dr:_LLMs fail at integrations because they lack ground truth, lack state, lack debugging ability, and lack access to real system context - everything API integrations fundamentally require.

Here’s what we found:

1. LLMs are only as good as the data they’re trained on:_when docs change, APIs evolve, or systems are niche/long-tail, they use outdated patterns, guess missing pieces and hallucinate endpoints and parameters.

2. LLMs are stateless, but integrations are stateful:_auth handshakes, pagination, retries, multi-step flows all need memory but LLMs can’t persist intermediate values or reason across steps.

3. LLMs produce code that “looks right” but fails at runtime: LLMs cannot isolate the failing step and understand real error messages, so they can’t change what’s broken or retry with new hypotheses.

4. LLMs can’t reliably interpret imperfect API design:_humans can infer the intended function, LLMs will hallucinate what looks reasonable.

We open sourced the benchmark so you can test your own APIs or contribute new ones: https://github.com/superglue-ai/superglue/tree/main/eval/llm...

Curious to hear your experience, and of course always happy to share more learnings.

creator of superglue here. we are trying to crack one of the hardest problems in infra management, with the goal to get rid of nasty systems and expensive consulting projects. always looking for the gnarliest of systems and how to make them future-compatible.

self-reflection is very important for both humans and LLMs, indeed

Hey HN, I'm Adina, Stefan's co-founder at superglue. When we started working on LLM-powered integrations about a year ago, the models were barely good enough to handle simple mappings. We started benchmarking our performance as an internal evals project and thought it would be fun to open source it, to create more transparency around LLM performance. Our goal here is to understand how we can make agents production-ready and improve reliability across the board.

Hi HN! Adina here from superglue. Today I’d like to share a new benchmark we’ve just open sourced: an Agent-API Benchmark, in which we test how well LLMs handle APIs.

tl;dr: LLMs suck at writing code to use APIs.

We ran 630 integration tests across 21 common APIs (Stripe, Slack, GitHub, etc.) using 6 different LLMs. Here are our key findings: - Best general LLM: 68% success rate. That's 1 in 3 API calls failing. Would you ship that? - Our integration layer scored a 91% success rate, showing us that just throwing bigger/better LLMs at the problem won't solve it. - Only 6 out of 21 APIs worked 100% of the time, every other API had failures. - Anthropic’s models are significantly better at building API integrations than other providers.

What makes LLMs fail hard: - Lack of context (LLMs are just not great at understanding what API endpoints exist and what they do, even if you give them documentation which we did) - Multi-step workflows (chaining API calls) - Complex API design: APIs like Square, PostHog, Asana (Forcing project selection among other things trips llms over)

We've open-sourced the benchmark so you can test any API and see where it ranks: https://github.com/superglue-ai/superglue/tree/main/packages...

Check out the repo, consider giving it a star, or see the full ranking at https://superglue.ai/api-ranking/

If you're building agents that need reliable API access, we'd love to hear your approach - or you can try our integration layer at superglue.ai.

Next up: benchmarking MCP.

thank you!

they are in our YC batch! great product

thank you!

huh interesting. we're exploring extraction from html

not that we're aware of!

we're not affiliated