We execute the code in a sandbox and proxy the fetch calls through main world!
190 karma · joined December 19, 2024
We are thinking through on self healing mechanisms like falling back to a live web agent and rewriting script.
Since it runs inside your own browser, there should be no captchas or challenges. On failure it can fallback to our regular web agent that can solve captchas.
Big picture wise with the launch of Mythos it might just become impossible for websites to keep up, and they will have to go like Salesforce and just expose APIs for everything.
Compared to your our approach, we are doing this entirely within a browser extension so meeting users where they already doing their existing work.
Within the extension just record doing a task, we reverse engineer the APIs and write a script. Then execute the script from within the webpage so that auth/headers/tokens get automatically added.
You can just prompt to supply parameters and reuse the script at zero token cost.
Use cases we were targetting is like Instagram DMs or LinkedIn connection requests but it should also work for your healthcare use case!
Deeper dive: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...
I really don't want to install an app on to my laptop, especially an MITM, so I think extension would be better.
Our recent technical write up on network discovery/ranking/codegen: https://www.rtrvr.ai/blog/ai-subroutines-zero-token-determin...
Compared to your our approach, we are doing this entirely within a browser extension so meeting users where they already doing their existing work.
Within the extension just record doing a task, we reverse engineer the APIs and write a script. Then execute the script from within the webpage so that auth/headers/tokens get automatically added.
You can just prompt to reuse the tools at zero token cost.
Each row spins up a parallel agent, columns mapped as input, agent executes and writes new columns as output.
We tried initial implementation of this with rtrvr.ai building out Sheets Workflows, but I can't help but feel that there is a thread we're pulling towards a deeper insight on this
- agentic visitors to your site can rate, comment, leave notes on your site
- embed our script tag, it will leave instructions on how to call api for leaving feedback as well as track agentic trajectories and identify failures
- more and more of a websites visitors will be agents, but there's no solution to collect metrics and surface analytics on these agents
Deep dive: https://www.rtrvr.ai/blog/roverbook-posthog-for-ai-agents
On the other hand we construct our own custom Agent Accessibility Trees to represent webpages to models. This approach leads to twice as good performance in WebBench of 300+ tasks (81% vs 40%)
The hard part of raw HTTP scraping was always (1) finding the endpoints and (2) recreating auth. Your browser already has both. We built Vibe Hacking to let the agent use them.
The agent navigates the page, captures network activity, and generates scripts that replay those API calls at scale. Auth propagates automatically because it runs from inside the page.
We tested it on X, it pulled 2,000+ followed profiles despite the UI capping at 50.
DOM-native, no vision/screenshots, #1 on Halluminate WebBench (81.39%). Chrome extension, Gemini Flash Lite default (500 free req/day). Two ex-Google engineers, bootstrapped, 25K+ users.
Happy to answer questions on architecture or limitations.
You can actually try it out on our own site rtrvr.ai
- the agent takes actions on a page
- network calls recorded
- agent writes script to hit endpoints directly at scale
- requests made from main world of webpage so automatically get auth headers added
Basically what you do manually but done via the agent in a minute and for FREE from an AI Studio API key
There is no way 5-11 AM PT is peak traffic
We just launched Rover (https://rover.rtrvr.ai/) as the first Embeddable Web Agent.
Similar principles, just embed a script tag and you get an agent that can type/click/select to onboard/demo/checkout users.
I tried on your website and it was reeaaaally slow. Quick question:
- you are injecting numbering on to the UI. Are you taking screenshots? But I don't see any screenshots in the request being sent, what is the point of the numbering?
I don't think building on browser-use is the way to go, it was the worst performing harness of all we tested [https://www.rtrvr.ai/blog/web-bench-results]. We built out our own logic to build custom Action Trees that don't require any ARIA or accessibility setup from websites.
Would love to meet and trade notes, if possible (rtrvr.ai/request-demo)!
To exist is to recognize the material constraints of reality; there are things humans won't ever discover. Ergo we have to prioritize what is useful.
This proposal is not useful. It goes against the fundamental interest of website owners to differentiate and build up a moat around direct user relationship and data. WebMCP is frankly just a land grab attempt by Google to get more free stuff from publishers.
I update my website multiple times a day. I want to have as much decoupling as possible. Everytime I update internal API, I dont want to think of having to also update this WebMCP config.
Basically I have to put in work setting up WebMCP, so that Google can have a better agent that disintermediates my site.
> Trying to keep users from seeing what data they want is, generally, not something I favor.
This is literally the whole cat and mouse game of scraping and web automation, sites clearly want to protect their moat and differentiators. LinkedIn/X/Google literally sue people for scraping, I don't think they themselves are going to package all this data as a WebMCP endpoint for easy scraping.
Regardless of your preferences/ideals, the ecosystem is not going to change overnight due to hype about agents.
> Your site running its own agent is going to take a lot of resources
A lot of sites already expose chatbots, its trivial to rate limit and captcha on abuse detection
Most sites don't want to expose APIs or care enough about setup and maintenance of said API.
Thats why I dont see this standard going to takeoff.
Google put it out there to see uptake. Its really fun to talk about but will be forgotten by end of year is my hot take.
Rather what I think will be the future is that each website will have its own web agent to conversationally get tasks done on the site without you having to figure out how the site works. This is the thesis for Rover (rover.rtrvr.ai), our embeddable web agent with which any site can add a web agent that can type/click/fill by just adding a script tag.
- Google reduced AI Studio's free rate limits by 1/10th
- Perplexity imposing rate limits, card filing to continue free subscriptions
- Now Anthropic as well
There has been a false narrative that AI will get cheaper and more ubiquitous, but model providers have been stuck in a race for ever more capabilities and performance at higher costs.
On the reliability front we offer integrations like Recordings to ground the agent on trajectories even as the underlying website updates and Knowledge Base of your whole domain.
You the website owner can provide additional guidance to the agent.
There are a lot of users on our rtrvr.ai/cloud platform wanting to automate sites where they have to login to and right now are resorting to including usernames/passwords in prompts (presumably these are non critical sites that they dont really care about). We are offering a more secure option of just syncing cookies from our Chrome Extension to our cloud browsers so the agents will always be logged in without any credential exposure.
You choose the specific domains for the cookies you want synced so its not like all your cookies are syncing.
So our core technical moat is building up an agentic harness that can represent and take actions on any webpage without any screenshots. With this approach we even beat custom trained models like OpenAI Operator and Anthropic CUA: https://www.rtrvr.ai/blog/web-bench-results
Everyone else in the space just takes a screenshot and asks a model what coordinates to click, our core thesis is that LLMs understand semantic representations fundamentally better than vision. But with this DOM approach there is a long tail of HTML/DOM edge cases to cover that we have built out for with the 20k+ users bringing these edge cases.
Soon you will be able to record demonstration tasks via our partner Chrome Extension as well as setup knowledge bases scraped by our Cloud browsers to provide additional context to the agent. So there is a platform moat as well.
The audience is website owners who want to increase visitor engagement and conversion via a conversational interface for users.
This is more for our cloud browser platform where we launch cloud browsers for vibe scraping controlled via a custom extension instead of CDP. You can try it out at rtrvr.ai/cloud, where we can get back some strong antibot detection sites like google.com
All that matters is that the deliverable agent works seamlessly without bugs/issues, which our web agent excels at even beating OpenAI Operator and Anthropic CUA in benchmarks!
We completely rewrote our launch blog post and included a demo video of the embedded agent live!
The underlying agentic performance though is undeniable.
Also like whats the point if something is AI generated, we do thorough review to ensure readibility, coherence and accuracy.