614 karma · joined July 21, 2022
We help companies automate workflows in the web using our open source browser automation tool
Check us out here: https://github.com/Skyvern-AI/Skyvern
I couldn't find where browser-use published their run results (expected to see it here https://github.com/browser-use/eval)
We went ahead and published our full run at https://eval.skyvern.com so our run could be independently audited
We're working on 2 major improvements that will get cost down at scale: 1. We're building a code generation layer under the hood that will start to memorize actions Skyvern has taken on a website, so repeated runs will be nearly free 2. We're exploring some graph re-ranking techniques to eliminate useless elements from the HTML DOM when analyzing the page. For example, if you're looking at the product page and want to add a product to cart, the likelihood you'll need to interact with the Reviews page will be 0. No need to send that context along to the LLM
Let's say that you are a parts procurement shop and want to order 10,000 of SKU1, and 20,000 of SKU2. If you go on parts websites like finditparts.com, you'll see that there is little ambiguity when it comes to ordering specific SKUs
We've seen cases of companies that want to automate item ordering like this on tens of different websites, and have people (usually the CEO) spending a few hours a week doing it manually.
Writing a script can take ~10-20hours to do it (if you know how to code).. but we can help you automate it in <30 minutes with Skyvern, even if you don't know how to code!
What are your opinions on lists like this?
We'll improve it soon!
Just curious, how do you differentiate from other open source competitors like n8n and Active pieces?
1/ the current prompt + payload structure is definitely on the complicated end of the spectrum, but we've found that we can use an LLM to help generate this payload for our users
The technical users want to learn more and generate their own payloads, and the non technical users prompt LLMs to help them generate the ultimate skyvern prompt to get going
This was very unexpected -- but a surprisingly logical chain of events.
Phase 1: build the thing the complex way (playwright) Phase 2: build the playwright thing with complex prompts (we are here right now) Phase 3: build the thing that builds the playwright thing with simpler prompts
Each phase lowers the technical bar to build your automations
2/ re: frequency of website changes
This IMO is a smaller value prop of LLM based automations. The biggest one is being able to handle highly dynamic situations. Consider the case where you're automating an e-commerce website where the popup offer changes every week. skyvern doesn't even notice those, but playwright scripts would break
Similarly, I love using the Geico example because it highlights something that was very difficult to automate before: The form changes every time you run it
Skyvern breezes through it.. but another case that was hard to automate before.
3/ data correctness
We're actually rolling out a workflows feature that allows you to chain multiple tasks together. The cool thing about this feature is that you can add steps in to have Skyvern self-validate it's own unless before continuing.
For example, you can add n products to cart, then navigate to the cart and validate the cart state
... As you can guess, this creates the foundation to have another agent go and use these tools to self-build workflows with simpler prompts
TL;DR -- we're on a pretty long journey to use LLMs to make BPA easier and easier, and this is just the first step
2. We don't want Skyvern to be used on websites that prohibit this kind of behaviour (LinkedIn is the obvious example). Specifically, we didn't open source any of our anti-bot or captcha related code because we get requests to make "Reddit upvote rings" and such. We don't want to support bad actors like that
(3) I think this is a net net good thing. AI browser automations= less need for APIs = no need to maintain both an API and UI = streamlined experience + less code = simpler systems
(4) I'm not 100% sure about this one. We usually just assume companies don't build APIs because they don't have budget for it. Ie for non malicious reasons. Companies like LinkedIn will likely thwart any attempts at automation, but we're not interested in participating in this cat mouse game
This isn't true anymore -- we just need to build and launch support for it
One very subtle advantage of doing HTML analysis is that you can cut out a decent number of LLM calls by doing static analysis of the page
For example, you don't need to click on a dropdown to understand the options behind it, or scroll down on a page to find a button to click.
Certainly, as LLMs get cheaper the extra LLM calls will matter less (similar to what we're seeing happen with Solar panels where cost of panel < cost of labour now, but was reversed the preceding decade)
It's still super early in the open source x vision model space. The limiter actually seems to be the vision encoder -- advancements here will pay off huge dividends
https://huggingface.co/spaces/opencompass/open_vlm_leaderboa...
I never realized how important it is to track invoices in Europe (where VAT needs to be closely tracked), and a large % of vendors require you to log into their portal to fetch them
Their product and launch is super cool. It's incredible how much it's able to do by just relying on tool use + micro agents + screen shots + coordinates to interact with websites.
There are a couple of thoughts here:
(1) Will their competitors wait around and not build something similar? Will xAI / Gemini / OpenAI / Mistral / MetaAI teams wait around? Probably not. This is likely a huge part of the future, and one company will not "take it all"
(2) How is value actually derived from these systems? Is a demo + cool usable product enough? Likely not. Most people actually want their workflow automated. For personal use-cases, this might be enough.. but enterprises likely want something more complex
(3) Will this be optimized for Claude only? What if you want to run this with your own open source LLMs? Or you want to point this at the best model on the market all the time? Will you get that flexibility through a solution provided by a big player? Likely not -- Anthropic has incentive to get you to use Claude under the hood
The last point is the one that gives me hope. Our open source users are able to pick their favourite model to run on. You're not locked into Cluade. You can run it on Gemini / GPT-4O or open source ones such as Llama 3.2.
Toughest was Skyvern accidentally talking to a support agent when the website said "your request failed, please contact support"
https://www.linkedin.com/posts/suchintansingh_we-received-20...
Ex: filling out contact forms on hundreds of websites? It's really tough for normal code to be able to handle that cardinality. No problem for an AI agent
It's going to be our way of beta testing CC transaction and testing them for reliabilty
For example, we're currently using a multi agent architecture where we have micro agents run to analyze SVGs, fill out dynamic autocompletes. This would have been really hard.
Frameworks like langchain are good for early prototyping, but it's too restricting when you want to push the limits