suchintan

We DEFINITELY will. I think we're planning on pushing that next week -- we've been super excited about it

Just created this: https://github.com/Skyvern-AI/skyvern/issues/72

Nice! Thanks for sharing this.

We tried approaches like VimGPT before but found the rate of hallucinations to be a bit too high to be used in production. The sweet spot definitely seems to be to combine the magic of Dom parsing AND vision

We're going to definitely work on logic generation and execution, but we're taking it a bit more carefully. Many of the workflows we automate have changing workflow steps (ie I've never seen the exact same Geico flow twice), but this certainly isn't true for all workflows

We're quite different than LaVague. LaVague passes in the entire HTML DOM to the LLM to help it generate XPaths and valid Selenium code. (https://github.com/lavague-ai/LaVague/blob/main/src/lavague/...)

Try this at your own risk.. any reasonable website would result in extraordinarily high input token costs

We spend quite a bit of our time building a layer between the HTML and the LLM call to distill important pieces of information down to actions the LLM can take.. better weighing cost vs output. We're still not at 100% coverage.

Thanks! We definitely experimented with V only (that's the dream), but there's too much context missing:

1. What's behind a select option? You don't know until you click it, which means you need another iteration. This sucks. 2. How do you consistently correlate things in the images to actual actions (ie upload a file to a file input, click on a button, insert a date into a date)? Having the additional HTML Tag information dramatically improves the action selection process (click vs upload vs type)

Saw the launch yesteday. Love all of the excitement in the space!

LaVague is all about generating selenium code to interact with a specific page, and do it step-by-step

Skyvern is all about taking a simple instruction and converting it to a series of LLM-driven actions. It's meant to be more autonomous ("tell Skyvern what to do")

I love all of these ideas!!

1. You can set a "max steps" limit when you run it locally https://github.com/Skyvern-AI/skyvern/blob/d0935755963b017ed...

We also spit out the cost for each step within the visualizer. Click on any task > Steps > there's a column that's dedicated to how much things cost to run

https://github.com/Skyvern-AI/skyvern/issues/70

2. We have a roadmap item to "cache" or "memorize" specific tasks, so you pay the cost once, and then just run it over and over again. We're going to get to it soon!!

We have a simple API we're building as a part of our cloud offering. It's in private beta today -- if you'd like to check it out please email me at suchintan@skyvern.com and I'd be happy to chat

Thanks for the feedback re: costcotravel.com Skyvern definitely does NOT have 100% coverage of the web. This is one of the reasons we were excited to open source -- so we could learn about more websites where it doesn't work as expected

I've filed an issue for this case here: https://github.com/Skyvern-AI/skyvern/issues/69

Ah cool -- we weren't familiar with OpenAdapt. Will check it out.

One big decision we made was to focus on browser automations (instead of computer automation like Adept or OpenAdapt). The reason for this was that we wanted to leverage the information available inside of a DOM to improve the quality of our agent's actions. We found that relying on image-only analysis with X,Y coordinate interactions wasn't able to offer high enough reliability for production workflows

thank you!!

We tend to charge per request our users send us.. although the exact amount depends a lot on the exact task you want to run. Want to send Skyvern on a 40+ page journey to answer a question? It's a bit more expensive than just navigating to a page and extracting information

I'd love to chat about your use-case. Happy to follow-up over email (suchintan@skyvern.com) or over a quick call (https://meetings.hubspot.com/suchintan)

We know very little about this space, except that the entire process is a little bit crazy.

We've talked to a few companies now that would use a product like Skyvern to just automate billing information gathering to make sure patients don't get screwed in the billing process

Are you open to chatting? I'd love to pick your brain about what's behind the kimono

suchintan@skyvern.com or https://meetings.hubspot.com/suchintan

Love this video

> self-operating-computer This is quite different than https://github.com/OthersideAI/self-operating-computer

Self-operating-computer uses pixel mapping to control your computer. This is a very good approach, but it's extremely unreliable. GPT-4V frequently hallucinates pixel outputs, causing it to miss interactions, or enter fail-loops

>The approach by AI Jason

AI Jason is using image-only methods to interact with the browser. This is a great first step, but this approach tends to be rife with hallucinations or errors. We do dom parsing in addition to image anaylsis to help GPT-4V correlate information in the image to the interactable elements within the DOM. This dramatically boosts its ability to perform the same task over and over again reliably (which proved impossible with the image-only approach)

That's a great idea! I hadn't thought of pen-testing as a possible value prop for this product

It's surprisingly dumber than you think!

I'm always fascinated by how far you can get with heuristics in certain situations. Check out the code here -- https://github.com/Skyvern-AI/skyvern/blob/d0935755963b017ed...

I've heard of a lot of success sifting through email spam using custom gmail scripts + GPT-4. Kind of interesting that we can use LLMs to both create and detect spam to some degree of effectiveness

2FA and logged-in experience is sort of a proxy for eID. I suspect that's why so many companies require that you log in with something that knows your identity (log in with google), or ask you for your phone number to confirm your account

Agreed.

We didn't open source this functionality on purpose, and are very very specific about what use-cases we onboard that require it.

That being said, we've gotten to learn a lot more about browser fingerprinting and captcha solving and it's a really interesting space.

If you're curious about it, check out this blog post: https://antoinevastel.com/bot detection/2018/01/17/detect-chrome-headless-v2.html

You're right -- we should have written days to weeks.

What's interesting here is that large companies like UI Path charge thousands of dollars to build a single robot for companies.. I wonder if that large up-front expense will still be necessary in this new world

I agree -- this will likely get commoditized, which is why we didn't focus on making this a chrome extension. The API access pattern makes this particularly appealing as you can run multiple instances in the cloud

I think this is pretty interesting -- I wonder if websites could allow agents to self-identify, and not count them towards advertising CPM to prevent dilution in the advertising metrics

Perhaps a similar thing as robots.txt is in order (agents.txt?)

Nice! We love what you're doing at Kadoa.

We're trying our best not to move into the web scraping space -- we're focusing on automating uncreative, boring, tedious tasks.

We've seen a lot of success going after form-filling on government websites, which would usually be very boring, but happens to work pretty well for us

Yes absolutely. You can prompt it to "terminate" if some state isn't met (ie XYZ text isn't displayed on the screen), and treat terminated results as failures

For example, you could instruct it to go to hackernews and terminate if you don't see a comment from giamma by passing in this payload:

{ "url": "https://news.ycombinator.com", "navigation_goal": "goal is met if you see a post from giamma. Terminate if you don't" }

This is a great call-out. It's something currently in our roadmap

Re: cost for execution. This really depends on the page, but currently it costs between 5 cents and 20 cents per page to execute (today).

We have an improvement planned to help it "remember" or "cache" actions it's done in the past so it can just replay them and bring the cost down to near zero.

Re: LLMs it's capable of working with, currently it's only GPT-4V. I'll get this updated soon!

I suspect that web traffic will encapsulate both. Many websites (government ones in particular) aren't interested in API-based access patterns.

This kind of pattern makes it so you can serve both users and agents with a single interface

This is a really good question we've thought a lot about

You're right that this kind of escalation is inevitable

a. From a business POV, we don't onboard any types of use-cases that we think go against the spirit of a good free web. I've had people ask if they could use our product to create Reddit voting or spamming rings and we didn't entertain it

b. From an open source POV, we prefer technologies like these be open source so website owners and other businesses can know what can happen, and decide how to approach it. Tools like selenium have existed for a long time -- largely to the benefit of the world!

Or putting it into the image for the screenshot-driven agents a la https://simonwillison.net/2023/Oct/14/multi-modal-prompt-inj...

You can message me on our discord or email me suchintan@skyvern.com

Haha we are not unique there. We chose AGPL-3 as well -- become some would argue it's like an open source virus -- everything it touches must become open source! How exciting.

I'll create an issue to create a Docker file for Skyvern. Would make that much easier