jonasnelle

I think there are two main reason the fully "self-driving" end-to-end agents that demo well don't work.

1. Planning is hard and exponential decay: Most demos try to start with a single sentence e.g. "order me a Dominos pizza" and go do the whole thing. Turns out planning has been one of the things that LLMs are not that good at. Also, even for a low probability p of failure at a given step, you'd get all steps rights with probability (1-p)^n which gets bad as n grows.

2. Reliability matters and vision is not quite there yet: GPT4V is great, and there have been a handful of domain-specific open source models more focused on understanding screenshots but most of them are not good enough yet to work reliably. And for most applications, reliability is key if you are going to trust the agent to do things on your behalf.

Disclaimer: I'm one of the founders of Autotab (https://www.autotab.com/), we're building a desktop app that lets anyone teach an AI to do a task just by showing it once. We've gone all in on reliability, building our own browser on top of Chromium to give us the bare metal control needed to deliver 98%+ reliability without any site-specific fine tuning.

The other opinionated thing we've done is to focus on "Show, don't tell". We've found that for most important automations it is easier to show the agent the workflow than it would be to write a paragraph describing the steps. If you were to train a human, would you explain where to click or just share your screen & explain with a voice over?

Some stories from our users: One works in IT and sometimes spends hours on- and off-boarding employees (60,000 people company), they need to do 20 different steps across 8 different software applications. Another example is a recruiting company that has many employees looking for candidates and sending messages on LinkedIn all day. In general we mostly see automations that take action or sync data across different software applications.

Have been digging into Playwright a bit - can you say more about the trace viewing to step through a script execution? Very curious!

Fyi we fixed this so now you can get an API key at autotab.com/dashboard and then that is sufficient to authenticate when you run `autotab record`, no more logging in to Google in the Selenium browser!

I agree, to really debug they need coding knowledge today.

But there's something in between where you can say "try again" or even give high level feedback like "don't click the element with that exact title, just click whichever one is first in the list". My working hypothesis is that these are bigger than the ones where you need coding knowledge. Certainly this category will only grow as language models become better.

Katalon looks like a cool project, hadn’t heard of them before! We’re more focused on automations as opposed to testing, which is most important for where we take the product next. More focus for us on handling tasks that are less deterministic/require a bit more intelligence.

We’re actually mainly focused on automations more than testing. We want to build autotab into your intern/assistant for browser-based tasks that you would rather have someone else do.

For example, autotab helped a designer automate client handoffs. He set up an automation that exported assets from Figma and uploaded them to Google Drive, updated his CRM and notified the client.

Yes, we process the contents of the page when users are recording on our servers but don’t store the page HTML. I don’t see any world in which we would sell data, but in case you’re worried about the security of our servers I’m not sure there’s much I can say other than that we use standard security best practices including SSL, 2FA on cloud accounts etc. Does that answer your question?

Thanks so much! Glad to hear it

That’s super helpful, thanks for explaining!

Interesting! We are more focused on automations as opposed to testing - would you say the same applies there as well?

Also Selenium does have 10x the downloads of Playwright on PyPI - is Python different or do you think that metric is misleading?

Interesting! I think the way we’ll handle this is the sane way we handle Google 2FA atm: We have you log in and do 2FA manually once, then we save the logged out cookies.

Why the logged out cookies? That way even if someone gets the cookies they still need the password, but the automation can log in with just your password, no 2FA needed.

How’d you guess my secret plan? ;) Definitely in the cards going forward

Thanks! Excited to see what you do with it

BrowserBox looks cool, I couldn’t tell from the README what browser engine it uses. Did you build your own?

Yes! Two things we’re working on that would be very cool are 1) seeing the mirrored actions executed live as you record and 2) even if the website changes later, using AI at runtime to auto-heal the automation.

Unfortunately we have no plans to support R at the moment, though in my experience GPT4 is pretty good at translating code between languages

Interesting, can you say more about why you prefer Playwright’s selectors? Helpful to have some anecdotes from real experience!

We currently only allow log in with Google for logging in to autotab itself, but will add login with username/password and other options soon! Depending on feedback we might also build a bring your own Open AI API key option

Couldn’t agree more! Working on this actively. Interesting questions include 1) what format is most effective for the user to convey their intent (maybe not point & click) and 2) how to represent the model output such that it is auditable and editable.

Right now the generated script is the ground truth but we’ve been working on augmenting this with images & videos to fall back on. We think defaulting to code is good because it is faster, cheaper and more easy to reason about in the 95%+ of times it works. Plain old Selenium will get you pretty far, especially if creating scripts is much easier.

That’s exactly what we do - we sample relevant parts of the DOM and use the model to write the logic for selecting that element. This works pretty well and saves a lot of time that developers otherwise spend inspecting the html structure to write the selectors themselves.

Going forward we’re excited to experiment with more intelligence at runtime e.g. using AI to try to recover if the selector wasn’t found.

We use OpenAI's GPT4 to understand the position of the element being interacted with relative to the structure of the website overall. Lots of cool things we want to try out with open source models, but for now GPT4 is good enough.

The main difference is probably that autotab outputs a Python script, which is easier to integrate with existing Python codebases.

Going forward I think the flexibility of Python code & the Python ecosystem will make it easier to build a bunch more things we want to do. For example, a lot of browser-based work is not totally deterministic but rather requires more intelligence during the automation, e.g. summarizing information on a web page or dynamically filling in a text box with the right answer.

Good to know! The comments have definitely made us update towards looking into Playwright again quite seriously

This is very interesting, may I ask what macros you would define? Definitely something I'd like to learn more about

Hm, that unfortunately sounds like a chromium/Chromedriver error which can be somewhat finicky. I recommend check that the versions you're using match (`chromedriver --version` and at chrome://settings/help). Let me know how it goes and happy to help debug further (though GPT4 is probably more helpful than I will be)

Hey, we're looking to build out our library of authentication plugins. Many websites offer "Sign in with Google" which makes things easier, you can just specify that in the autotab.yaml file. If you have a specific website in mind, I'd be happy to add support for it!

Thanks! It's still very much early days for the Sidepanel and the API feels very much in flux but an exciting new form factor for browser-integrated experiences. Makes total sense, supporting other browsers is likely something we will do in the future

I used https://www.screen.studio/ for the demo and past demos, it works quite well I find though it can get a bit too excited on the zoom ins and I have to cut out some of them

autotab is not an always on browser extension like you are probably used to. autotab is only added to the Selenium-controlled browser window you use to write the automation while you're recording.

Also my experience differs more broadly, I use browser extensions quite a bit, e.g. for ad blocking and password management. Afaik Honey was basically a browser extension and was bought by Paypal for $4B.

Glad to hear it, let me know how it goes! And yes, other browsers are definitely in our future at some point

I'm more interested in making it useful than extracting value.

Longer term I think you'd monetize the infrastructure to run the automations at scale in the cloud.