> For the 1st point, I generate a script with hashed check points so next run is automated unless something changes in the UI to invoke AI. I make this possible by proxy wrapping playwright library so I can take over every method. Users use playwright like they always have but with one extra method called act.
How would you determine that something changed in UI by just looking at a screenshot? Would you additionally compare HTML/DOM or approximate the two screenshots?
> Omini parser lets you split section of the UI to hash and watch for changes that are relevant.
I wasn't aware, thanks for sharing!
> For 2, can you give some examples?
Specifically, if you take the Shortest tool (https://shortest.com), a test runner powered by Computer Use API, write a test "Validate the task can be pinned" for https://todomvc.com/examples/vue/dist/, and run it — it passes. It should have failed because there is no way to "pin" tasks in the app, yet it pretends that completing the task is the same as pinning.