Selenium project creator here...
Very cool! I totally expect testing to be one of the killer apps for AI. And this is an easy way to get there.
How do you ensure the AI isn't hallucinating any details in the generated code?
Very cool! I totally expect testing to be one of the killer apps for AI. And this is an easy way to get there.
How do you ensure the AI isn't hallucinating any details in the generated code?
Right now we have the basic sanity check that the Selenium xpath selects exactly one element in the DOM. Going forward I think the best way to do this is to have a live preview where you can see the mirrored browser copy your last action and debug if it errors or isn't what you wanted.
That is a potential problem Because at this point the user will still need good HTML/Selenium/coding knowledge to debug. We ran into the same issue using chatgpt generated scripts for ui.vision. Once a QA person has to figure out why some generated code does not work, it becomes a hassle and removes any initial advantage over the classic record & replay approach.
But there's something in between where you can say "try again" or even give high level feedback like "don't click the element with that exact title, just click whichever one is first in the list". My working hypothesis is that these are bigger than the ones where you need coding knowledge. Certainly this category will only grow as language models become better.