We use OpenAI's GPT4 to understand the position of the element being interacted with relative to the structure of the website overall. Lots of cool things we want to try out with open source models, but for now GPT4 is good enough.