Slightly OT question - can I use GTP-4 vision to drive a web browser, e.g. to automate tasks like "sign up for this website using this email and password; don't subscribe to promotional emails"?
I believe tying it together might be a challenge. For example, if you were to use the model to get the text of buttons, you would still have to write code to find the HTML elements for those buttons and drive the click/fill actions.
I've used GPT-4 to help with selenium and it gets the answer eventually, but almost never on the first try. So automating this without human intervention sounds tricky.