I was looking for something similar recently and had found CogAgent[0] that looks quite interesting, has anyone tried anything similar?
0. https://github.com/THUDM/CogVLM?tab=readme-ov-file#gui-agent...
0. https://github.com/THUDM/CogVLM?tab=readme-ov-file#gui-agent...