We (Google Research) have a similar work that we recently published: https://blog.research.google/2024/03/screenai-visual-languag...