This is being worked on. AFAIK, Apple is the first to incorporate this approach into a released product, with the Screen Recognition feature of VoiceOver starting in iOS 14.
Their effort seems currently limited to iOS based Phone screens. iOS is perhaps easier to solve given the strong Apple design guidelines for apps to pass the App Store review process.
Perhaps a community supported distributed approach to help build the database of annotated screens for the model to learn from, combined with open source models for all kinds of screens and applications( not just Apple) would be interesting project to work on.
[1] https://machinelearning.apple.com/research/creating-accessib...