This is what Ubuntu tried to achieve with Unity. Even as far as being able to search for GUI commands within individual applications. I thought it worked pretty well and was gutted when they scraped it.
I never tried it, seems like such a wasted opportunity.