Parsing text (eg html) is a bit easier than parsing images, although maybe the difference is going to zero
Fluent Search is a desktop app and this feature is meant to work regardless of which apps are on the screen (browser, Notepad, whatever else) so I think there's a legitimate point here although it might be a niche one not relevant to you.
(I don't know enough about Windows internals to answer with confidence if a better solution is available like examining the OS accessibility tree, but I guess an advantage of Fluent Search's approach is that UIs from toolkits not implementing accessibility standards aren't excluded. The current approach has drawbacks like false positives too though because the ML model isn't 100% accurate.)
- Security testing - Interacting with a device that may have malware on it. You use a robot so that it is 100% unmodified while you study it. For further safety, the device is run inside a Faraday box, and you use the robot to access the device from outside. (Otherwise, you do everything manually in a human-sized Faraday cage.)
- Performance testing - Getting timing data of app performance is more accurate when the device is unmodified and you use an external tool to interact with the UI.
- Testing medical devices - Required to be tested exactly the way a user would interact with the device.
- Point of Sale testing - Some real world end-to-end tests involves inserting or tapping a card and entering a PIN on the screen.
- Or you're interacting with a touchscreen that has no other programmable interface, but you still want to test automate it (like any touchscreen on any device that is not a phone or tablet, e.g. an HVAC or alarm system panel.)
(These are all things I've been paid to make robots for over the last decade.)