Hacker News Reader

Comment by tbrownaw | Hacker News Reader

Hm, makes me think of http://blog.objectmentor.com/articles/2008/06/22/observation... .

It sounds... scary. Like it will work well enough at first, and then explode when someone changes their desktop theme (especially icon theme), or wants to upgrade to a new version of whatever. Treating things as change-controlled APIs when they aren't just seems dangerous. Still I guess there's at least some amount of change control coming from platform conventions and human interface guidelines, and this comes closer to operating at the correct level of abstraction to benefit from that.

View on HN

Yes,

One problem with GUIs from the start has been that automating their behavior is fragile.

Automating actual keystrokes and mouse movements is the most fragile and kludgy of automations. Even drilling to the level of system messages is problematic.

Aside from it's other good qualities, the web is really nice for reducing possible user interactions to a very codified set of operations.

The article here points to ... the wrong way of doing things

I wondered about that, too---how fragile will this be in the face of minor GUI changes? (A fuzzy-match parameter would be a nice addition to the API.) One encouraging sign is that the screenshots are stored as .pngs in the file system. If an app update does nothing but change button graphics or text, the script could presumably pull new graphics from the same place.

That just leaves the other 90% of GUI changes that involve moving a setting to another screen, or changing the way an entire interaction works. ;-)

All the stuff you said about GUI automation being klunky is true. Still, this thing could come in handy as a kind of "GUI batch file," or perhaps as a tool to help produce screencasts.

Also, GUI applications are generally stateful, so while you might generally get the same behavior from, say navigating to an options dialog, you're not guaranteed the same behavior - especially since the creator hasn't made any effort to get the same behavior from the same actions. (The web can the problem of statefulness but at least on the web you're supposed to avoid it).

The do say that it does fuzzy matching, so it can still work with minor changes.

I don't think any program guarantees that the GUI is a stable API! I do wonder about its ability to handle unexpected conditions, or for the scripter to know that these conditions exist in the first case. For example, if the GUI changes based on the day of week, the script will probably either get very confused, or do the wrong thing. I imagine there's some way to do conditionals (if you see this, then do thing 1, otherwise do thing 2) but it's hard for the scripter to discover all the possibilities. Still, might be useful for some things.

Reply on news.ycombinator.com