In my opinion the mistake was to move away from the terminal in the first place. Many gui commands like "draw button in bottom right corner" could be done via escape codes and creating a gui program would be as simple as writing a bunch of commands to stdout. As far as I know the earliest prototypes of X worked just like that.