In addition to what you say here, most of the ATC job is about maintaining order and safety, while for a pilot there are many other factors (and accidents happen when pilots get distracted by those other factors). The ATC is constantly thinking about possible accidents, while the pilot has more opportunity to have their thoughts elsewhere.