My biggest mistake, BY FAR, was not visiting all those schools, talking with prospective advisors and current students, and using my gut instincts on those conversations to guide my decision.
I picked the school that I felt was the best but wound up poorly matched with an advisor. I had been a very proactive and driven undergrad (returned to undergrad in my late 20s) but reverted to a passive approach in grad school - it was a terrible mistake. By the end of my first year, I was completely demotivated and very unhappy. I turned it around my second year by dropping my advisor and taking some control in the process back. But I also totally punted on a PhD and settled for a MS. But it left me very unsatisfied in the process and in a position of asking "what if" in many ways.
One big part of why I was dissatisfied is that in hindsight, many of the professors I interacted with were simply not very friendly or considerate of students. Maybe they were used to being able to be more dismissive of undergrads coming straight to grad school, but as an adult I expected adult interactions. That experience was partly my fault - there are always people who are prickly to deal with. I should have been a little more self-directed about making sure I was working with better people even if it meant maybe doing thesis or research work in an area that was slightly less interesting to me personally.