this is really well summarized.
I'm in the same camp as you, I've got way more in my setup than the average person, but its mid-fi at best, and mostly on the headphone side. I like music, and I have headphones on for 5-8 hours a day, I'd like to optimize that experience. The good news is the average pair of headphones today is SO so much better than it used to be, and in a weird way some of the overpriced stuff from Apple/Sony/etc... is actually getting better sound to people that hadn't experienced it before.
I really wish there was more emphasis on the transparent + objective measurement side of things industry wide. There is in some places, but at the end of the day entire companies are built around not having objectively perfect sound reproduction, because there is a following of people that like the house-sound they engineer in to their speaker/headphones. And there is value there, its just hard to quantify.
About 12 years ago I built a pair of large (relatively) full-range bookshelf speakers for near-field studio monitor type use at my desk with a wild 3 chamber porting setup. No traditional physical filter-network whatsoever, I borrowed a decent mic, measured the response, generated an inverse impulse filter that I could use to preprocess the signal going to the amp to arrive at a "perfectly" flat frequency response. Guess what? it sounded ok at best.
Audio is hard. Your ears are weird.