I don't believe it's not about "not caring". I think it's more about having a finite number of productive brain-cycles per day, and what you spend them on (opportunity cost).

If you spend all day thinking about others, you won't get any math done. If you spend all day thinking about math, you won't be in tune with others. If you can balance things out properly, you can do both.

Sometimes it's necessary to go to one extreme for a long period of time. If you're working on a really hard problem, it might be best to isolate yourself from society for days/weeks/months at a time to figure it out. If a close family member is stricken with a major illness like cancer, it might be best to focus and spend your thoughts, time, and emotions entirely with them.

In reality it's usually a balancing act between different things. But you can understand why it's sometimes necessary to go to extremes.