If solar time does not matter, why do you care about at when number you get up, go to work, come back and get back to sleep? This would be even more ridiculous. People care about the time, because it's a proxy for sun.
Most people start and endworking at common time. If you want to spend time with other people for hobbies or socializing, you need to find a matching spot when people are done working.
Sure we can stick to solar time and adjust office hours instead. But then we're back at square 1...
But I'm pretty sure perma-non-DST people would not be happy with such solution.