This same phenomenon is seen not just with work/leisure (with time being the limiting factor), but also with food/leisure (with money being the limiting factor). Studies show that people who are malnourished due to living in poverty, still spend money on leisure activities despite not having enough food to eat. Leisure time is important for people.

Even if you enjoy your work, it's still work, and you're still going to need leisure time. The cause is probably different from person to person, but the solution isn't going to be just "work less", but prioritizing, scheduling, and getting the most out of your "leisure time", which also means different things for different people.