As a consumer, you often see guilt or other attachment with car and home purchases. I'm definitely guilty at work. I really enjoyed the work at some jobs that were objectively awful. The customers, mission or colleagues were great in some way, but the company gleefully exploited that emotional attachment.
In my case, I attribute that inappropriate attachment to work to my own failure to deal with conflict or emotional issues in my personal life. It's really the only regret that I have in life, as it's a problem caused by my own stubbornness. Time is the enemy -- age, death and other events make it impossible to take back or fix. Whenever you find yourself thinking "we can do <X> next time, I have this <important project/task> to do", stop. The kids will grow up, your partner will get sick, your parents will die. Someday there won't be a future, and you'll find that you don't care about or remember the "important" things you did, but you will regret that things you didn't