You're right, it's impossible to out-exercise a bad diet, but there are secondary reasons that being sedentary leads to weight gain besides just the raw number of calories burned.

For one, an hour you spend exercising or walking around is probably an hour that you're not eating. Many Americans are constantly eating when they're at home, either mindlessly snacking or eating a huge meal. Given that you only have 8 hours between work and sleep, occupying yourself for 1 of those hours could mean a significant decrease in time spent eating.

Incidentally, this is why intermittent fasting works well for some people.