I sometimes binge on cheese and nuts or peanut butter when I'm not at my best, but even when I do this for weeks, and I'm talking an extra 2000 calories in a day which is easy with those things, I haven't gained weight (fat). I don't feel as good and I don't recommend it, but I think it's safer than eating sugar if one is going to binge. I had a couple years 10 years ago where I ate sweet things most days and did gain noticeable weight (fat). So it's not like I'm genetically not going to gain weight either.
As an aside, physical activity might not affect weight change as much but it will affect the fat/muscle ratio.