I've seen similar with movement. Once they break a hip or something which severely reduces their physical activity level, cognitive abilities usually go downhill fast. Part of that is probably cardio-related, but I've long thought it was related to lack of stimulus.
Just doing the dishes or feeling the wind creates a ton of sensations, and may trigger memories, that sitting in a chair watching daytime TV just won't.