How was their sleep? I have sleep difficulty (including a non-24 hour circadian rhythm) and significant early memory issues. I also have a family history of dementia (I only know anything about the sleep of one family member with dementia, who did have significant sleep issues). I did a quick search and found there are some signs of a potential connection and this seems to be under more intense research at this point.
Also, manual labor isn't the same as cardiovasular exercise and it is possible (and likely much easier these days with all of the powered machinery and transport) to not get much cardiovascular exercise despite a physically intensive job. Although I'm guessing even today a logger would.