Productivity gains don't only come from technological progress. Accumulation of capital, such as infrastructure, education, access to healthcare etc, also increase productivity.
Capital is not simply "anything that I can tie to improving my work output".
We could then go a step or two forward and posit that a sick populace means a sick consumer class means reduced demand for goods that generate growth, but those are just details.