Nevertheless, if you block all the ~850nm IR light from reaching the lens, then there cannot be any retro reflection at this wavelength, meaning the ToF cannot see anything.
Yes, I agree. I believe military-grade optics detection systems use multiple wavelengths, and some even use short bursts of visible light, making it harder to block.