> although the shields were fudged somewhat to allow knife fights
Right. Shields can protect against high velocity things, e.g. bullets or shrapnel. But not slow moving things, e.g. air and hands.
This contrivance means ancient weapons still have a purpose.
This effect is similar to non-Newtonian fluids. (In fact, there has been research into using non-Newtonian fluids for body armor [1].)
---
> It would also seem that mankind lost the ancient secrets of body armour.
There are a few instances of armor in the movies (Harkonnens, Leo Atreides and Gurney Halleck, one of Paul Atreides' visions), but the books make virtually no mention of it.
The in-universe explanation is that the fighting style was much closer to ninjas than knights. Which means light-to-no armor was preferred.