That's because the recommendation systems are optimizing for the wrong things: typically dwell time. You are highly likely to dwell longer on scifi, so heres more scifi for you!

Older recommendation systems use background and foreground timeframes to build frames of references, hence repeats.

The question to ask then is: what is the correct thing to optimize for? Dwell time is usually chosen for advertising and stickiness (you are more likely to stick with a service that plays what you like). Optimizing for novelty is difficult because the set of unknown things is much larger than the set of known things. Plus, it is risky from a business point of view.