>>the exception of the rare swimmer is much less nimble than a pedestrian or biker (and thus more predictable).

The problem is that boats are an absolute pain to turn and are the very opposite of "nimble" too. Yes if you see a swimmer in the water 20m ahead of you they probably won't move very far, but also to avoid them you should have started making a turn about a minute ago. Having done some canal boating in the UK it's just inevitable that you will keep colliding with things, and unfortunately you leave a lot of kayakers pissed off because it's almost always them having to take an evasive action.