I think they're smart enough to differentiate the ones (us) that would cause too many problems to eat.

In our area (Discovery Islands), there is no lack of humans swimming, or in kayaks, or diving. The transients are common here, far more than the residents. Of course, so are seals and sea lions.