Libertarians don't agree that the inability to protect the environment makes their philosophy invalid either. The fact is though, most people look for reciprocatity in their moral philosophy.
It's not that every individual has to agree not to kill you. It's that we have to agree that punishment is appropriate for killers. Getting specific - it's incoherent to say it's immoral for humans to kill rabbits but moral for foxes to kill rabbits. To have a consistent moral philosophy, you have to either punish the foxes and try to prevent them from killing the rabbits, or you have to admit that killing rabbits isn't _really_ a problem.