Off the top of my head:

Ever tried removing a submodule? Why is there no git submodule rm? (No, "it's confusing" is not a valid argument.)

If the pull fails (say, from the wrong protocol), you end up with a half-baked submodule. Impossible to move, impossible to remove, impossible to update unless you dive into the config.

Misspell the folder you wanted the submodule in? That sucks. Go delete it from two separate locations, and try again!