As a first measure I think we could make it so that once you removed yourself as a collaborator from a project, it would not be possible for the person to re-add you. I realize that's not ideal but would be much more trivial to implement than UI / additional server state for confirmation before being added.