Resolving the race robustly is actually kind of hard, in Bash... You can check whether `rm` returns a nonzero exit status, but that might fail for all sorts of other reasons (e.g., bad permissions). I guess you could case/select the exit status numerically, but that could turn into a real case of bedbugs, really quick.
I guess you could repeat the `stat` check afterwards, if the ~`rm` fails? In that case, if the path doesn't exist afterwards, you can just swallow the `rm` error and let it roll.
Practically speaking, I try to avoid using Bash for anything where automatic, reliable error handling is that important... I love cool Bash tricks, but it's just not designed for sophisticated control flow.