A solution using xargs instead of exec:

find . -name "*.swp" | xargs rm

I believe find -exec forks and execs once for every file found, xargs may only fork and exec once, provided that all of the files find finds fit in a single command line. Hopefully not an issue in this case, but it can greatly speed up a deletion when you have a lot of files to find and delete.

[EDIT] Doesn't properly handle spaces in filenames, see the comments below for other slick solutions.