It is worth doing, especially if you've never used another integrated Lisp environment, but I wouldn't recommend laying it on top of learning Clojure. I'd say learn Clojure first, then try out emacs and the emacs integration.
There are several heavyweights in the Clojure community that use VIM and have developed good plugins, and Rich never seems to be using SLIME during presentations, so you can definitely get by fine without it.
See this video for an overtuned (!) Emacs https://github.com/overtone/live-coding-emacs
The configuration can be found here: https://github.com/overtone/live-coding-emacs
At minimum, you need a easy way to paste top-level forms into a repl. This is the simplest way for Vim: http://www.vim.org/scripts/script.php?script_id=3023 That script is actually crazy useful & I now use it for a whole bunch of different use cases.
That said, the Emacs->Slime<->Swank<-Clojure bridge is really much better & think that I'm going to invest some more effort into becoming proficient at Emacs. My limited experiments with Slime were very positive & nothing in the Vim world even comes close. I tried all the various attempts in their current form as of two weeks ago. Most are hard to get running, don't work exactly right, struggle with subprocesses management, etc. And you're totally on your own with those, no warrantee and no maturity.
might be better
Pretty good clojure and coffeescript modes.
http://jaydonnell.com/blog/2011/10/07/setting-up-aquamacs-fo...
The steps are pretty much the same for regular emacs, you just need to put things in ~/.emacs.d/ instead of ~/Library/Preferences/Aquamacs\ Emacs/
The one that drives me really crazy is that on Lion, both Emacs freak out if you try to resize from from anywhere except the bottom right corner. Similar issues with the green (+) button; full screen mode, etc.
Also, since you'll probably remap cmd to meta, you lose some system shortcuts which can be convenient when switching back and forth between various apps and your brain can't keep all the shortcuts straight.
(global-set-key (kbd "A-j") 'find-file)