Don't do it for speed. I type exactly as fast in colemak as in qwerty, even after years of using colemak. (110 WPM). The limitation feels to me to be in my brain, not in the keyboard layout. Colemak does feel much more comfortable than qwerty. Although I stopped using qwerty for a month while I initially learned colemak, I have been able to use qwerty or colemak interchangeably since. It takes no effort to switch 'modes'.
The biggest thing I hate about colemak is that it breaks vim keybindings. I actually use this script http://www.vim.org/scripts/script.php?script_id=2865 to have qwerty key mappings in vim while having colemak in insert mode
If you're interested in learning a new keyboard layout, I recommend checking out these links before you decide
http://colemak.com/wiki/index.php?title=Easy_to_learn
http://colemak.com/wiki/index.php?title=Ergonomic
Below is from the colemak FAQ on dvorak
http://colemak.com/wiki/index.php?title=FAQ#What.27s_wrong_w...
What's wrong with the Dvorak layout?
The main problem with Dvorak is that it's too difficult and frustrating to learn for existing QWERTY typists because it's so different from QWERTY. Colemak has been designed to be easy to learn.
Placing 'L' on the QWERTY 'P' position causes excessive strain on the right pinky. Colemak doesn't place frequent letters where the pinkies stretch.
'F' is on the QWERTY 'Y' position which is a difficult stretch on normal keyboards.
'I' is very frequent but isn't on the home position.
'R' is very frequent but isn't on the home row.
It is significantly lopsided so that the right hand does too much work.
It's not comfortable to use Ctrl-Z/X/C/V shortcuts with the left hand while holding the mouse with the right hand. Colemak conserves those shortcuts in their QWERTY positions.
Even though the design principles are sound, the implementation isn't optimal because it was designed without the aid of computers.
'L' and 'S' form a frequent same-finger digraph on the right pinky. Same-finger for the pinky is very rare in Colemak. In particular, Unix commands such as 'ls -l' are very uncomfortable to type.
Some punctuation (in particular the curly/square brackets) is less comfortable to type on Dvorak. This affects mainly programmers and advanced Unix users.