Aside from less overall movement while typing, Colemak keeps many common keyboard shortcuts the same. The first few hours are very frustrating, but the overall time-to-competence is short. There's a lot of upside and little downside.
I wrote about the process of learning Colemak in my book on skill acquisition, and posted a summary of the process and the tools/techniques I used on the book's website. [2]
[1] https://twitter.com/joshkaufman/status/1334632614368583680