If you find you have trouble remembering two layouts then I would suggest using a specialized ergonomic keyboard with a non-QWERTY layout for most of your typing and leave all the normal slab keyboards and laptops as QWERTY. For much the same reasons you don't get confused between a piano and a computer keyboard, using a keyboard which is physically significantly different helps separate the two layouts in your muscle memory.

For example, I use Colemak on a Kinesis Advantage and QWERTY on all flat keyboards. Since the physicality of they keyboards are so different I don't have any confusion while typing, though sometimes I'll mix up single character presses. That is, if I type a sentence I never have troubles, but if I want to hit just the 't' key I may mistype the first time. Other than that I can seamlessly switch between the two and my speeds are comparable across both layouts.