Add a small colour wheel to the main playing area (maybe only for the first few challenges per game?). It's on the pause screen already, but I know when things got hectic I'd have liked it to be there in front of me.
The single colour challenges were pretty punishing, in that there really weren't enough bars coming of that colour to push through to the next challenge. Perhaps give a small amount of time for non-challenge related matches?