I don't think you can fully claim it is irrelevant. Especially not and point out that it can make things a bit more efficient.
I think it is safe to say that the most important part is to be consistent between what you train on and what you infer against. However, it could be that one of the color spaces is more amenable to training. To my knowledge, that has not been completely ruled out. Indeed, it seems supported by your claim that they can get a tiny bit more efficient with some spaces.
That said, this post is my long winded way of asking if you have links on that. :)