Looking around a bit, I found an interesting paper on the exact topic I was thinking about[1]. It turns out that the exact difference does affect color recognition, but the effect is "eliminated by a verbal, but not a spacial, dual task".
[1]: http://www.pnas.org/content/104/19/7780.full
Of course, I think this is really the same effect as in the original article. The only reason it's important for me is that I experience it directly by knowing both languages.