I think a lot of it comes down to what a visual designer can contribute. FOSS projects are typically the sum of individual contributions from a lot of people, all scratching their itches. There is limited scope for someone to tell other people what they need to do.
A designer who can't code is dependent on other people donating their time so that the design can be realized. This puts the designer at an immediate disadvantage.