It is quite large, although not in a way that really aligns with the rhetoric of the anti-immigrant parties like SVP. They are mainly attacking "culturally foreign" immigrants who fail to assimilate and don't share Swiss values, mostly meaning Muslims, and secondarily meaning poor eastern Europeans. But the majority of immigrants to Switzerland are "culturally close" and assimilate fairly easily. The two biggest groups are: German immigrants to the German-speaking region of Switzerland, and Italian immigrants to the Italian-speaking region of Switzerland. Yet somehow SVP posters don't indicate that Bavarians and Milanese are the invaders...
There are some more numbers here: http://www.bfs.admin.ch/bfs/portal/en/index/themen/01/07/bla... 23% of Switzerland's population are not Swiss citizens, and Germany+Italy+Portugal+France account for about half of that.