>In October the finance ministry, in its annual report on the issue, estimated that in 2018 immigrants from non-Western countries and their descendants drained from public finances a net 31bn kroner ($4.9bn), some 1.4% of GDP. Immigrants from Western countries, by contrast, contributed a net 7bn kroner (see chart).

https://archive.ph/TSsQa#selection-1021.77-1025.92

So yes, it has downsides... The downside of not working at all. (In this comment, I am separating Western and non-Western immigrants, since you mentioned xenophobes)