This has been studied before and it seems that yes your brain has trouble ignoring music with lyrics.

Based on this study[1] you lose at least 10 IQ points when studying and listening to music with lyrics.

[1] https://sites.psu.edu/siowfa15/2015/12/04/listening-to-class...