The Spotify use of the coreaudio API has been massively broken for years.
Basically they are preventing hardware decoding (for security reasons I guess?),
and drive the API in a way that decodes everything 5-8 times.
Not a lot you can do, apart from switching to a Spotify alternative, or at least a different client.