Outside church settings, classical music of the Baroque, Classical and early Romantic eras was for the elites, and the masses had their own musics. This only changed gradually over the 19th century.
It's also weird to still be hung up on atonality in the contemporary classical world, when the pop-music world has for decades now had acts like Merzbow that are often farther out into noise.