Cadmium-telluride and CIGS cells use more rare elements, but they have been overtaken by the ongoing fall in cost of conventional silicon wafer cells, so you won't see too many of them being installed at the moment.

(The failure of CdTe is probably for the best given how spectacularly poisonous both of those metals are)

Despite their name rare-earth elements are not that rare either. They occur in rather low concentrations though and never pure either (i.e. always as compounds) which makes mining and processing them complex.

https://en.wikipedia.org/wiki/Rare-earth_element

It's just the price is set by people who can mine it for nearly free from the most rich deposits in the world.