It's been considered, for a variety of heat sources. I'm long-term bullish on a related technology (thermophotovoltaics), although presently the efficiencies are not great & the compatibility with nuclear radiation is probably poor.
If we could combine heating, refrigeration, with a configurable output of either end on top of it being a generator of voltage… is quite amazing.
The major issues I can see are (1) achieving sufficient efficiency at a reasonable cost, by recycling the low-energy photons instead of absorbing them as heat (2) avoiding radiation damage to this sophisticated device - maybe this isn't too hard with a liquid salt heat exchanger?