If you want high thrust you just superheat reaction mass and expel it. Hydrogen gives you the best efficiency I think but is hard to handle. Water, carbon dioxide, and hydrocarbons are easy to handle. The nuclear engine can sustain much higher temperatures longer than a chemical engine to achieve both high thrust and higher specific impulse. You get a lot more delta V per unit rocket mass than just chemicals.

If you want maximum specific impulse (propellant efficiency) then you want what you are thinking of: a reactor powering ion engines. Ion engines can use very little propellant but generate low sustained thrust for a long time. Solar can power ion engines too, but I think a reactor can do so with less mass and solar becomes increasingly useless much beyond the orbit of Mars. If you want to visit Europa or Titan you need nuclear energy… or it’s a very very long trip.

Hybrids are possible too like a reactor that can run in either or both modes.

They feed the fuel into the reactor to superheat it and then it comes out the back like a normal rocket, but then doesn't need oxygen for the reaction to take place.

I think they intend to use Hall Effect thrusters