Apparently it’s fine according to this xkcd: https://youtu.be/EFRUL7vKdU8
The reactor vessel can contain heavy water or water with boric acid in it, neither of which are good for you. Additionally, neutron radiation from the reactor will cause the formation of radioactive tritium in the vessel water, so the liquid itself becomes radioactive.