The neutrons coming from the fusion reactor do have the potential to make the surrounding material radioactive. But that depends on exactly what the surrounding material is, some elements will become dangerous when exposed to neutrons and some won't. An important part of designing a fusion powerplant, and one of the reasons it is difficult, is that you have to make sure that the materials you use can safely handle the neutrons coming from the fusion reaction without transmuting into dangerously radioactive forms.
This is in contrast to a fission reactor where the fuel itself turns into dangerously radioactive elements when exposed to neutrons.
EDIT: The exception is that fusion powerplant designers will want to surround the reactor with lithium in the hopes that it will absorb a neutron and turn into tritium. The tritium is then carefully gathered because it forms the the fuel for the reactor and it's hard to get except in a nuclear reactor.