Not exactly. The initial collisions do indeed cause localised damage cascades, but the overall effect of a large neutron flux is embrittlement of the material. This is a problem already in fission reactor vessels, but the neutron fluxes are a whole order of magnitude larger in fusion reactors, and it's a very difficult problem to solve. To my knowledge, current materials can only just about deal with with the neutron fluxes in standard fission reactors, with no current materials being capable of withstanding neutron fluxes in a fusion reactor over any long (i.e. commercial) timescale at present. It would be a big achievement and a big jump in materials science to discover such a material, completely separate of any fusion project.
(Disclaimer: this is not remotely my field, but I have looked at some of this stuff in the past, and been to talks about ITER.)