You'd be surprised.
One of the best things about books is that it triggers the imagination, every reader imagines a slightly different version of the characters and settings.
Once a visual portrayal exists, it's quite difficult to not 'see' that character in your imagination as that version, be it Daniel Radcliffe, Ian McKellan, Sean Astin, or whoever.
Not that it "destroys" it as such, but it does affect/influence it.