My impression is that it's the kind of claim that's actually true in general, despite being false in all the particulars.
Like, every step of the way, yes, virtual DOM adds extra steps to the process, and they have a cost. But, in the big picture, if you're manually manipulating the DOM then you're liable to just swap out entire chunks of the tree rather than making surgical manipulations. Because manually managing all those surgical manipulations would quickly become a fiddly, unmaintainable mess. But, if you do it that way, you're asking the browser to do a lot more re-rendering, and that would be costly.
I worry that this conversation falls into the same trap that most performance discussions fall into, where we quickly become focused on what things would be like if people had unlimited time to chase the best-case scenario, rather than what things are like for the 99% of us who are just trying to get something that works acceptably by the next deadline.