What matters oftentimes, is that other finance flows are affected. So, the loss of the notional value has a knock-on effect. Down the bottom of the stack somebody loses a job. Somebody has a payrise denied. Somebody has a higher insurance cost or a rise in mortgage cost, because <reasons> which ultimately relate to the "tail wagging the dog" element of imputed value, and knock-on effects.
Does the wealth "go anywhere" -No.. But the lack of the wealth does cause things to happen, and some of them actually make money move between people to compensate for the loss of the wealth.
When the wealth is created from two credit unions double trading CFDs between themselves to make value appear, and then the music stops and the last guy standing with no chair has the problem, its pretty bad. People get hurt.