If you are using a normal FFT this is no problem. You can reconstruct the original signal with a very small amount of error. However, this works because the FFT preserves phase data for each of the buckets. The spectrogram does not preserve phase data, so it will really mangle things.
(I’ve tried this, but it’s been a while. You get a sound which is recognizeable but total garbage otherwise.)