When synthesizing audio from the short-time Fourier transform (STFT) sometimes you have unknown or noisy phase, and there’s an algorithm called Griffey-Lim that’s pretty common for finding the corresponding time-domain signal. You start in the STFT-domain and continually swap between time and STFT domains, each time fixing the STFT magnitudes. Eventually the phase tends to converge (but not sure if that’s guaranteed).
Maybe there’s something similar here where you keep swapping back and forth while applying the blue spectrum and uniform sample histogram constraints (or partially applying them in a gradient-descent fashion).
(Side note that there are other/better phase estimation algos for this problem, but Griffin-Lim is simple and relatively common)