A question comes to mind: What would be preserved if converting the visuals back into audio?
This may help answer another question: What type of visual would allow feeling the music without hearing?
This may help answer another question: What type of visual would allow feeling the music without hearing?
I think this intermediate representation that I've extracted using this algo would be much better for machine learning on the sound data than either 1) raw sound or 2) frequencies extracted with FFT. But there is an engineering difficulty to overcome: the frequency data that the algo extracts doubles in its amount (more frequent frequency samples) with each octave... It's like wavelet data... It's a challenge to feed this data to standard ML algorithms, need to think of how to configure the inputs, it would have to be highly hierarchical.
My initial goal was to "learn" different instruments from raw sound data, and this intermediate representation is good, because it allows "translation invariance" across frequencies. I've described it here: https://vsound.app/high-precision-in-frequency-domain.html
It's a work in progress... This app is just for me to see if people are generally interested in this area, I don't want to do something for months only to discover nobody wants it (although it's been a few months I've worked on this app LOL).
I might go further by letting people hear the difference between them (by transposing the visuals back into audio, to ground our sense of quality loss in the original medium, sound).
In both cases though it's possible to losslessly round-trip the audio within numerical precision (neither the STFT nor the Wavelet transform lose information).
If you are using a normal FFT this is no problem. You can reconstruct the original signal with a very small amount of error. However, this works because the FFT preserves phase data for each of the buckets. The spectrogram does not preserve phase data, so it will really mangle things.
(I’ve tried this, but it’s been a while. You get a sound which is recognizeable but total garbage otherwise.)
One way to get high frequency resolution from the STFT or Wavelet transform is to use the phase derivative within each frequency band, which is usually called "instantaneous frequency", and is closely-related to the phase-vocoder (different from the daft-punk-famous vocoder).
The main issue with this kind of instantaneous phase estimate is that it assumes there's only one dominant frequency within each band - does your method improve on that?
About the question of "assuming there is only one frequency within each band" -- yes, it's a problem, and that's what's causing the "frequency leakage" that you see in the app with the "high-precision: off" (and FFT). To solve this, I've added some extra math on top, which is available via "high-precision: on" in the app. I will release more details when I open source the algo. Please subscribe to email list if you want to receive an update, I am not super active on social media otherwise :).
I would say that the spectrogram is a visualization of the STFT magnitudes, in the same way that a line plot is a visualization of the FFT magnitudes. The STFT and FFT are both invertible though.