Do you mean the numbers? I was going to add that feature, will add it in another release to turn this off/on. Sorry, I’ve been working on this for a few months, so I decided to release what I have now, and then get on with improvements... Thanks for suggestion, now I know people want it.
Who wants to get NOTIFIED about the DESKTOP app, so you can take screen recordings to make videos with your music, please SUBSCRIBE to email list on the app page! I will make the same version as on iOS available as DESKTOP apps for free as well. (Plus the visuals look much better if you feed sound directly from the sound interface, not microphone.)
I haven’t published the details of the algorithm. The science bit is supposed to give some idea, but not details. I know people want the details straight away, but I don’t work in academia, so my goal is not to make a publication, but to gauge interest in applications of this.
There are different ways to describe it in one sentence :) You can say it’s just a wavelet transform. Or that it’s just vocoder. There are many internal implementation details actually. I haven’t published details of the algorithm (yet).
That would be hard to target different phones... I am just using SSE/AVX (via Apple's Accelerate framework) for now for some parallelisation, helps to speed up the detection x10.
The core algo is basically like a vocoder (I think), tracking certain frequencies, and as others pointed out might be very very similar to https://en.m.wikipedia.org/wiki/Constant-Q_transform (I guess another question is how to implement this effectively in "real-time").
About the question of "assuming there is only one frequency within each band" -- yes, it's a problem, and that's what's causing the "frequency leakage" that you see in the app with the "high-precision: off" (and FFT). To solve this, I've added some extra math on top, which is available via "high-precision: on" in the app. I will release more details when I open source the algo. Please subscribe to email list if you want to receive an update, I am not super active on social media otherwise :).
No I haven't seen this, it's very related. I guess if you wanted to use this for real time, you would still need to solve the question of window size...
It’s equal temperament scale (each note’s frequency is 2^1/12 above the lower one) with 12 notes per octave, have a look at user guide for more details. Many people asked for open sourced version of the algo. I want to do it. Just need to think a little about how to do it properly, so that I can still build something on tops of it to “monetise” what I’ve been doing for a year now... (Everyone wants to do smth enjoyable in their life.) I’ll try to do it soon! Subscribe to email list, I’ll let everyone know.
Thanks so much for looking! You’re right, all the visualisations apart from the first one arrange the notes in octaves (12 notes). I’m starting to work on better 2D and 3D visualisations. Unfortunately Apple recently deprecated OpenGL support, so I’m trying something that’s only supported by Apple now (Metal), which means it would have to be for Apple only. I’m keen to make some 3D visualisations... I’m imagining something like a 3D environment, where you could “walk” and watch things react to music like tree leaves and waterfalls! :) The app is basically just “better” at detecting notes (frequencies) than FFT, it’s basically doing an online wavelet transform. Thanks for your feedback!
I've done this initial app just to gauge if there is any interest in this at all (friends were telling me "no", but they don't know anything about music :)). I am thinking that I could do some visualisations for free, and add the fancy ones as a paid option. There are many visualisation apps out there, but none have the same frequency detection algorithm... Many people asking to open source it now, but I don't know how can I do it, and also get paid for making something people can pay for on top of it... Just trying to make something that would allow me to leave my (already high paying!) job... I'll need to think :)
I guess you're missing the detail of how it's actually implemented in practice. The short description would be just "wavelet transform" I guess. The details is in how to implement it efficiently so that it can run on the phone in real-time, and draw the visualisation at the same time, without drowning the CPU (plus the additional feature of high-precision detection, which is not solved by wavelet transform itself). I will describe it in more details when I'll be open sourcing the algorithm (after I've implemented something on top of it that I could potentially start selling).
Yes, soon! Please subscribe to my email list on the app page. I will try to post on HN again, but I am not sure it'll reach the front page again :) P.S. Please upvote.
It's ok, they didn't ask rudely :) People don't be angry, give me more upvotes instead, so the link can stay on front page longer and other people can see it! :)
Thanks! I'm also planning to release the desktop app (currently for Apple only first), it would be quite easy to make actually, since I have decided not to charge for it anything :) I've actually made videos that are on the front page by doing screen capture on my laptop! I just need to figure out how to sign the desktop app properly, so it can be distributed for Apple... I might be even able to do a Windows app, since I am using JUCE framework. If you want, feel free to subscribe to my email list (link on the app website), so that I can send updates. Sorry, didn't make twitter account for this :)
I am sorry, but I am using JUCE framework [1] which has some C++ features (aligned allocation) that are only supported from iOS 11. I'm constrained by the framework, I've set the minimum iOS version it supports. Sorry again!
There is a trick to do this :) If you think about it, you don't need high-resolution information about low frequencies, they don't change that fast. The lower the frequency, the fewer data points (complex) you need to reconstruct the original sound, because the lower frequencies don't change as fast throughout time. You wouldn't be able to do this with FFT, which I have started out with, but was unsuccessful. So I had to "invent" the new algorithm, which I've figured out later is equivalent to Wavelet Transform. The only new thing I've done is to make it "real-time", and I have some extra math for "high-precision" frequency detection, which deals with the problem of frequency leakage post-factum (I am willing to reveal this has something to do with rotating complex numbers :)).
Haven't seen that video, thanks! I am a bit of a home DJ thought, in addition to my full time work in machine learning :) I've thought about DJ applications as well. It all comes back to the original goal of the project to do machine learning on music. One of the potential applications I've thought about is a new interface for mixing! I've imagined an app where you would "grab" different parts of songs/tracks and "drop" them into the mix, and remove some other parts. The ML algo would need to break down each track into its composable parts, and make them available for mixing with parts from different tracks. Possibilities are endless! :) That would require a lot of work (my dream job) :)
As a matter of fact, I've converted the frequency representation that the core algo extracts back to audio without noticeable quality loss. Although in the app I am just showing the amplitude, but the phases are also important (people usually say that the phase of the note is not important for perception, but actually in complex sounds it's important to preserve relative phases of different frequencies).
I think this intermediate representation that I've extracted using this algo would be much better for machine learning on the sound data than either 1) raw sound or 2) frequencies extracted with FFT. But there is an engineering difficulty to overcome: the frequency data that the algo extracts doubles in its amount (more frequent frequency samples) with each octave... It's like wavelet data... It's a challenge to feed this data to standard ML algorithms, need to think of how to configure the inputs, it would have to be highly hierarchical.
My initial goal was to "learn" different instruments from raw sound data, and this intermediate representation is good, because it allows "translation invariance" across frequencies. I've described it here: https://vsound.app/high-precision-in-frequency-domain.html
It's a work in progress... This app is just for me to see if people are generally interested in this area, I don't want to do something for months only to discover nobody wants it (although it's been a few months I've worked on this app LOL).
thank you! I can clearly see the interest among people now! I will think what's the best strategy. Definitely be releasing the core algo open source, need to prepare, figure out what's the best way to do it. Thanks again!