Audio processing in TensorFlow
medium.com
medium.com
The landing page is aimed at musicians but if you dig down there's extensive documentation on how to do low-level DSP processing. https://www.native-instruments.com/en/products/komplete/synt... and https://www.native-instruments.com/fileadmin/ni_media/downlo...
Yes, there's a learning curve to flow-based programming. But this is what all IDEs will eventually look like. You shouldn't be writing everything in code for the same reasons you shouldn't be writing your whole project in assembler: your reinvention of the wheel is probably not as great as you think; every time you forsake a standard modular component in favor of your own way of doing it, you're creating technical debt for whoever has parse your code later; A lot of the actual work in coding is syntax, glue, and scope checking, and those jobs are better done by a computer - making people type out all that stuff by hand is a distraction from the domain-specific problem to be solved.
Music software may seem like a toy to non-musicians/sound engineers but for them it's mission-critical real-time software with an extremely demanding user base that frequently has advanced domain knowledge of its own. Tools like Reaktor are built on the foundations of earlier tools like pd and supercollider, which in turn were built on text-based languages like CSound. People have been building neural nets and suchlike on these platforms for over a decade already.
I think this is worth exploring. But my priors gives this a low probability. Smart and capable folk have tried this. Many many times.
Audio signal processing software generates a stream (or set of streams) of samples which should eventually cause speaker cones to vibrate. General-purpose programming isn't like this at all - the output could be projected on a screen, or it could move a servo, or it could be passed along a wire to another machine. The program could run straight through, or react to inputs. It could stop, or run forever. Even though working with audio (or the layers of a deep learning network) can be intuitive with a visual analogy, that analogy is not going to work as well for other kinds of programming. This is a trap that people have fallen into over and over again, trying to extend a graphical interface designed for a domain-specific purpose to all of programming.
I have a theory which s̶h̶o̶u̶l̶d̶ ̶m̶a̶k̶e̶ ̶s̶e̶n̶s̶e̶ you might like as an audio person: CPU instructions are processed in a stream, just like audio samples, and the various control structures and loops in that stream are a bit like the frequencies in the audio signal. So you can think of the "true" representation of a computer program as a higher-dimensional or infinite-dimensional vector in some kind of abstract program space. Any method used to display this vector on a two-dimensional screen will make some features hard to recognize and work with. The best interface for general-purpose use has to be another stream of code.
> A lot of the actual work in coding is syntax, glue, and scope checking
The vast majority of practical computer work is breaking down and understanding the problem you have to solve. With modern tools, actually writing useful code takes very little time. It would take much longer if you had to design a visual abstraction for your problem domain first.
Ignoring evidence in front of your face and inaccurately characterising the nature of the tasks performed is misleading yourself.
Also, I don't only do audio, that's just where I made my living for much of the last decade. It's because I had prior experience in other technological contexts that I'm telling you that flow-based programming is the best future paradigm.
> Ignoring evidence in front of your face
Do you have an example of a complex UI built entirely with Reaktor?
By the way it looks incredible, I've owned an old version for years but never tried it. Is it really that much better than SuperCollider once you get into experimental stuff? I'm really surprised to hear that it goes all-in on the interface and doesn't have any "write code mode" unlike like most graphical programming environments. Users are asking for that, of course:
https://www.native-instruments.com/forum/threads/my-requests...
I'm certainly not against any code, it will always be important (I like assembler too believe it or not). But I think it's really important for IDEs to develop good structural tools. I think a lot of the resistance to flow-based programming is from IT people who kinda want to stay in their department, just like there was resistance to software from ASIC programmers, resistance to high level languages from low-level programming experts and so on. People are always wary of a new level of abstraction, partly because it involves a loss of control I think.
And I agree with all of your second paragraph. I'm probably still going to be skeptical of any "general purpose" flow-based IDE, but who knows, as programming matures over the next century we might eventually have a good grasp on the similarities between domains, and how to design better and/or visual tools that work in many domains. My theory could be totally wrong too.
> every time you forsake a standard modular component in favor of your own way of doing it, you're creating technical debt for whoever has parse your code later
I don't disagree that you shouldn't re-invent the wheel whenever possible, but modularity is absolutely not unique to flow-based programming. Well designed libraries should provide good levels of abstraction and be highly modular. Flow-based programming might stop people from shooting themselves in the foot, but I'm not sure if the cost to flexibility makes it worth it in every case.