I've been similarly frustrated with DAWs too. In the last few weeks I had a go at writing a DSL that shells out to SoX for audio manipulation. This way I don't have to manipulate audio samples myself.

It's way less polished than pydub but here it is if anyone is interested:

https://github.com/emillon/tmc

Example:

https://github.com/emillon/tmc/blob/master/Music/TMC/Example...

At first it was a pretext to play with free monads, a way of building EDSLs. But right now I'm not sure it's not just a complication. Though, having an intermediate representation before executing the SoX commands makes it possible to write an optimizer (for example, collapsing two audio shifts).