For video theres VapourSynth[0], not sure of an equivalent for audio outside of just scripting ffmpeg but that’s probably not as flexible as what you’re after.

[0] http://www.vapoursynth.com/