Echoing a lot of sentiment here, you can do pretty much all of these things with SuperCollider. I'd go straight to SuperCollider rather than a "frontend" like Overtone/SonicPi. Not that these frontends are bad, but SuperCollider also has a nice frontend - the IDE works great with builtin doc searching, autocompletion, etc.
Sample wrangling becomes pretty thorny quickly as if you are triggering samples you want to make sure they don't "clip"/"pop" when you change them (due to discontinuities between the abrupt stop and the new start). I wrote a SuperCollider tutorial that demo's how to make a really high-quality sample player that takes care of these kinds of things plus gives you lots of control over rates, looping, etc. [1] That was my main hurdle with SuperCollider as that it has surprisingly few tutorials for such a long-running and deep program. Hopefully that helps :)