Physical objects only ever do exactly what they're 'told' to do, period. Rocks don't have a mind of their own, and physics isn't magic. Yet I don't want to live in a world where we don't spend time twiddling with dials and poking around.
I agree with the thrust of your statement, but it's worth keeping in mind that while physics isn't magic it is not deterministic either, once you start working with really small things.