The goal is to minimize counterparty risk, not entirely eliminate it... you are right that in many cases that just isn't possible.
For example there's a fad of putting "ownership" of physical assets into bitcoin tokens for example, fine, that works until someone steals the token and not the physical asset. Now who owns the asset? Reality is out of sync with what the block chain says so, there's no support to help me get it back and no backdoor.
Privacy is even harder. With Bitcoin everyone knows what you spend, where you spend it, probably what you spend it too with some extrapolation. With "smart contracts" now everybody knows who your arbitrator is, who you play poker with, what brand of fridge you own. There's no getting around that if you want a public ledger.
If you're taking input from an oracle (you almost always are), you have the same level of trust in it anyway, you just moved computation into the stupidly expensive and massively redundant portion of the system. The 'solutions' for distributed oracles with ShellingCoin and related garbage is just hand wavy nonsense with so little basis in reality that it's not worth talking about.
There is this notion of a contract. In a democracy a valid contract is an agreement (say between 2 parties) which is enforceable by courts and executive. Enforceable means that there is a cost of cheating. Even the most simple purchase contains a purchase agreement. If Alice walks into Bob's store and takes something without paying, then this is what we call theft (although the precise details of theft can be argued about ad infinitum).
In the real world when we enter agreements, we usually know to some extent who we are dealing with (Alice knows Bob's store). What would an arbitrary agreement over a network look like? That's hard to say. An analogon of a court is some arbitrary human decision, which can serve as an input. In Bitcoin there is no notion of an agreement, only cash transactions (and some limited facilities for escrow).
I believe that the concept of an oracle, and several other crypto-terms like Zero-knowledge proofs are an academic dead end. For my purposes I'm assuming that most people who want to use crypto systems don't understand or care about these things. There will be a great need for mapping abstract concepts to useful tools, such as an GUI operating system maps very abstract physical calculations and processes and makes them usable. I suspect 99% of Bitcoin users don't understand public-key cryptography, much less Elliptic curves and distributed systems. "Key", "signature" and "coins" are useful metaphors.
What is already happening are the first beginnings of joint stock ownership developing (DAC's) with mixtures of online formal agreements and informal agreements. Voting is not a panacea, but it helps. That's the kind of contract that can be developed in the short term.