Without any middleman, this doesn't work, because the parties are in a prisoner's dilemma, rather than an iterated prisoner's dilemma—nobody can effectively punish the other party in a future transaction, so there's no incentive to not screw the other party over.
Without smart contracts, what you need here is a "marketplace owner"—a legal agent where both sides have a legal agreement with them. This changes the situation so both sides are playing an iterated prisoner's dilemma with the marketplace owner, who can punish a user on either side for what they did to a user on the other side in a previous one-off transaction.
But this marketplace-owner role is also known as a "middle man." Their role in punishing defection is valuable enough that the market doesn't disappear if they extract rent from both sides—and so they usually do. Ideally, we would have an alternative where everything still works out, but where rent cannot be extracted.
With smart contracts, the marketplace-owner role is performed by, essentially, the buyers and sellers all automatically voting on whether a given transaction was legitimate. The smart contract is just the specification both sides have agreed to, in a given case, for judging the legitimacy of a transaction; and the blockchain is just a database which all parties doing this auditing synchronize amongst themselves, containing copies of both 1. those specifications, and 2. all the data required to validate any new transaction against them (which, in the case of a ledger-like transaction, is usually "all the valid transactions executed so far").
There still needs to be a human-level legal recourse for disputes—but those disputes would only ever be disputes about the specification of transactional legitimacy being wrongly specified, which are much rarer than disputes of the form of "he defected!"/"no I didn't!", to the point that the service of arbitrating such disputes would not be seen, at a human level, as valuable enough to justify rent-extraction.