E.g. a ring structure whose substituents are affected by steric hindrance in the left-handed scheme.
And the path of least resistance was just to adapt and build around it. Once that precedence was set everything became as such. I expect in the earliest stages of life this would have been an immense factor as metabolism was not nearly as sophisticated as we know it today.
And this selective process may have ocurred well before anything we have observed/modeled, and may well be erased. Which is to say I agree, but with the caveat that it was a substrate-dependent mechanism which selected the downstream components rather than random chance.