Yes, this is where the collusion comes in. Many seller agents will simply ignore buyers without a buying agent.
If the seller's agent is comfortable acting as the intermediary, and recognizing the limits of conflict of interest, then it's in their best interest to keep all of the 6% commission instead of giving half of it to the buyer's agent.
Your area might be different, of course.