That's just blatantly false. Distribute your binaries with source. Or distribute them with an offer of source.
There, problem solved. Heck, may your binaries non-redistributable without source.
Either way, users have the option of using/distributing your binaries, or not. You've given them nothing they didn't already have.