IANAL, but my understanding is that anyone who obtains a GPL'd binary is entitled to receive the source as well, from the source that provided them with the binary.

If you are careful with who you distribute the binary to, you could in theory keep the source sort of "secret", though I don't think it's a good general approach to GPL software.

For GPLv2, it depends. If you distribute the source with the binary, you don't need to give the source to third parties. If you provide a written offer for the source, that offer must be good for all third parties whether they've received the binaries or not.

I think to anyone who requests it, within a reasonable period of time, for at most a reasonable fee to cover your expenses for sending it to them IIRC.

Correct, unless someone asks for it, I believe.