IrDA could absolutely be used without my knowledge.
Having actually used IrDA on Sony, Nokia, and Ericsson devices, no it couldn't.
In the real world, two IrDA devices have to be very specifically aligned, and also brought within just a few inches of each other. There's no way data transfer would happen without your knowledge.