Almost everything starts to make noises at higher currents (e.g. CPU voltage regulator). You have of course coils making noise, you have ceramic capacitors making noise to due piezoelectricity, even good old fashioned electrolyte caps can make noises and transistors buzz due to thermal expansion of the die under switching stress. Not in computers, but even batteries, cables and bus bars make noises under pulse current loads, as do film capacitors (used in high power snubbers).

In the case of capacitors it also goes the other way; tapping a capacitor induces voltage spikes which can disturb analogue signals.

It's often both. Capacitors also have a capacity (hah!) for making noise. Coils generally get it first since they are a more straightforward electromagnetic device.

Ceramic capacitors still make noise (and can pickup noise) thanks to the piezoelectric effect. Coils typically make more noise though.