I think you're right. Cage's point was that silence doesn't exist. Much of his work was about denying the distinction between "music" as a predefined composition and "noise" as the other sounds going on. I remember an interview where he talked about how much he enjoyed listening to the traffic outside his apartment. It was clear that for him this was no different than listening to a musical performance; the thing that made it beautiful was the conscious attitude of the listener.

(Disclaimer: I haven't read the article in full yet, only the part mentioning Cage.) I think their point is different.

Cage’s silence is indeed imperfect (as in, you can hear the sound). Their point is that even a perfect silence is ‘Coloured.’ Quoting the article:

“He was asserting that the bits in his copy of 433.mp3 [silence created by a particular method] had a different Colour from those in a copy of 433.mp3 I might make by means of the /dev/zero procedure, even though the two files would contain exactly the same bits.”

This sounded quite weird and slightly crazy at first to me (and then author mentioned the experiment was done as a joke anyway), but I started to see it like a neat example of how factors such as knowing how the recording was made shape our listening experience.

I'm yet to finish the article, and want to thank StavrosK for posting the link.