I don't think there is any part of my little software empire that is dependant on code for which I don't have the source or underlying .dll checked into source control.
It's part of your project. You absolutely need a copy of it.
These are building blocks in a normal dev environment, and it would take me massive amounts of time to manage everything on my own.
The local copies are fragile and not as easily shared.
You need to archive them somewhere anyway (to mitigate the issue we're discussing here), so why not keep them in the obvious place?