I create a "Library" directory on my machines because of this, it's not a great fix at all but it makes it a little more sane.

With this I can also more easily sync my files across machines.