"most is in use most of the time"

I don't know how you measured that, but Linux will use all of your memory all of the time with various caches.