Does opening another program cause system instability? I don't know how you are monitoring it, but it sounds like Firefox is making use of the available ram for a cache. There's no real difference between cache and free memory, so using more is only a good thing.