The problem is that you have to invest actual effort to memorize things. Memorizing things you very rarely need is not worth it, even if you're amazing at memorizing things. It's impossible to memorize everything you would benefit from having access to in written form.