I believe that this is how your brain works, but it's far from universal. I've certainly never made anything like either of those number associations, nor do I associate most abstract memories with physical places (though I am familiar with the memory palace technique).