You can't really understand programming without having built a compiler. Or at the very least having done assembly programming. I don't think you can truly understand any basic programming concept like variable, pointer, reference, etc without learning compilers or assembly. Yes, a pointer "points" to something. But what does that really mean? When you realize that it's all just higher level concepts of specific assembly mechanisms of accessing data, you'll have a eureka moment. Don't see how you do that without digging into compilers/assembly.
And ultimately, it'll make you a better overall programmer since compilers/assembly is fundamental to every programming language.