> If you don't learn about grammars, lexers, parsers, ASTs,... you're at a huge disadvantage in many situations.
Can you get an example? I don't recall a time where I used this knowledge in the last 5 years as a systems generalist in big tech.
Your predecessors had zero regard for C programming best practices, leaving you with a pile of supremely brittle code that just barely works, but you have to tiptoe around it in silk gloves whenever you want to modify it in any way, because of unreasonable standards on ownership, calling conventions, concurrency, whatever. They might have had a reason in the beginning, but now it took over creating a monster that no regular programmer can work on. They didn't think that the code will ever need to be multithreaded. This is no environment for a sane person. A young, inexperienced fool would suggest to rewrite it from scratch. You know better. The whole business logic of a working, profitable business is imprisoned in this pile of shit, and yet it's too far gone to continue forward with it. Nobody understands all the silly rules of engagement with the codebase. You decide to: 1) break everything into chunks, 2) write extensive unit tests, 3) use automated tools to fix the whole codebase yourself to some sane standard. Some of the automation can be just grep + xargs + sed, some is much more complex. Regex replacements are unreasonably efficient, but not omnipotent. The only sane way to fix 40k LOC at a time is to write tools that operate on the AST, find toxic patterns and smartly replace them. This takes dozens of iterations, but you can always roll back with git. You're not transforming the code incrementally, you're writing a transformer. Eventually, your fixers are good enough that they fix 95% of the issues, you fix the rest manually. Your company can now hire any old mediocre C dev off the street, and this dev can productively contribute to the rejuvinated codebase. The company no longer needs psychics, regular C devs will do. If they stray from the path, the newly set up CI with dozens of custom clang-tidy plugins swiftly puts them back in their place.
Your company is in a highly regulated space that has insane coding standards. There are (shitty) compliance tools that cost >$100k a year for your team. You're able to replicate 80% of the functionality of the tools, fix some issues and generate compliance reports from the code on this in ~8 months. Slap on some CI, and the bean counters don't even have to do anything but download the latest PDF report.