Yes, there's so much to learn that you'll never have time for it, even if specializing in a small area. It reminds me of a Chomsky interview. He said that he has so many books left to read in his office alone that a lifetime wouldn't be enough. You're in good company.
It may sound obvious but don't forget that HN isn't one person. The guy that knows about particle physics is usually not the one that tell you about the latest type theory research. Don't compare yourself with a collective mind.
Besides, I'm sure there are people less bright than you in all positions you can imagine. Retrospectively, I realize that there are a lot of things I didn't even try for fear of failing or because I thought I wasn't smart enough. It's only a few years later that I realized I missed so many opportunities.