Being a polyglot is like magic, but it's not the Harry Potter kind of magic: it's the David Copperfield kind of magic. We look like we're doing one thing, but we're really doing something else.
Most polyglots are only really strong in two or three languages at a time. They may be passable in a few others, but they only remember bits and pieces of most of the languages they've worked with. They only have so much space in their head, just like you; the difference is in how they manage it. They organize their headspace differently, and there are upsides and downsides to that.
There are four main skills to being a polyglot. The first is knowing how to separate programming from coding. Your first language is always the hardest, because you're actually learning two things at once: you're learning the mindset and methods to create programs, and you're learning how to write down what you create in a particular language. When you go to relearn your second language, you don't have to relearn programming: you already know how to do that. This makes your second language, and later languages, easier than the first.
The second skill is understanding when and how it's okay to forget. There is only so much space in your head, and you forget things you aren't using. You mention this as a problem, but polyglots are no different. The trick is that almost every programming language was created because its author thought something was too difficult in the lanuages of the time, and wanted to program that thing in a way that flowed more naturally in the way they thought about it. These are the areas where you want to focus your attention: learn them as thoroughly as you can, and incorporate the concepts behind them into your general knowledge of how to program. The rest can be forgotten unless you're using the language often.
The third skill is how to draw connections between languages. As you learn new languages, focus on figuring out how to do the things you already know how to do in other languages. This will help you learn them faster, and relearn them faster (because you're going to have to relearn things you've forgotten sometimes). There's a book called "Seven Languages in Seven Weeks" which is good practice for this: you will not become an expert in any of the seven, but that's not the point. The real goal of the book is to teach you how to evaluate a new language and break it into parts that are easier to understand.
The last big skill is getting help. A polyglot is never going to have the same depth of knowledge in a language's libraries as a single-language expert will, so you need to know where to go when you don't know something. The official documentation is good. Other people are better, but not always available. You need to be able to have a sense of what you're looking for, where to get that information, and how to get it from there. You get faster at this with time.
That's most of it, really. And note that last paragraph: it's important to rememver that "polyglot" doesn't just mean "awesome". Polyglots get the job done and they look good doing it, but there's a price to pay for that. A single-language expert may or may not produce code that's any better than a very careful polyglot's. But the polyglot will take longer -sometimes much longer- to produce the same code, due to the overhead of relearning. This makes experts great, when you can find an expert that can do what you need. But sometimes, especially when your needs are very specific or demanding or the language you want to use is very new, it's not feasible to find an expert. That's where the polyglot shines: if you can't find an expert able to do what you need, the polyglot will become able to do what you need. Experts sculpt the language; polyglots sculpt themselves.
You're worried about whether or not you're cut out for this career, but I think you can succeed. If you want to become a polyglot, I think you can do that: it's a skillset you learn, not a talent you're born with. If you would rather develop as a Go expert, that's also feasible: Go is gaining a lot of popularity and seeing a lot more use than it used to, and there's a real need for experts. There are ways to branch out and carry on a good career with either path.
But in the end, it's your decision. This is a big step, and I can't tell you what's right for you. You don't have to change careers, but you can. You don't have to become a polyglot, but you can. You don't have to stick with this one language, but you can. You are not trapped.