I think how well it handles Norwegian despite the small number of Norwegian tokens present for GPT3 (I'm not aware of up to date overviews of token distributions in the training data) is a good indicator that a lot of it does transfer.
the 'low resource languages jailbreak gpt-4' paper (https://arxiv.org/abs/2310.02446v1) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it's possible that RLHF does not do a great job here but the base models are clearly cross-applying intelligence from high resource languages to low resource languages.