1. Yes, I think almost anyone can write code to procure the training corpus, in theory, and test it on a small scale
2. No, only the biggest labs and universities have enough resources to process such huge amounts of data and iterate on models with that scale. But that's just a matter of resources that can be overcome with partnerships between industry and academia that are common anyway. All the big labs already have huge efforts underway to reproduce GPT-X and it's just a matter of time before they catch up.
With OpenAI not being really open, it’s hard to say for sure what exactly ended up in the training materials, though. GPT-4 is even more of a black box to anyone outside of OpenAI with very little information released on how it was trained.