This has definitely been discussed. There have even been some projects, although I haven't checked on the status of any of them lately. As best as I can recall, there are some specific structural reasons why it's hard to train LLM's this way, but I don't recall all the details offhand.
https://www.google.com/search?q=distributed+model+training+s...
https://news.ycombinator.com/item?id=35799843
https://www.reddit.com/r/ArtificialInteligence/comments/18q6...
https://www.reddit.com/r/slatestarcodex/comments/1gtnxgd/wha...
https://github.com/BOINC/boinc/wiki/Using-BOINC-for-AI
https://the-decoder.com/ai-startup-prime-intellect-trains-fi...