GPT3 really is remarkable.
If you told me in 2015 that I can give just a few examples of movie recommendations, on a model trained for general text, and get perfectly coherent recommendations, I wouldn’t have believed you.
If you told me in 2015 that I can give just a few examples of movie recommendations, on a model trained for general text, and get perfectly coherent recommendations, I wouldn’t have believed you.
They might as well rename to Standard AI.
I maintain an open source ML deployment platform, and I've interacted with a bunch of teams that have used it to deploy GPT-2. It was actually the platform AI Dungeon built their app on. GPT-2 is a beast to deploy—it's huge (almost 6 GB fully trained), requires GPUs, and scales fairly poorly. You need to autoscale GPU instances aggressively to handle any kind of real time inference situation with it, and even with spot instances, that gets expensive quick.
GPT-2 is 1.5 billion parameters, and at the time, was scandalously large. GPT-3 is 175 billion. For a model that large, there's real questions around whether it's even feasible for the average team to use it if it is not hosted somewhere else as a third party API.
From that perspective, I think the value OpenAI captures with the API is less about the exclusivity of the model itself, but the exclusivity of their infrastructure. Because of that, I wouldn't be surprised to see them open source the model for research.
However, I 100% agree that the fact that the model still isn't open is concerning, and it casts some doubts on whether or not it will ultimately happen in the future.
https://pdos.csail.mit.edu/archive/scigen/
Example:
https://pdos.csail.mit.edu/archive/scigen/rooter.pdf
In a way, I find the output better than GPT-3's. GPT-3 output is always a bit creepy to read, as if you are talking to a deranged person.