Second: Massive capital expenditure, specifically in this case the huge cost of building or leasing enormous GPU clusters, is *exactly* what he means by this.
He didn't seem to have specific definition at all really.
I think most people attribute it to a "secret sauce technology" in the case of OpenAI, I'm not sure if "finances to lease a huge cluster of GPUs" makes sense here because the main competitors (Google, AWS, Apple, etc) also have access to insane compute as well yet have struggled to get close to GPT4's performance in practice.
That said I do agree that it's a moat for the startups like stability/mistral, etc. They also have access to $/compute, albiet a lot less. And you can see this in their research, as they've been focused on methods to lower the training/inference costs.
So at least in the battle between OpenAI and Google, their moat right now are their models.
If it was as simple as dropping 10's millions on compute they could do that, yet google's bard/gemini have been a year behind GPT4's performance.
That said I do agree that it's a moat for the startups like stability/mistral, etc. They also have access to $/compute, albiet a lot less. And you can see this in their research, as they've been focused on methods to lower the training/inference costs.
*I'm measuring performance by the chatbot arena's elo system and r/locallama