harisec

I think OpenAI has a few hidden advantages that are not obvious at this point. It could be additional training data, filtering/preprocessing that data, some changes to the architecture, who knows? None of the open source models are even close to GPT 3.5, what to speak about GPT 4? I've tried everything and the 60G llama variants so i'm not sure it's about number of parameters. They definitely have some hidden sauce.

Same here. If you believe the following research (which I do), the ability to perform complex reasoning is likely to be from training on code:

https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...

I think it's essential to increase the quantity of code tokens.

This is a common misunderstanding; fine-tuning a model does not mean teaching the model new information. Fine-tuning is used to adapt the model to perform better in a specific task or domain using the information it already has, in a specialized way (like a chatbot for question/answer interactions). Training a LLM on new data is extremely expensive so it's not possible.

Bard is a lightweight model version of LaMDA. LLMs are very expensive to run.

Buried in the article is this gem: "The latest language model, GPT-4, has 1 trillion parameters." - have no idea if it's true or not.

Hidden inside the article: "The latest language model, GPT-4, has 1 trillion parameters."