Would really want to see some benchmarks against ChatGPT / GPT-4.
The improvements in the given benchmarks for the larger models (Llama v1 65B and Llama v2 70B) are not huge, but hard to know if still make a difference for many common use cases.
The improvements in the given benchmarks for the larger models (Llama v1 65B and Llama v2 70B) are not huge, but hard to know if still make a difference for many common use cases.
> [Llama 2] models have demonstrated their competitiveness with existing open-source chat models, as well as competency that is equivalent to some proprietary models on evaluation sets we examined, although they still lag behind other models like GPT-4.
It also seems like they used GPT-4 to measure the quality of responses which says something as well.
[1] https://ai.meta.com/research/publications/llama-2-open-found...
> In addition to open-source models, we also compare Llama 2 70B results to closed-source models. As shown in Table 4, Llama 2 70B is close to GPT-3.5 (OpenAI, 2023) on MMLU and GSM8K, but there is a significant gap on coding benchmarks. Llama 2 70B results are on par or better than PaLM (540B) (Chowdhery et al., 2022) on almost all benchmarks. There is still a large gap in performance between Llama 2 70B and GPT-4 and PaLM-2-L.
"The largest Llama 2-Chat model is competitive with ChatGPT. Llama 2-Chat 70B model has a win rate of 36% and a tie rate of 31.5% relative to ChatGPT."