Hacker News Reader

> This is unfortunately non-trivial to quantitatively evaluate the performance against ChatGPT :-(

Compare using the loss function?

In LLM world, loss or perplexity may not be the best indicator of model performance :-( Perhaps HELM (https://crfm.stanford.edu/helm/latest/) but we didn't take deeper look as we are not the developers of this model

Reply on news.ycombinator.com