What test cases do folks here recommend for measuring this new model's ability to reason? and, specifically, if it can reason about code with similar (or better!) performance to ChatGPT4? Has anyone managed to get it running locally?
You could clone that repo, adapt the oaieval script to run against different APIs, then run the evals against both and compare the results.
Is that 61% using the non-parallelizable RNN mode or the standard mode? I wonder if it's the latter.
This new model may be a viable alternative to ChatGPT, which is not only closed sourced but can be shut down in the future just as they did with the older text-davinci models.
Plus, the alignement and safety has rendered ChatGPT useless for helping with areas such as critical analysis of social issues (that go against the aligned views) and any and all critical thinking that goes against the aligned views of those who own and program ChatGPT. This could a viable free (as in freedom) alternative.
https://paperswithcode.com/sota/common-sense-reasoning-on-wi...
macrolocal linked this below: https://paperswithcode.com/sota/common-sense-reasoning-on-wi...