Hacker News Reader

Comment by GaggiX | Hacker News Reader

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/

HuggingFace: https://huggingface.co/openchat/openchat_3.5

On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

View on HN

What is the sum of odd numbers in this set: 4, 7, 12, 1, 3

The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

You're using an LLM as if you don't understand how or why it works. Math is not what it's for. Here, look at what ChatGPT 4 does (for a case that hasn't been fed to it by a hundred other users yet):

    What is the sum of odd numbers in this set: 12345654321, 123456543212, 123456543213, 12345654324? Output only the sum, no code.

    The sum of the odd numbers in the set {12345654321, 123456543212, 123456543213, 12345654324} is 246913086434.

What does that prove? Only that I am using an LLM poorly and/or do not understand what it is. Using OpenChat-3.5 for what LLMs are actually good at (e.g. asking it for shell commands to perform certain operations, getting some general information about a topic) seems to work surprisingly well for a 7B model.

Not sure what you mean by 'using'. The prompt is from one of the benchmarks used in scoring LLM models - don't recall which. Here is another from BBH [1]:

>>> (-1 + 2 + 9 * 5) =

... (-1 + 2 + 9 * 5) = 43

[1] https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main

This is even worse than the other, just straight up use a calculator.

The use isn't the calculation, but the benchmark for its arithmetic capability.

is this really what you are using AI for ?

Technically 3 is even. \s

Are there any comparisons with Mistral-instruct? I've yet to see anything under 30B beat it in any way.

Mistral-instruct is in the arena, 1008 ELO vs 1070 ELO (OpenChat-3.5)

Reply on news.ycombinator.com