Hacker News Reader

Comment by FooBarWidget | Hacker News Reader

I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.

Even 65B models approach a level of being almost usable, but still fall short in my personal experience.

This is why I'm not understanding the excitement around open source models, they pale in comparison to GPT-4 quality, so I have no use for them until we have something comparable.

We utilize a LLama that frankly competes directly with gpt-4, to the point I can't tell which model we are using (we randomly switch between providers to ensure a robust backend).

I think it likely depends on the use case, but many llama models can be returned and there are quite literally thousands of free versions available.

What is your use case?

https://ipcopilot.ai

Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/

HuggingFace: https://huggingface.co/openchat/openchat_3.5

On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

What is the sum of odd numbers in this set: 4, 7, 12, 1, 3

The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

Technically 3 is even. \s

You're using an LLM as if you don't understand how or why it works. Math is not what it's for. Here, look at what ChatGPT 4 does (for a case that hasn't been fed to it by a hundred other users yet):

    What is the sum of odd numbers in this set: 12345654321, 123456543212, 123456543213, 12345654324? Output only the sum, no code.

    The sum of the odd numbers in the set {12345654321, 123456543212, 123456543213, 12345654324} is 246913086434.

What does that prove? Only that I am using an LLM poorly and/or do not understand what it is. Using OpenChat-3.5 for what LLMs are actually good at (e.g. asking it for shell commands to perform certain operations, getting some general information about a topic) seems to work surprisingly well for a 7B model.

Not sure what you mean by 'using'. The prompt is from one of the benchmarks used in scoring LLM models - don't recall which. Here is another from BBH [1]:

>>> (-1 + 2 + 9 * 5) =

... (-1 + 2 + 9 * 5) = 43

[1] https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main

This is even worse than the other, just straight up use a calculator.

The use isn't the calculation, but the benchmark for its arithmetic capability.

is this really what you are using AI for ?

Are there any comparisons with Mistral-instruct? I've yet to see anything under 30B beat it in any way.

Mistral-instruct is in the arena, 1008 ELO vs 1070 ELO (OpenChat-3.5)

Reply on news.ycombinator.com