No-one should have the expectation LLMs are giving correct answers 100% of the time. It's inherent to the tech for them to be confidently wrong
Code needs to be checked
References need to be checked
Any facts or claims need to be checked
Code needs to be checked
References need to be checked
Any facts or claims need to be checked
Or maybe these benchmarks are all wrong
Makes you wonder what 97% is worth. Would we accept a different service with only 97% availability, and all downtime during lunch break?
It's not okay if claims are totally made up 1/30 times
Of course people aren't always correct either, but we're able to operate on levels of confidence. We're also able to weight others' statements as more or less likely to be correct based on what we know about them
Of course it does. The vast majority of software has bugs. Yes, even critical one like compilers and operating systems.
You must be new to LLM benchmarks.
As a user you can influence that behavior.