The whole point of LLM/FMs vs good old fashioned ML is generalization to unknown domains, not just unknown tasks. The hunt for "gotchas" is the hunt for "not in your training data".
The whole point of LLM/FMs vs good old fashioned ML is generalization to unknown domains, not just unknown tasks. The hunt for "gotchas" is the hunt for "not in your training data".
This is described as a brand new flash model - still experimental - from a lab that is a side project for an investment firm that has never had a vision model before. That doesn’t scream flagship or frontier to me.
Unless we're talking about AGI, I couldn't care less if an LLM is bad at things they won't be doing anyway.
I'd rather focus training data on more useful tasks.
The whole point of these models is they're meant to be able to generalize fairly well, not just answer questions the got trained on.
Then I ran it 9 more times and it got right 9 out of 10 times.
Lesson learned to double check what I read on the internet.