>that also makes good use of that full window?!
To be honest the "Needle In A Haystack" test is the most trivial test for a model that relies on full attention, it's expected to be easy to pass if the model was trained correctly.
To be honest the "Needle In A Haystack" test is the most trivial test for a model that relies on full attention, it's expected to be easy to pass if the model was trained correctly.
Phi-2 was 2.7B and it was already regularly outputting complete nonsense.
I ran the 0.5B model of the previous Qwen version (1.5) and it reminded me of one of those lorum ipsum word generators.
The other new Qwen models (7B and up) look good though.