That is like saying "We've known the impact of CO2 on atmosphere for a 100 years, you just need basic knowledge of chemistry, no need for any further study"
The author clearly had some understanding that context windows could influence their results, but they still decided to release an analysis that does not separate one data gathering technique from the other, allowing them to cherrypick LLM answers from either technique as needed, depending on whether they want to show more or less hallucinations.
It's not that we don't need a study, it's that we don't need bad studies.
So both ways.
Are you saying they took both methods and intermixed results to skew a narrative? That might be a bit of a leap, but I didn't go find the raw data to disprove that.
It looks like they asked questions within a context window, and also isolated in separate context windows. And compared results.
It seems like this was actually part of the study. How much does the context window skew results, versus if questions were independent? How is that a bad study?
You are saying the study is bad for doing what the study said it was doing. How can using the same context window be bad if studying the context window is what they were looking at. It sounds like you wanted a different study done where the data gathered would be different.
"context windows could influence their results" How much and in what way is useful to study. And, as windows get longer, many common users are just going with one long context and not starting a new window.