>Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space.
This is true and most researchers understand this on some level (https://arxiv.org/abs/2305.07759) but understanding it doesn't really make the problem any easier. How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ?
EDIT: This paper seems to take a good stab at the question. https://arxiv.org/abs/2309.04564