Part of the issue is they are training a pretty tiny model, it's not like GPT-2 ~100M is especially coherent either.