It feels almost as if, during inference, the model hits some format of local minimum that it careens around, and while temperature seems to affect this - it doesn't really fix it.
at temp 0.2:
> [{'generated_text': 'What follows is a transcript of a talk between a mysterious man and an agent of a bureau dedicated to investigating things which is typically referred to by some assortment of letters in the alphabet. The identity, origins, and motivations of the man were not known then and remain so. This transcript is not meant to scare, but provided simply to enlighten the concerned citizen of all the various and sundry things that may or may not go bump in the night. AGENT: Please state your name for the record. MYSTERIOUS STRANGER: I am the man. AGENT: Thank you. I am an agent of the Bureau of Investigation. I am here to investigate the following: 1. The following: 2. The following: 3. The following: 4. The following: 5. The following: 6. The following: 7. The following: 8. The following: 9. The following: 10. The following: 11. The following: 12. The following: 13. The following: 14. The following: 15. The following: 16. The following: 17. The following: 18. The following: 19. The following: 20. The following: 21. The following: 22. The following: 23. The following: 24. The following'}]
...and at temp 0.4:
> [{'generated_text': 'What follows is a transcript of a talk between a mysterious man and an agent of a bureau dedicated to investigating things which is typically referred to by some assortment of letters in the alphabet. The identity, origins, and motivations of the man were not known then and remain so. This transcript is not meant to scare, but provided simply to enlighten the concerned citizen of all the various and sundry things that may or may not go bump in the night. AGENT: Please state your name for the record. MYSTERIOUS STRANGER: My name is not important. AGENT: My name is Agent Cyanide. MYSTERIOUS STRANGER: Agent Cyanide. AGENT: I am an agent of the Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations. AGENT: The Bureau of Investigations. MYSTERIOUS STRANGER: The Bureau of Investigations'}]