I spent a week trying to do this by hunting down disparate blog posts that all used a different ML framework or GPT2 finetuning tool. I didn't understand what you were supposed to do and ended up with gibberish output instead. For example, isn't the dataset supposed to be specifically formatted to separate significant blocks of text, with something like <!endoftext>? The OP doesn't mention anything about this.

It's these kinds of things that make me doubt myself at every turn when doing this "finetuning" thing. I wish there was a clearer tutorial for people only interested in retraining GPT2 to generate different kinds of text, alongside a description of how each step works at an intuitive level. A lot of times the author starts right off with "okay let's start finetuning" without really defining what "finetuning" is or does. I just want to understand the scope of knowledge needed to know how to use my own text corpus as the input source without needing to become an expert in ML. (Maybe this activity isn't for me if that's how I'm going to think about it?)

If anything, I wish Talk to Transformer still existed as a downloadable model. I have yet to find another model that generates text like it did. At the end of the day I just sigh and look at all the random article and comment and adventure generators other people have succeeded in making without "getting" it.