Is the third quote really part of what GPT 2 outputted? Not sure if I should be amazed at the AI or ashamed I couldn't catch the joke.
GPT2 output quality is highly variable. Well crafted prompts can inspire some great output without much digging through samples. I believe I've gotten much better at writing prompts-- even using the right typography matters-- after spending some time doing it.
It sounds like GPT3 gives something much closer to best-of-gpt2 all the time. I hope I get access to it at some point (I also hope the api exposes raw probabilities).