Given that a few people mentioned it having better than average knowledge of obscure things from the internet, I asked it what the llm tools from Simon Willison is. Gpt2-chatbot said "Log Lady's Mug". Gpt-4-turbo said "Log Lady Mysteries".
I tried a few versions of the prompt, including asking first about shot-scraper. It knows shot-scraper was made by Simon Willison and mostly knows how to use it, but that didn't help it with llm. Long tail is still a problem.
If I had API access to this, I might be interested in trying to find if there exists a prompt that can improve this sort of thing.