That wasn't my experience. ChatGPT dealt with everything I threw at it, except when it refused to, and even then I could get round it by telling it to pretend etc. Every wild example that I saw on twitter I was able to reproduce
https://twitter.com/Replit/status/1620445121476202497?t=6Mm9...
I think the nuance here is that using an LLM is a bit like googling. It doesn't seem like it would be a skill but it is. You kind of have to nudge it in a way similar to what you would do in your own mind after finding the almost relevant stackoverflow post.
- Implement quadratic voting (a concept defined in Radical Markets by Weyl). When trying to explain it, it still invented smth that looked good
- Implement a First in first out calculation to file tax returns where within a 1 year holding period, tax rate is 0%. It calculated some really implausible stuff and it seemed easier to just figure it out myself.
- there were many such cases for me, but also good outcomes too