This feels like re-discovering DSLs, except the syntax is English, and the implementation is a blackbox.
The point of DSLs are to provide a deliberately limited-scope language optimised for a specific problem or problem domain. LLMs that use general human language is like the furthest opposite of a DSL - its the broadest scope language for describing any problem, and they try to solve them all.
Also, few popular DSLs are truly blackbox in the sense chatGPT is - many of them have exposed source or even line-by-line debuggers available. There are a ton of other reasons this doesn't make sense to compare.