LLMs can be deterministic if you set the temp to zero.
It's not feasibly debuggable though, I'll give you that.
On this point it's not really different in principle from having a non-OSS compiler. Using OSS compilers are great because you can look at the source and trace problems and fix them, but people also use closed source compilers in their work too.