Eh, that really depends on how well the LLM understands edge cases of the particular need. Quite often these cases are hidden deep in files nobody understands any longer and will never get in a training set.

You end up with a system that works right up to the moment it doesn't and fails spectacularly and expensively.

This is one of those reasons you always hear about sweeping medical/hospital records systems being upgraded going tens or hundreds of millions over budget. The edge cases are demons.