I'm using this sort of spec-driven + feedback-loop workflow at work to pretty to good effect, and would say I'm quite familiar with the pros and cons.
IMHO the main downside is that you're simply not as familiar with the code base as before, unless you spend just as much time studying the LLM output as writing the code manually in the first place.
E.g. you have about the same distance to the actual implementation as a manager of a traditional programming team who from time to time skims over the source code to check for signals that things start to go sideways, and otherwise mainly feeds feature specification tickets into the team which are coming from a separate 'design department'.
I also would never call any of this work a 'labour of love' either. Not that this sort of soulless "specification-in-implementation-out" was much different before though, it's just how industrial software development works - essentially an assembly line for features.