> but [LLMs] won't come up with [ideas] on their own.

They can, in fact, generate plausible performance optimization ideas on their own.