We can't measure small changes and we aren't great at comparing across orgs.
However, at the director level we can certainly see a 50% or 100% productivity improvement in our teams and with individuals in our teams.
We aren't seeing changes of this magnitude because they don't exist.
Perhaps developers are now slacking off.
Perhaps we have added more meetings because developers have more free time.
Or perhaps developers were never the bottleneck.
We can see large productivity improvements when we make simple changes like having product managers join the developers daily standup meetings. We can even measure productivity improvements from Slacks/Zooms auto-summary features. Yet gpt/copilot doesn't even register.
While not code generation, this auto-summary is powered by the same tech. I think using it to sift through and surface relevant information, as opposed to generation of new things, will have the biggest impact.
By far the greatest value I get out of LLMs is asking them to help me understand code written by others. I feel like this is an under-appreciated use. How long has this feature been in Copilot? Since February or so? Are people using it? I do not use Copilot.
Now that's dangerous thinking, but I think you are onto something.
Edit: and now I see chillfox made the same point.