Yes. Agents are good at solving densely represented (embarrassingly solved) problems, and a surprising and disturbing number of problems we have are, at least at the decomposed level, well represented. They can even compose them in new ways. But for the same reason they would be unable to derive general relativity, they cannot use insight to reformulate problems. I base this statement on my experience trying to get them to implement Flying Edges, a parallel isosurface extraction algorithm. It’s a reformulation of marching cubes, a serial algorithm that works over voxels, that works over edges instead. If they’re not shown known good code, models will try and implement marching cubes superficially shaped like flying edges.

You are still necessary to push the frontier forward. Though, given the way some models will catch themselves making a conceptual error and correct in real time, we should be nervous.