Lately, my thoughts on our LLM future darkened a tad, too.
I find the parallel between LLMs and Sophons from the 3-Body Problem novels increasingly accurate. That is, specifically, in terms of their effect of stalling human progress. Sure, anybody can opt out of using LLMs (unlike with Sophons), so the restriction on progress seems kind of "soft" in this sense, but the chances of someone choosing to do so aren't too high, IMO. It is so much easier to avoid forcing yourself to learn or explore hard stuff and just ask the model instead.
While I tend to support this in a field such as SW Engineering (if only the Standards and Quality Control aspects advanced as fast as raw code generation...), I share the same fears as Tao and others regarding Mathematics (and not just Mathematics; Computer Science, which ultimately underlies SWE, is on the spotlight as well).