> it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths.

That's very true. I think LLM's have passed the Turing test and I remeber that used to be a very high bar. I remember being flabbergasted when chatgpt went public.

I guess I'm just not convinced self-improvement is a thing...yet

I think it comes down to an information theoretic question of: Can a LLM create information that is a projection outside it's data subspace? Something like can it increase it's "matrix rank" and not just create a dependent row. What ever metaphor... it's fun

> but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.

That's a good point too. The number of possible bad and good things goes up and every gets access to doing more complex things previously contained to someone's skill silo.