The dread is in the fact that this advancement could (and will) be amassed in the hands of a small minority of rich individuals. We, the plebes, will become from irritating but useful necessities to economic liabilities. Conspiracy theory time: this is the main reason, I think, that so many rich people invested money in AI: they saw that it could be a disruption of their wealth and they seized the technology in advance.
2. For those who say it is "just a language model".
We went from "it is just an autocomplete" to "it can write junior SWE code, undergraduate essays and, if prompted, imitate the style of writing of any author it has been fed" and this without being connected to the web. Given the speed of advancement how much time will pass before the economic possibilities of ascension for the lower classes will be forever shut off?
3. On the optimism for the future.
Every time a new "life changing" technology has been invented it has always been used for evil and every prediction of utopia has been shattered, again and again and again... This technology has a very higher reach than those before: it can write propaganda, it will be able to impersonate someone else and lead you to admit "wrong think", in the more optimistic future it will be used to hook you into the digital world so you don't reflect on your life: an infinite number of void possibilities, an infinity of nothingness. We already see some of these effect today with a more tame algorithm.
I do not believe in God, but if I did I would say: "May God have mercy on our souls". Sorry if I'm being melodramatic, I'm actually seeing (and I've seen) many door closing. The bar if ever higher for survival.