Hacker News Reader

Comment by berkes | Hacker News Reader

> Pretty soon GPT-4 will not be the best in the field. The next generation will perform much better.

What makes you believe that progress is linear, or at least a line forever going up?

I keep seeing people predicting rapidly improving AI, based on how rapid it improved over the last x months.

But why is that not an outlier? How do we know we haven't hit a ceiling and stagnating? Isn't progress typically very bumpy and sudden?

View on HN

>What makes you believe that progress is linear, or at least a line forever going up?

I assume neither of those things. I have however read a lot of the papers published since GPT-4 was trained. There have been a lot of advances since then, so much so that simply saying "a lot" seems to be a massive understatement.

I think it is a reasonable assumption that at least a portion of those advancements would be able to build upon the existing technology of GPT-4 to produce something greater.

I am not assuming discoveries yet to be made. I am considering existing discoveries that have not yet made it into the top level of production.

Technological innovation is not truly an exponential. It is instead a series of logistics. If we do not have further breakthroughs then AI technology will plateau. I do think we'll have those breakthroughs but it's impossible to say when they will happen.

I would say that Microsoft/OpenAI’s attacks on open source, whether it be through “AGI safety” BS as a front for regulating their way to a monopoly, or attempting Embrace, Exentend, Extinguish on companies like Mistral, and Cold War-style fear mongering about China, are the greatest near-term risks to linear progress. And it’s worth noting on that latter point that China is not similarly constrained and so could end up outcompeting the U.S., regardless

It’d be a bit hypocritical on the part of Microsoft/ClosedAI if they try to restrict AI tech in the name of safety while stepping around the fact that they themselves built it while essentially ignoring copyright.

Their commercial LLMs would not be possible without original creators who are now being ripped off and squeezed out of their jobs; if they don’t keep the tech open it will be very difficult to justify.

> China is not similarly constrained and so could end up outcompeting the U.S.

With what technology ?

US has long term export controls on China and as they have demonstrated with Russia recently once you have secondary sanctions in place everyone falls into line. So it's pretty likely they will be effective.

China actually has very advanced technology in literally every area of the sciences now. They just don’t bother talking about it in english

I recommend you follow some of the specialist chinese AI substacks to see what’s happening over there.

Esp around chip building

Not really buying the idea that China secretly has equivalents for Nvidia, ASML, TSMC etc.

SIMC right now relies entirely on existing US/EU hardware for their chip building.

New versions of which are no longer available to them.

>I recommend you follow some of the specialist chinese AI substacks to see what’s happening over there.

Do you recommend any in particular? I'm not familiar enough with the Chinese AI scene to know who to check out

I’m guessing you don’t read many machine learning papers. Chinese research in AI coming out of their top institutions (e.g. Tsinghua University, Shanghai AI Lab), is pretty much on par with state of the art in U.S. labs.

The export controls simply don’t work. Those chips still make their way into China. All those export controls do is slow it down a bit.

But China can outfit itself with more hardware even if it’s not as fast as the latest iteration and still speed past the U.S. while the U.S. and the EU argue about AI being racist or not.

> Those chips still make their way into China

Not at the scale you need to build a world-class AI.

Let me know when Huawei is able to place an order for 300,000 GPUs.

Okay:

https://asia.nikkei.com/Business/Business-Spotlight/How-Chin...

The greatest near term risk to progress here is cost but nobody wants to talk about it.

Architecture improvements and stuff like the 1-bit networks that have been all over HN recently are going to drastically reduce training costs over time and could easily result in very powerful local models. That’s why there is a rush to regulate and create monopolies. Microsoft and OpenAI know they have no real moat relative to the current pace of R&D.

Reply on news.ycombinator.com