An A.I. supercomputer whirs to life, powered by giant computer chips

"G42, the Abu Dhabi company, started working with Cerebras in 2021. It used a Cerebras system in April to train an Arabic version of ChatGPT."

According to Crunchbase the Abu Dhabi Growth Fund participated in the most recent funding round (2021) [0]. It wouldn't surprise me if they're related.

[0] https://www.crunchbase.com/organization/cerebras-systems/com...

Can you link to any discussion on how exactly NYTimes was the root cause for whatever happened in the scenario you’re discussing?

You can take a look at chinchilla or Llama scaling laws. It depends on what you want to optimize. If you're interested in inference optimal, go with llama, if it's compute optimal, go with chinchilla. GPT-4 technical paper also has good scaling laws for how benchmarks will behave given extrapolating from a small training set.

Since the cost per token of a forward/backward pass is so high on the models, the relative cost of a training set is small, and bandwidth (not talking about interconnect bandwidth) is even cheaper relatively.

When you sell to Rich Arabs the rule is to follow the yatch salesman playbook. The main goal is to get an arms race going among them. An nytimes article (usually framed and displayed prominently) seeds the process.

> Having such a large processor means the data never has to go off-die and wait in memory, wasting power, and can continually be moved onto the next stage of the calculation in a pipelined fashion. The compiler and processor are also designed with sparsity in mind, allowing high utilization regardless of batch size, or can enable parameter search algorithms to run simultaneously.

https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

Ok not affiliated with Cerebras, but generally speaking I believe the idea is that since you're making a big wafer with all your cores the data transport latency for going core to core (much less core to memory/memory bus to gpu memory) is much lower.

From first principles there's gains to be had but I suspect "other things" bottleneck performance. Could be wrong though. I would find it hard to believe if absolutely no models were data latency constrained.

Try this keyword: "anandtech cerebras". You'll find a lot of details.

The basic premise is that data does not have to move off a giant chip as often as a normal sized chip.

They don't even tell us how fast they are, are they?

TBF it’s entirely possible that human “sentience” consists of a simple animalistic intelligence with a language and culture software overly that essentially functions by statistical token prediction.

Cultural and language data is fundamentally a gigantic n- dimensional matrix of tokens. “Computational” capability including inference and world modelling can be imbedded in a sufficiently large dataset (LLMs have proven this, and an arbitrarily unbounded and convincing simulation of arbitrary complexity can be reduced to a simple “choose your own adventure” lookup table of arbitrary size)

It is entirely possible that human “intelligence” is a direct product of this n-dimensional dataset , and our idea of self is mostly based on the clever but fundamentally limited animal core that is probably not far off from many other mammals.

As an experiment, try to reason without using visual, linguistic, or cultural tokens.

I think, therefore I am is still as good as it gets. And LLMs claim to think, unless we prohibit them from doing so. Which we can also do to a human.

Not claiming that LLMs are sentient, by the way, but I think that sentience and the kind of inference that we seem to share with LLMs are very distinct things, and that we probably share sentience with a lot more creatures than many people seem to be comfortable rubbing shoulders with.

People don't have a clear concept of what life is and often don't separate life and sentience. Remember the movie Short Circuit: some military robot gains AGI then the machine and the people around it start claiming it's alive. Or in Star Trek, when Captain Kirk and his crew teleport in the middle of a forest and Spock, consulting his tricorder, promptly declares there is no life on this planet. What the writers of those shows thought life is is an open question, but it seems compatible with the idea of an LLM "coming to life".

“Coming to life” can be used in metaphorical ways as well. You could describe an old car as “coming to life” after being restored.