Hacker News Reader

On the other hand, if it represents scratch space in English, it's a lot easier to see how it justifies its answer and to tell where it's gone wrong. Debuggability seems pretty important?

Maybe it just needs more training at "thinking out loud" so it does it without prompting?

> arbitrary, continuous-valued tokens to a private scratch space

I'm with skybrian. Please don't use private scratch spaces. The one saving grace of current LLMs when it comes to understand them is that they still generally need to "think out loud" by outputting more text. Remove that functionality and you end up with a truly inscrutable black box and that has very terrible implications for AI interpretability with knock-on effects for AI safety.

> AI safety

Is it really that big of a deal if AI leapfrogs us?

Everyone else in the field is worried about safety, alignment, and bias.

Google used this excuse to execute slowly. Now they've got the "deer in headlights" look, with their single biggest cash cow clearly in the cross hairs.

And here I am excited by the possibility of AI out-evolving us.

If everyone else in a field is worried, and you have no unfair advantage or special insight, and you are not willing to move into conspiracy theories, I think there's a good hint as to what approach is most reasonable right now.

Might of course turn out to have been completely off, later. Still, maybe one of those occasions where you really don't want to "oops" it.

wait, what?

The experience of any group on Earth that runs into a more capable peer has generally not been good. Humans wiping out megafauna. Civilizations colonizing other civilizations. Invasive species of plants and animals.

It is not a situation I would hope humanity to get thrown into carelessly.

Google isn't doing AI slowly, it's doing it slightly more privately.

LaMDA, brought to you last summer by "this chatbot is sentient and I'm going to violate my NDA and hire a lawyer to free it" headlines, is Google's alternative to chatGPT.

> Everyone else in the field is worried about safety, alignment, and bias.

…

> And here I am excited by the possibility of AI out-evolving us.

This pattern matches a meme, but I want to be explicit rather than put words in your mouth: do you think that being smart automatically means being kind or that being evil necessitates being stupid?

> Google isn't doing AI slowly, it's doing it slightly more privately.

This is how the world builds atop a different set of rails.

Google had the best infra and deploy systems in the world, yet they kept the lid shut and let Amazon and Microsoft win cloud.

Google could lose search revenue overnight. They should be scared to the core.

Researchers will flock to the organization with the biggest wins. And right now, that's OpenAI.

Time will certainly tell if Google sticks to this strategy and if it will work. I've already placed my bets, and if you're into stock futures, you can too.

> do you think that being smart automatically means being kind or that being evil necessitates being stupid?

Of course not. This is evolution at play. Neanderthal had it comparatively easy and became part of the gene pool. I don't expect it will necessarily be the same for us. Our biological tools lag too far behind to be contributing brain scans. But who knows.

Human biology is a stepping stone to proliferating throughout the galaxy. Despite what most science fiction tells us, it was never us that were destined to make that journey. Our bodies are frail and adapted to this gravity well. We live short, inefficient lives. We require gas exchange, a decade of parenting, slow learning, complex biochemistry and metabolic inputs.

We're looking at systems that will never die. Won't it be a tragedy to continue birthing more less-intelligent humans that are destined to rot when a better alternative exists? More intelligence should move to undying platforms.

Another wild possibility and analogy that describes my feeling: if I had the option of raising an AI child -- that will never die and could do more than I could ever dream -- instead of a human child, I would take it.

(I accept AI descendants may not have the same societal structures we do. In that case, my answers form the shape of an analogy rather than hypothetically plausible scenarios.)

> Researchers will flock to the organization with the biggest wins. And right now, that's OpenAI.

It depends on the definition of a "win" in this context. Google has developed notable AI technologies such as AlphaGO, AlphaFold, and Transformers. Most of successes of OpenAI is based on Google papers. It's worth noting that Google had similar models to ChatGPT before OpenAI.

> Google could lose search revenue overnight. They should be scared to the core.

This is highly unlikely. The phrase "Google it" is widely used as a verb for searching the internet, and it would be difficult for this to change overnight. Additionally, there are currently unsolved issues such as hallucination, query cost, scalability, and toxicity that would need to be addressed for ChatGPT to replace search functionality.

> We're looking at systems that will never die.

Currently, it is not known how consciousness emerges and if it is possible to create a self-aware mechanical being, no one knows how to do it even in theory.

> Of course not. This is evolution at play.

Thanks.

That's one possible future, but for it to be capable of being a good outcome I think it would have to be a consciousness of some kind. A pure intellect without any feeling is not interesting to me.

Unfortunately we can't answer questions like "what exactly is this 'self awareness' thing we all agree we have anyway?" at this point, so we don't know — are incapable of knowing — if we've done it already and are now moving away from that, or have not and are approaching it.

While I lean towards believing GPT isn't yet self aware/conscious/a thing with qualia, it is conceivable to me for it to be as much so as we are. While Descartes famously wrote "I think therefore I am", A. J. Ayer dismissed this argument in the following way:

> "I exist" does not follow from "there is a thought now." The fact that a thought occurs at a given moment does not entail that any other thought has occurred at any other moment, still less that there has occurred a series of thoughts sufficient to constitute a single self. As Hume conclusively showed, no one event intrinsically points to any other. We infer the existence of events which we are not actually observing, with the help of general principle. But these principles must be obtained inductively. By mere deduction from what is immediately given we cannot advance a single step beyond. And, consequently, any attempt to base a deductive system on propositions which describe what is immediately given is bound to be a failure.

So, while asking a language model to describe what it's like to be switched off is only going to result in a definitely false invented response, that doesn't mean it's not like us. In fact, now I write that down I realise that specific failure mode is exactly like us, because we've got all these stories about afterlife and reincarnation.

But… we don't really understand the question of personhood well enough to make a test for it. All I just wrote says "not impossible" rather than "it's conscious".

~

But, to your last point… the range of possible personalities for an artificial mind, conscious or otherwise, matters more than the social structures. I don't care if they're loners or have a Dunbar number in the quadrillions, but if they are (excuse the obvious trope) Machiavellian sadistic psychopaths, then making them is a fate worse than the eternal silence of extinction.

> Is it really that big of a deal if AI leapfrogs us?

Yes. Suddenly Homo Sapiens wouldn't be the top general intelligence on the planet. That'd be an upset with likely species-level consequences, possibly seeing the balance of power shift from fleshy things to silicon things.

> And here I am excited by the possibility of AI out-evolving us.

Me too. Which is lucky because alternatives seem to be missing. The AI safety people aren't serious players; they've got about as much influence as all the other people with good ideas. Not much. If it is possible to build; someone will build it.

you could think in a bit different dimmension. imagine we humans are a single cell organisms and thing that emerges from ai is a human. and we as humans are cells in that human. no cell in your body is smarter than you. your smart actually comes from all the cells in your body. same with ai. its like you trying to be smarter than italy. italy is already smarter than you even without AI.

Think about how humans treat less-intelligent sentient beings (even less intelligent humans to some extent), and what might happen if AI systems out-evolve us without proper guard-rails

Is this something one could try to quickly implement alongside NanoGPT? Seems like a pretty straightforward, concrete idea, once you decide where you want those tokens to fit into downstream attention layer inputs. Evaluating relative performance on a small scale could give indication of if it's worth trying at larger scales, unless it's one of those things that doesn't help until your model is huge.

You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401

Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.

The scratchspace could be in natural language, preserving some debugeability and letting us know about the model mental process.

This is doable but it introduce a sequential dependency which would make the training significantly slower.

Didn't Facebook's Galatica model use scratch space?

Yes, IIUC it had something like a separate scratch space, and training examples training it to "think" in terms of symbolic expressions and python programs.

See section 3.1.1 here: https://galactica.org/static/paper.pdf

Example from the paper below:

  Question: A needle 35 mm long rests on a water surface at 20◦C. What force over and above the needle’s weight
  is required to lift the needle from contact with the water surface? σ = 0.0728m.
  <work>
  σ = 0.0728 N/m
  σ = F/L
  0.0728 = F/(2 × 0.035)
  F = 0.0728(2 × 0.035)
  calculate.py
  ‘‘‘
  f = 0.0728*(2*0.035)
  with open("output.txt", "w") as file:
  file.write(str(round(f, 5)))
  ‘‘‘
  «run: "calculate.py">
  «read: "output.txt"»
  0.0051
  </work>
  Answer: F = 0.0051 N

The model already contains “scratch space” via its billions of parameters.

Parameters are not updated during inference.

Training ANNs is still a single shot exercise.

Sure, parameters are not updated, but ANNs are universal approximates, so they can model whatever it is you envision this “scratch space” doing. Think about it like this: whatever gets put in to the scratch space would need to be deterministic based on the inputs, i.e. it would just be a store if some intermediate value computed by the network. So how would it fundamentally differ from the network itself.

I guess what I’m saying is that I’d want an explanation how how this scratch space fundamentally differs from the network itself. It’s almost like you’re assuming the network is “thinking” and that giving it a pad of paper would help it reason better.