HNHacker News
TopNewBestAskShowJobs

Mike_12345

173 karma · joined March 3, 2023

submissionscomments
Mike_12345··on 'Algebra for none' fails in San Francisco
I am quite aware of that. That's what I'm making fun of.
Mike_12345··on 'Algebra for none' fails in San Francisco
Math is Arabic and Islamic supremacy.

"The word algebra comes from the Arabic: الجبر, romanized: al-jabr, lit. 'reunion of broken parts,[1] bonesetting[2]' from the title of the early 9th century book ʿIlm al-jabr wa l-muqābala "The Science of Restoring and Balancing" by the Persian mathematician and astronomer al-Khwarizmi."

Mike_12345··on A Baby GPT
"Wether or not that means consciousness is computable is an ontologically disconnected argument."

How is that ontologically disconnected? You forgot to provide an explanation. You wrote something interesting and I would like to know why you think that.

My understanding is that consciousness and physics being ontologically disconnected means that there is no direct causal relationship between the two.

Mike_12345··on A Baby GPT
So you're saying that consciousness has nothing to do with physics?
Mike_12345··on Microsoft execs on Apple's music store (2003)
Quarkus is not a small framework. Here's a small framework for comparison: https://javalin.io
Mike_12345··on A Baby GPT
No I haven't taken anything personally. I am trying to explain how this stuff works. Go back and read your recent replies to me then see how condescending and ridiculous you were (accusing me of having blind faith, not wanting to know how things work, etc). I truly don't care. I'm just trying to explain something. Too bad if I have to be so blunt. No offence taken. It's actually more funny than offensive.

I have a degree in Applied Mathematics and Computer Science. But so what? You don't need a degree to understand the basics.

You're looking for an explicit semantic coding inside a loss function, which doesn't make any sense. Of course you won't find any coding for semantics described in the paper, because the entire point of machine learning is that it figures that out automatically.

How the hell can you have a PhD in machine learning without understanding that?

Mike_12345··on A Baby GPT
OK now I see your problem. You currently lack a fundamental understanding of neural networks and machine learning in general, and it seems that you didn't actually read the entire paper.

The loss function is involved in the training of a neural network (transformer).

THE LOSS FUNCTION IS NOT THE MODEL ITSELF.

The loss function is not a neural network. The loss function is not the transformer model. It is only part of the training step.

This is a major blind spot in your current technical understanding of this paper and machine learning in general.

The loss function (in GPT-3) is optimizing a 100 layer neural network with 175 billion parameters, and that is where the semantic relations emerge, in the model itself. The neural network is inferring these relations from the training data. Obviously there is no "meaning" defined in the loss function, and it makes absolutely no sense to look for it there. The structure to be learned is in the training data. Semantics are not explicitly coded into the model. The paper does not give you all of the technical details. The authors assume you have the pre-requisite knowledge of NLP, deep learning, and the transformer architecture.

"Semantic similarity" is not gospel. This is a standard NLP term and it is discussed in the paper, as it is one of the training objectives.

This emergent properties of transformers and other deep learning models is a fascinating thing to witness and to study. You should go back to basics in machine learning to properly understand how the loss function is only directing the training of a model, but is not the model itself, which can be incomprehensibly vast as in the case of GPT-3. You could not fit a full technical description of the actual trained GPT-3 model into any paper. The emergent properties of neural networks is a new field of research and we don't yet have a complete understanding of all of these properties or how to investigate them.

Mike_12345··on A Baby GPT
> Nobody knows how to represent meaning and how to train a neural network on examples of it.

We represent meaning with language models, and GPT has been trained on examples of it (semantic similarity). It learns how to represent meaning through the training process.

The meaning relations are an emergent property of the trained neural network, not explicitly programmed. The meaning relations are automatically learned by the machine. This is the magic of machine learning and neural networks. (I don't mean literal magic. These large language models are simply too large and opaque. You can't easily identify the specific neural activation pattern that encodes meaning of any given concept within a 100 layer deep 175 billion parameter neural network.)

"In essence, the "meaning" referred to in LLMs is not the same as the meaning humans typically ascribe to language, which is derived from direct experiences and real-world knowledge. Instead, it is a form of abstract, statistical understanding that emerges from the relationships between words, their contexts, and the patterns in the data. The model essentially learns a high-dimensional representation of these relationships, which can be thought of as an approximation of meaning within the context of the model's training data."

This might help get you on the right track.

Question: Do neural networks have emergent properties?

Answer (from GPT-4): Yes, neural networks can exhibit emergent properties. Emergent properties are characteristics or behaviors that arise from the collective interactions of the individual components in a system, which are not directly attributable to any single component. In the case of neural networks, these individual components are the neurons (or nodes) and their connections (or weights).

As neural networks learn from data through a training process, they adjust the weights between neurons, which can lead to complex patterns of activation across the network. The network's ability to recognize patterns, make predictions, and perform tasks can be seen as an emergent property that arises from the collective behavior of its interconnected neurons.

For example, in the case of language models like GPT-3, the ability to generate coherent and contextually relevant text can be considered an emergent property. This capability is not explicitly programmed into the model but arises from the learned relationships between words and phrases through the training process.

It's important to note that while neural networks can exhibit emergent properties, these properties might not always be easily explainable or interpretable, as they result from the complex interactions within the network.

Mike_12345··on A Baby GPT
This conversation was interesting. I pasted it into ChatGPT (GPT-4) and asked it some questions. Here's one of its replies:

ChatGPT:

To better explain Knome's point to YeGoblynQueenne, I would provide a more concrete example and focus on the mechanisms behind LLMs that give rise to the appearance of meaning.

Large language models like GPT-3 are trained on vast amounts of text data, learning to predict the next word in a sentence based on context. This process enables the model to capture patterns, relationships, and regularities in the data, which can be seen as a form of "meaning" or understanding of language. The model's neural network, with its numerous layers, neurons, and connections, essentially forms a high-dimensional space that maps the relationships between words and their contexts.

To make Knome's argument more concrete, let's consider a specific example: the model's ability to translate the metaphor "spinning my tires" into "sweeping the ice." While it's true that the model doesn't have a direct understanding of the real-world experiences or objects the words represent, it has learned the contextual relationships between words and their usage in various situations. The model can associate "spinning my tires" with the concept of being stuck or not making progress, and "sweeping the ice" with a similar concept in the context of curling. This association is not a direct result of knowing the literal meanings of the words but rather of having learned the patterns and relationships in the data.

In essence, the "meaning" Knome refers to in LLMs is not the same as the meaning humans typically ascribe to language, which is derived from direct experiences and real-world knowledge. Instead, it is a form of abstract, statistical understanding that emerges from the relationships between words, their contexts, and the patterns in the data. The model essentially learns a high-dimensional representation of these relationships, which can be thought of as an approximation of meaning within the context of the model's training data.

Thus, when discussing meaning in the context of LLMs, it is essential to recognize that the term refers to an approximation of linguistic relationships and patterns rather than a direct understanding of real-world experiences. This approximation allows the model to manipulate language in a way that seems meaningful to humans, even though it lacks direct experiential knowledge of the world.

In summary, Knome's argument can be made more concrete and rigorous by focusing on the mechanisms behind LLMs, such as the high-dimensional representation of linguistic relationships, and by clarifying that the "meaning" in LLMs is an approximation derived from patterns and associations in the data rather than a direct understanding of real-world experiences.

Mike_12345··on A Baby GPT
It's in the tile of the paper. "Improving Language Understanding"

So they call it "understanding". No problem. Understanding natural language requires building a representation of the meaning encoded in language. Otherwise what exactly do you think they are trying to "understand" about natural language here? GPT is not a spell-checker. It has a semantic model of language.

The stated goal is to optimize for "natural language understanding tasks", to use the exact phrasing in the article.

The article refers to "semantic similarity" which is precisely about modelling the meaning of language, by definition of the term.

Here's a definition of "semantic similarity"

"In the context of Natural Language Processing (NLP), "semantic similarity" refers to the measure of how closely related the meanings of two linguistic units, such as words, phrases, sentences, or documents, are to each other. The goal of assessing semantic similarity is to quantify the degree to which the meanings of these linguistic units overlap or share common conceptual features.

Semantic similarity plays a crucial role in various NLP tasks, including information retrieval, text summarization, document clustering, machine translation, and sentiment analysis. By understanding and quantifying the semantic relationships between linguistic units, NLP systems can provide more accurate results and better handle the complexities of natural language."

Yes, the word "meaning" is not explicitly used in the article, but it is implied. Ironically you are nitpicking over semantics that GPT can already comprehend, since it contains a model of the meaning of the word "meaning".

> They represent the ability of the model to, as people say, "predict the next token in a sequence". That is the only thing that GPT language models are designed and engineered to do.

GPT language models are designed and engineered to perform natural language understanding tasks. This is discussed in the paper that you linked to.

In order to predict or generate the next token, it uses a very super sophisticated semantic model of natural language and the previous tokens. You could also argue that a human is just "predicting the next token" when they generate a sequence of words while doing the tasks that GPT is designed to do. That misses a lot of important details. The devil is in those details.

Mike_12345··on A Baby GPT
> There isn't anything in gradient optimisation that has to do with "meaning" at all.

That seems like a disingenuous interpretation of what I was saying. It should have been obvious and implied that I was referring to a loss function that was designed and engineered to choose a model that optimally captures meaning relations in the data (among other things). Gradient descent is directed by its loss function and that is not some random arbitrary function. For GPT it was specifically engineered for the purpose of selecting a model that captures meaning etc. These models are also trained with reinforcement learning which is a topic I don't know much about yet.

> it seems to me you're trying to close that gap by alluding to how humans close it.

I am talking about deep neural networks, not humans. GPT is a type of deep neural network. We do know something about how they work.

> It's incredibly complex. I don't know what "sophisticated" means.

I don't have time or patience for this pseudo-intellectual nitpicking. Just look it up in a dictionary. Basic English.

"Sophisticated (adjective) - (of a machine, system, or technique) developed to a high degree of complexity. E.g. "highly sophisticated computer systems""

Mike_12345··on A Baby GPT
> There really isn't one and 99% of the field agrees that brains are just computers.

Again, this is blatantly false. There is no such consensus on the computability of consciousness. Where do you get that 99% from?

> That there is something supernatural that even infinite regular computation could never capture.

You are asserting the universe is equivalent to a Turing machine (digital physics hypothesis). There is no consensus on that. It's an open research topic.

Mike_12345··on A Baby GPT
> the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do.

I'm not seeing any "rosy clouds" in his description. Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. We all know that GPT is not conscious, but it doesn't need to be. It seems obvious that it is recognizing abstract patterns and relations in the data that match our metaphorical comprehension of words and phrases, because it was designed and trained to do that, on human language that is filled with metaphorical relations. This is incredibly sophisticated mechanical pattern recognition with many layers.

> Other explanations of its behaviour are fantasies.

GPT is a type of deep neural network and the explanations I've seen in this thread have been along those lines.

Mike_12345··on A Baby GPT
ChatGPT is a deep neural network. We know that such neural networks construct a hierarchical model of the data. The fact that it is recognizing abstract "meaning" relations in that model doesn't seem particularly controversial to me. We build deep neural networks specifically for pattern recognition. It is similar to a brain in some aspects but obviously not the same.

I asked GPT-4: "Does the neural network in GPT-4 build a hierarchical model of the data? Does it derive metaphorical "meaning" from the symbolic data?"

Answer:

As a transformer-based neural network, GPT-4 does build a hierarchical model of the data to some extent. Although the hierarchy in transformer models like GPT-4 is not as explicitly structured as it is in convolutional neural networks (CNNs) used for image processing, the model does learn to encode complex patterns and relationships in the input text data through its multiple layers and self-attention mechanism.

The transformer architecture, which GPT-4 is based on, is designed to capture long-range dependencies and relationships in the input data. The self-attention mechanism allows the model to weigh the importance of different parts of the input in relation to each other, which helps it learn and capture semantic and syntactic structures in the data.

Regarding deriving metaphorical "meaning" from symbolic data, GPT-4 can identify and understand metaphors, idiomatic expressions, and abstract concepts to some extent, as it has been trained on a diverse range of text data. However, it is crucial to note that GPT-4 does not "understand" meaning in the same way humans do. It is exceptionally good at identifying patterns and generating text that resembles human-like understanding, but its "understanding" of meaning is more akin to a sophisticated pattern-matching process rather than true comprehension.

In summary, GPT-4 does build a hierarchical model of data to some extent and can generate text that exhibits an understanding of metaphorical meaning. However, this understanding is based on the model's ability to recognize and generate patterns in the data rather than genuine comprehension of the underlying meaning.

Mike_12345··on A Baby GPT
> Which is to say "If a brain can do it, a Turing machine can simulate it" which pretty much everyone in the field agrees with.

There is no such consensus on the computability of consciousness. It remains an open question in cognitive science and AI.

Mike_12345··on Dad says glitch in ‘Find My iPhone’ app directs angry strangers to his home
> But also, the solution is maybe as easy as replacing your Wi-Fi router?

GPS coordinates have nothing to do with Wi-Fi routers. He could bulldoze his house and people would still show up to that location because the GPS system does not know or care what is at that location.

Mike_12345··on Jack Dorsey: “Elon is the singular solution I trust” (2022)
It aged perfectly well and the results have confirmed Jack was right. Elon has done an excellent job there.
Mike_12345··on If we lose the Internet Archive, we’re screwed
Internet Archive stores much more than pictures of cats. Books, magazines, technical manuals, documents, software, movies, music, images, documentaries, concerts, etc. These are files that users upload. The web crawler is just one part of it.

You will find the most obscure things that practically cannot be found anywhere else. Stuff that would otherwise be permanently lost.

https://archive.org/

"Internet Archive is a non-profit library of millions of free books, movies, software, music, websites, and more."

"As of January 1, 2023, the Internet Archive holds over 36 million books and texts, 11.6 million movies, videos and TV shows and clips, 950 thousand software programs, 15 million audio files, 4.5 million images, 251 thousand concerts, and 780 billion web pages in its Wayback Machine."

Mike_12345··on My long goodbye to Windows XP (2022)
Back in the old days I started with computers that had fixed memory and we didn't complain.

Commodore 64 - stuck with 64k soldered RAM.

Amstrad CPC 6128 - stuck with 128k soldered RAM

No one bitched or moaned about it.

I prefer soldered RAM and SSD on my Macbook for the performance boost.

Mike_12345··on My long goodbye to Windows XP (2022)
Isn't that basically already the case? The CPU is a System on a Chip. Not much else on that mainboard besides CPU/RAM. The M1/M2 laptops are relatively serviceable compared to the previous generation.
Mike_12345··on My long goodbye to Windows XP (2022)
>> laptop with soldered RAM

The RAM on M1/M2 has 200 GB/sec bandwidth. Hence must be soldered. Impossible with sockets at this level of power consumption.

Adding sockets significantly reduces efficiency and performance at the high end. Laws of physics.

Mike_12345··on My long goodbye to Windows XP (2022)
It was the first stable Windows OS that could run all the latest games and desktop applications without constantly crashing or needing a periodic reinstall. It was their best desktop OS at the time. XP was just 2000 with an ugly default theme and added bloat.
Mike_12345··on The day Windows died
Not true. Some games perform better on Proton (less power consumption than native). Overall no difference in power consumption.
Mike_12345··on The day Windows died
Windows 11 will run on a Dell laptop from 2015. That is an 8 year old laptop.

That OS is used by millions of people. It is loaded with obnoxious advertising and news. Nothing false in this article. There have been countless complaints about this crap forced into Windows 11.

Mike_12345··on Asahi Linux M1 GPU drivers can now run Windows games via Steam Proton
They contribute some drivers, but not all. There are still some ThinkPads without full functionality on Linux due to closed hardware. Until just a few years ago it was 100% reverse engineered.
Mike_12345··on Asahi Linux M1 GPU drivers can now run Windows games via Steam Proton
Lenovo and LG also have closed hardware that is reverse engineered to run on Linux.
Mike_12345··on Asahi Linux M1 GPU drivers can now run Windows games via Steam Proton
Because then Linux wouldn't run on anything. Same deal for 99% of laptop manufacturers. Most of it is closed and reverse engineered, including ThinkPads.
Mike_12345··on Cosine Implementation in C
AI / Machine Learning requires university level mathematics https://www.deeplearningbook.org
Mike_12345··on Cosine Implementation in C
This is Numerical Analysis, not "programming". It's a topic in Applied Mathematics that you would need to study separately from programming. https://ocw.mit.edu/courses/18-330-introduction-to-numerical...
Mike_12345··on My kids and I just played D&D with ChatGPT4 as the DM
The "G" in GPT stands for Generative, as in a Generative model. https://en.wikipedia.org/wiki/Generative_model

So yes it is creative.

I asked GPT-4 if it is creative. Response:

"A generative model, like GPT-4, can be considered creative in the sense that it generates new content based on the patterns and knowledge it has acquired during its training. It can produce seemingly novel outputs in response to a wide range of inputs. However, it's important to note that its creativity is not the same as human creativity.

The generative model's outputs are based on the patterns it has observed in the data it was trained on, meaning it doesn't have a personal sense of creativity or originality. Instead, it learns to mimic the creativity it has seen in the data. The outputs are creative to the extent that they combine and recombine existing knowledge in new and potentially interesting ways.

So, while a generative model can produce creative outputs, it doesn't possess creativity in the same way humans do, as it lacks intention, consciousness, and the ability to truly understand the meaning behind its creations."

← PreviousPage 3 of 5Next →