Hacker News Reader

Comment by starik36 | Hacker News Reader

ChatGPT is probably the first software product that I have no idea how I'd go about implementing. I watched a number of YT videos about it including Andrew Carpathy's 2 hour coding session building a mini GPT.

I understand the process abstractly, but I am unable to grok the details about how it's able to take my vague interpretation of what I want and then write code and actually give me exactly what I wanted.

View on HN

The ChatGPT video by Karpathy is the last in a 7 video series. The one that really hit it home for me was the first video on MicroGrad [1]. The second video builds on MicroGrad and is also great for understanding how a basic NN works.

[1] https://youtu.be/VMj-3S1tku0

Yeah, unless you're already familiar with the advanced math behind all this don't expect an in-depth understanding of the implementation details the parent comment is concerned about.

It parses natural text into a format representing logic, that is the biggest thing to get.

Then once you have such a parser you can now make a logical Markov chain, it predicts the continuation of your text based on continuations of things that looks similar in the logic space rather than the text space, and that is what you get back from the model.

So ChatGPT does what we could have easily done if we had if natural language was more logical. Then we could make a simple model based on all the logic encoded in all writing of humanity that just looks up human done logic similar to what you asked for and then return an average of those logics. Now since you want human language output it now has to translate that back to human text from logic space, and it could be done in English, Spanish, it could speak like a pirate etc.

Do you know how you'd go about implementing a video codec? A Haskell compiler? Linux kernel? Efficient relational database? wifi 802.11n radio? An efficient classical AI chess engine?

I don't exactly how, but I have a good idea where to start. I've built a compiler before (in college). I've bootstrapped from hardware before, so I probably could built a very basic kernel. I've worked with radios at the low level in the 90s.

But with ChatGPT, it just seems like magic.

None of the actual moving bits are magic, though, and it's way more accessible than an OS kernel. The problem is that 1) you need a lot of raw GPU compute power to train and run a model that has GPT-3 capabilities, and 2) because the end result is a black box, nobody knows how it "actually works". In that sense, it is indeed technomagic.

I would recommend playing with the new ChatGPT API a bit.

Trial and error in terms of tweaking the system prompt is surprisingly instructive.

Reply on news.ycombinator.com