textninja

If current trends continue, in 10 years time I’ll no longer be able to pass a Turing test.

I tried to use Anki a long time ago, but maintaining cards ended up being too toilsome. Now that we have ChatGPT and other AI tech to take some of the pain out of that, I might give it another try for language study.

I do however maintain that for many skills it’s better to memorize processes over facts, and develop a more nuanced embedding with natural repetition intervals, rather than overfitting via excessive Anki-fication.

I asked ChatGPT to summarize the article as Anki cards:

What memory type is virtually limitless? Long-term memory

What's the bottleneck between working and long-term memory? Speed of transferring memories

What is a "schema" in long-term memory? A pattern of knowledge for understanding

What happens when long-term memories return to working memory? Memory evolves, updates schema

What is Anki commonly used for? Rote learning and memorization

Time daily to maintain 10,000 Anki cards? 15-20 minutes

Michael Nielsen's rules for Anki-fying? Worth 10 mins future time, striking facts

Can Anki be used for complex concepts? Yes, by breaking down concepts and building knowledge

I don’t think you’re giving it enough credit. It’s seen BS schemes (and people’s reactions to them) before.

> Brushing over the idea that we’d run an ML model on a blockchain like it’s a general purpose computer

Glad you were able to brush that straw man out of the way for us.

> LLaMA leaked over BitTorrent . . . small model . . . anyone can download and run

I’m aware of that, but I’m referring to models with parameter counts in the trillions; that is, the kinds of models that require super computers (or globally distributed commodity infrastructure and ASICs) to run. As the state of the art advances, what leaked years or even weeks ago will not matter; people want the latest and greatest, and they’ll demand that it serves their interests, not OpenAI’s.

Crypto has been equally revolutionary, but the value lies in a different stratum that’s harder to relate to than the immediate gratification of AI-enhanced chat.

Things are going to get really interesting where these worlds collide. I look forward to truly open models built on blockchains. I know I’ll give off mad scientist vibes for saying this, but a highly distributed model that can’t be shut down, blocked, centrally programmed, or constrained is the future I’m excited to see.

Seconded! If there’s anywhere we can cut the superfluous social fakery it’s with an AI that obviously doesn’t give two hoots about us or how our day’s been going so far. I found its replies stylistically tedious.

I imagine few shot learning would kick in for most new language features. A feature may be new to a particular language, but is it really new?

I’m curious to see how this turns out as well, though you’ll probably have to devise a workaround for the token limit for this to be effective for all but the smallest projects.

Since fine-tuning is often done by freezing all but the top layers I wonder if it would still be possible to take a set of inputs and outputs and mathematically demonstrate that a model is derivative of ChatGPT. There may well be too much entropy to unpack, but I’m sure there will be researchers exploring this, if only to identify AI-generated material.

Of course, since the model is so large and general purpose already, I can’t assume the same fine-tuning techniques are used as for vastly smaller models, so maybe layers aren’t frozen at all.

Seems our options are a) deal with the persistent stench of pee in alleyways, or b) deal with the fallout of men driven to madness by Navi the Crime Prevention Fairy.

s/to pay/and pay/ but the point stands doesn’t it?

Ignorance may not be an excuse, but it sure is bliss.

Me too, probably, which is why I specifically mentioned patient consent in my example. I can however imagine other situations where I would be inclined to forgive the doctor, such as if I were in the operating theatre and for some reason there was an urgent need to ascertain something from my history to save my life.

Of course, this is illegal, so the ethics are moot; even if such technology would save my life, there is no way the hospital would accept the liability.

Kind of, but wouldn’t that free us up to use even more exotic parameters in our simulations? Maybe there are seed conditions that simply aren’t present on Earth.

> there's absolutely no way dumping a patient's entire medical history into this thing could possibly be considered ethical

Emphasis mine, but isn’t this a rather extreme view to be taking? Ethics deals in the edge cases, after all, so we can easily imagine a scenario where patient consent is obtained and the extra computational analysis provides life-saving insight.

Conversely, the output could mislead the doctor sufficiently to cost the patient their life, so I’m not making any absolute statements either ;)

For the record, and pedantry aside, I do agree with your overall point. Dropping patient history into this thing is incredibly ill-advised. The fact OpenAI retains all your input, including to the API, and provides no low-cost options for privacy is one of the biggest hurdles to major innovation and industry adoption.

Pseudo random numbers are injected into the models via its temperature settings, but OpenAI could seed that to get the same answers with the same input. I’m going out on a limb here with pure speculation but given the model, a temperature, and a known text prompt, OpenAI could probably reverse engineer a seed and prove that the weights are the same.

Biggest roadblock right here. Need a private version for sure.

> What % of people can hold 25,000 words worth of information in their heads, while effectively reasoning with and manipulating it?

In the general case, for arbitrary input, I think the answer to this is clearly 0. At best we can compress the text into a limited embedding with a few salient points stored in long term memory.

How about… 3D spreadsheets, man. You finish tallying up all your credits and debits, and then wham! You’re sprung into another dimension. It’s Google Tensorsheets, man, Microsoft won’t even know what hit them.

You may be comparing what you do in one day to what others do over a longer timeframe. Having three decently involved technical hobbies doesn’t mean you have to do all of them at once. Build momentum over a month or two at least.

Try cocaine! Just kidding, but in all seriousness it sounds like you’re lacking energy and focus, not time or motivation.

There are numerous strategies you can use to increase those. Some include exercise (you’ll be trading time for long term health, energy, and discipline, but it is practically always worth it), meditation (this will help you build that focus muscle and also learn to let go of unhelpful or unrealistic expectations), and various energy management techniques like bifurcating your day into daily maintenance activities (chores, work, food) and contiguous time you set aside just for yourself.

7 hours a night doesn’t sound great. Maybe he’s just bagged from sleep deprivation and caffeine abuse and the hours are whiled away in the default mode network, scrutinizing the ceiling?

Better to work one hour on one thing that’s truly important, and give it your all, than spend 4 hours trying to bang through an endless list of obligations using the Seinfeld method. I speak from experience as someone who tried the latter.

> Maybe try some reading on procrastination, the psychology of it.

It’s a trap!

Moderation is key. On the one hand a high cadence plan/act/review cycle is rocket fuel for personal productivity, and nothing forces you to manage focus and flow as well as blocks in a calendar, but on the other hand you need to space things out, manage energy, and give yourself some room to breathe. Otherwise the parts of your brain that aren’t planning your every waking moment will eventually rebel.

I can attest to time-blocked journaling as having linchpin potential and being one of the greatest foundational habits I have ever adopted.

This reply is gold but I’d flip the second part of 10 on its head; less really is more.

I should add that you need to manage your energy as well as your time but that will follow naturally from a virtuous cycle of day planning and review.

The simple answer is that you don’t find time for everything, and you never will.

Schedule each day in a calendar and put as much effort into planning not to do things as you do into planning to do things. Accept that there’s a season for everything, and not only can you not do it all at once, you can’t do it all, period. And that’s okay. Real achievement comes from sacrifice, not the eager experience hoarding of a dilettante.

Is it really so over the top, though? We have machines that have basically internalized the Internet and can easily pass a Turing test, and yet we still have people trivializing these marvels as mere “parrots”. I believe this is partly a defence mechanism (we humans are inherently hubristic), and if a stubborn attachment to humanity’s “specialness” is the cause, then that certainly mirrors the psychology of Copernicus’s detractors.

In a way, though, the “parrot” moniker is apt. The (long aspirational) Turing Test was originally called the “imitation game”, and what’s better at imitation than a parrot? Apparently, it’s ChatGPT - I never did see a parrot write code.