Hacker News Reader

Comment by smackeyacky | Hacker News Reader

In some ways, this is a kind of "no shit, sherlock" moment that all machine learning faces. It becomes a dead end, feedback loop like holding a microphone too close to a speaker.

Funnily enough, genomic powered genetics in livestock production has the same characteristics. Without the flow of new phenotypic data the genomic models rot rapidly and become useless for prediction.

Predictive language models powered by older predictive language models will quickly become nonsense.

You have to remember just how deceptive something simple as Eliza was for people who didn't know how it worked.

View on HN

Changes to language are smaller. GPT-4 has a cutoff date in 2021 and is plenty smart and useful. It just doesn't know about the war in Ukraine.

The war in Ukraine started in 2014.

This is a form of pedantry that people consider insufferable. The reason is because everyone is well aware that the Ukraine was in conflict with russia since 2014, and even before really. Imagine your irritation at someone pointing that out, when you already know it also. We are all comfortable calling the recent full scale invasion "The war".

Before 2022 it was annexation and supplying separatists in Donbas. And sending some special ops. It was a conflict, but not

The same way Europe is in conflict with Russia now - we send supplies and all the support we can do without starting an actual war.

I love in Poland, spoke with many Ukrainians, and these seem to be commonly accepted definitions.

> Before 2022 it was annexation and supplying separatists in Donbas.

Invasion, occupation, and annexation of a large part of another country is war by every measure, even ignoring Donbas.

If it were, we would be saying that WWII started in 1938, not 1939

This is more inflammatory than it is informative.

The parent commenter is correct. You're the one mistaken.

https://en.wikipedia.org/wiki/Russo-Ukrainian_War

How so? Inflammatory would mean it's either irrelevant, unfair or harmful in some way. Stating a correction to a fact is hardly either.

If someone stated that the attack on Pearl Harbor happened in 1945 and someone else corrected them that it happened in 1941, is that inflammatory?

Edit: Maybe you're referring to it being called a "war" in the first place is inflammatory? I'd give you that, and we'd be better off calling it a conflict as none of the parties formally declared war, but then we start veering into semantics instead...

that goes against the narrative

There are at least two things that make information wrong. One is "right, but not right now" i.e time. The right answer delivered late is wrong c.f. real time systems.

The other is "garbage in, garbage out" which is related but a bit wider.

While these models cannot distinguish input from older crap models, you cannot consider their output reliably right.

I have high hopes in dataset engineering. Scaling the model is already reaching the limits of hardware and funding, the training corpus is almost all the text written on the internet (by some accounts just 10% of it, so maybe we got a couple more years left there).

So then what can we do? We turn to the quality of the data. Input examples can be enriched, facts can be checked for support. Based on existing corpus of text we can generate new training examples. Some of them, like code, can also contain execution data and tests.

Information is not just the public internet. Every text message, school paper, phone call, and recording of an individual could be future input. Then expand that to photos and videos. Then consume all of YouTube.

Each of these sources is still rapidly growing. We are far from drying up the well of the human experience.

We are a long way from capturing the richness of human experience. Your assumption is that everybody records everything all the time but they don't. Like linkedin never has anybody failing at anything.

It's going to end up a simulation of what people think other people want to hear, rather than what actually happened. The experiment has already failed because the assumption that the internet has captured everything is wrong.

Asserting that all our audio won't be captured is yet another assumption. The amount of data being collected is ever increasing, and it is conceivable that near unlimited microphone/camera access and storage could lead to this in the near future.

Either way, it isn't static or decreasing.

> It just doesn't know about the war in Ukraine.

Neither does most of Russia for that matter.

They do, and they (russians) mostly support it. It's not putin's war.

Maybe the next step in ML will be teaching them how to forget.

I think continual learning (and incidentally curriculum learning) are going to be some of the more important subtopics within deep learning pretty soon. These things seem under-researched to me right now, despite the fact it's a space where a small academic lab could still compete.

I think part of the problem is most papers are supposed to show an engineering feat or a proven mathematical result. Not enough room for developing knowledge on how networks learn scientifically IMO.

Reply on news.ycombinator.com