williamtrask

there's really no evidence of that (in the article or otherwise)

Related papers:

- https://arxiv.org/abs/2307.06962

- https://arxiv.org/abs/2309.12288

- https://arxiv.org/abs/2112.04426

- https://huggingface.co/papers/2405.14860

You seem to be focusing a lot on remembering or forgetting consequences. Yes, ensuring models know enough about the world to only cause the consequences they desire is a good way for models to not create random harm. This is probably a good thing.

However, there are many other reasons why you might want a neural network to provably forget something. The main reason has to do with structuring an AGI's power. Even though the simple-story of AGI is something like "make it super powerful, general, and value aligned and humanity will prosper". However, the reality is more nuanced. Sometimes you want a model to be selectively not powerful as a part of managing value mis-alignment in practice.

To pick a trivial example, you might want a model to enter your password in some app one time, but not remember the password long term. You might want it to use and then provably forget your password so that it can't use your password in the future without your consent.

This isn't something that's reliably doable with humans. If you give them your password, they have it — you can't get it back. This is the point at which we'll have the option to pursue the imitation of living creatures blindly, or choose to turn away from a blind adherence to the AI/AGI story. Just like we reached the point at which we decided whether flying planes should have flapping wings dogmatically — or whether we should pursue the more economically and politically competitive thing. Planes don't flap their wings, and AI/AGI will be able to provably forget things. And that's actually the better path.

A recent work co-authors and I published related to this: https://arxiv.org/pdf/2012.08347

Because we can get AI related technologies to do things living creatures can’t, like provably forget things. And when it benefits us, we should.

Personal opinion, but I think AGI is a good heuristic to build against but in the end we’ll pivot away. Sort of like how birds were a good heuristic for human flight, but modern planes don’t flap their wings and greatly exceed bird capabilities in many ways.

Attribution for every prediction and deletion seem like prime examples of things which would break the analogy of AI/AGI with something more economically and politically compelling/competitive.

Or to the articles point, it’s like the hysteria around cigarettes - and the research is still in early days.

"Any organization embedded in a larger organization or community needs to invest to a certain degree in establishing the right kind of social relationships to maintain this embeddedness. Incentives must be aligned, and both parties must also recognize this alignment. We did not invest enough in university politics and sociality to form a long-term stable relationship with our faculty. There also needs to be an understanding of how to communicate across organizational communities. When epistemic and communicative practices diverge too much, misunderstandings proliferate. Several times we made serious missteps in our communications with other parts of the university because we misunderstood how the message would be received. Finding friendly local translators and bridgebuilders is important."

https://static1.squarespace.com/static/660e95991cf0293c2463b...

I miss the word2vec days…a simpler time.

Honestly if the word AGI experiences some term inflation that wouldn’t be the worst thing.

Fwiw - I wrote a paper documenting LLM (and neural) regression a few years ago (Neurips 2018)

https://arxiv.org/abs/1808.00508

They can interpolate but extrapolation is hard because of the constraints from nonlinearities

Interesting direction! Let's assume the citation is me. Does that make content generation harder than verification (judgement)?

It is a promising idea, but it falls prey to the same tautology as in the game-playing agent days. In order for simulation to be useful, you need a really robust model of the world, but if you had a really robust model of the world, you wouldn't need the simulator. Simulation is really really good for one thing: learning a policy that can find a particular corner of the search space as fast as possible (such as a wining state in a Go game). But simulation is not good at actually generating the space. It's going to extrapolate mistakes really badly.

Also, while humans are better at judging (look at that fake thing!) than generating (drawing a realistic photo), I think you may find that — despite our abilities — detection is actually quite a bit harder. As an example:

"John Doe is dead."

This was very easy for me to create, but it's quite difficult for you to judge whether or not it is true due to a variety of factors (which John Doe, am I being honest, when was the last time you saw John Doe, do you know anyone who knows John doe and could check, perhaps John Doe had a twin who died, etc.)

So this is a good idea because it can create vastly more training data for a model to learn from. However, it seems likely that these models are going to hallucinate like crazy. As featured in the documentary, the AlphaGo program struggled with hallucination, and it performed self-play in a tiny world based on a perfectly rigid and exactly correct set of rules. LLMs already have tons of false corners and edges in their logic about the world, and this seems like it has the potential to spread those all around.

Hard to say — these things can be difficult to predict. I can see this working but there'll probably be some ratio of training data - self-play that we have a hard time getting past because it's a difficult-to-control form of extrapolation.

> “Just because mistake is common”

A good example of why having a language that is squishy to common errors is valuable. I still understood your point even though it’s grammar wouldn’t compile.

Machine learning is a massive counter example to this trend. Specialised deep learning hardware is just supporting an even more general GPT.

Devils advocate, if the post was about building an open source non-profit with that level of financial success I don’t think there would be so much salt.

Fwiw I don’t think those are “the great entrepreneurs”. They’re rich and famous, yes, but “THE great entrepreneurs”?. I’d prefer Tesla, Jobs, and even JCR Licklider over them.

To be fair. The title of the post only focuses on money and completely ignores whether the work has any real value to themself or society. Kindof implies pretty strongly “their identity”.

Care to comment on the idea that $ is over celebrated as a metric of success? That seemed like a bigger point.

There’s quite a few “if”s here that imply those at the bottom are able to partake in the financial maneuverings of those at the top.

Then they should upgrade their grid? What’s your point?

I don’t know why but that’s the funniest HN comment I’ve ever seen. I’m still chuckling. Took me a minute to figure it out. xD

More or less the same experience here. I also think it prepared me pretty well for WFH (which I now absolutely love).

> people have completely given up on society and said fuck it, we'll make our own.

Fwiw, this type of thinking played a pretty strong role in how America began (and in other social/moral movements). It's not always a bad thing.

> This is sad. The real solution is not to homeschool. It is to force ALL kids, regardless of age, location, and income to be forced into the same school system. Want to improve things? gotta make it better for everyone.

I get your point, but just to continue my analogy, King George might have said something similar in the late 1700s. A good general counter-narrative to this line of thinking is James Scott's "Seeing like a state" which documents the degree to which governments regiment society in ways that reduce social welfare.

...huh... there's an idea

Would just routing the data via starlink be good enough? (if you could hide the original location)

For a price…

We might not be able to force a cure, but I can think of a multi trillion dollar industry (or two) that’s undoubtedly pushing the opposite. Maybe we should start there.

True in the limit yet democracy and free markets both imply popularity has something to it.

Anyone reproduced that output? Seems a little bit picturesque given this paper.

> Less waste is almost always more profitable

…something something 3M …something something oil industry …something something tobacco industry, pesticides, fast fashion, disposable products, and factory pollution

The cost of fake reviews on society is subtle but may be high. And it’s too large of a scale for FTC to realistically pursue everyone. I suspect it needs to sound big and scary to work.