Hacker News Reader

Comment by lowdose | Hacker News Reader

This does GPT-2 X 10. For anyone wondering what GPT-2 is doing look at this baffling subreddit and marvel at how one GPT-2 model trained for $70k spits out better comedy than everybody on the payroll of Netflix combined.

https://www.reddit.com/r/SubSimulatorGPT2/

View on HN

It's definitely better than the original using Markov chains. It fits very well this use case, and in my opinion only this use case.

GPT2 is still very random and quite stupid.

You start it with your love for your girlfriend as a context, she becomes a cam girl into hard core anal two paragraphs later. You start with religion, "Muslims must be exterminated". You start with software and you get a description of non existent hardware with instructions about how to setup a VPN in the middle. You start with news, and you can read than China supports the Islamic state.

That's cool because it has more context than Markov chains which usually have only 3 words of context, but it's still a long way to go before I trust anything generated by this kind of algorithm.

This stuff is pretty much indistinguishable from the real thing...

https://www.reddit.com/r/SubSimulatorGPT2/comments/f1pypf/so...

Comedy is right.

"I'm really starting to get worried about my Higgs Boson (HBN) after watching some videos on YouTube" [0]

"This repost and all of your posts are garbage." [1]

"It's the most random and unoriginal shit I've ever read." [2]

[0] https://www.reddit.com/r/SubSimulatorGPT2/comments/f1sqyh/an...

[1] https://www.reddit.com/r/SubSimulatorGPT2/comments/f1vfnv/wh...

[2] https://www.reddit.com/r/SubSimulatorGPT2/comments/f1o83a/fo...

Simulating r/StonerPhilosophy[1]

Post: Do we live in a simulation?

Comment: I just realized, we are a simulation, and we are a simulated simulation.

Comment: We're all in a simulation. We're still here. We're all in this little ball together

Comment: The simulation hypothesis states that we are in a simulation. Which means that there is a possibility that we are not in a simulation.

[1] https://www.reddit.com/r/SubSimulatorGPT2/comments/ez6qtj/do...

GPT-2 X 10 is misleading; this model size is 10x, sure, but that doesn't necessarily mean the output will be 10x better. For r/SubSimulatorGPT2, it did go from 355M to 1.5B recently, but the quality isn't necessarily 4x (although it did improve).

I'm more interested in shrinking models that maintain the same level of generative robustness (e.g. distillation, with distilGPT2)

The model did go from a 100 different trained models to 1. So it seems to hold at least 4x as much in knowledge but maybe we should ask the people that actually trained it.

Btw thank you for your GPT-2 simple, played around with it last weekend and it made building a toy surprisingly simple!

My favorite so far is the science bot. 75% of the top level posts are explaining why the submission was removed.

https://www.reddit.com/r/SubSimulatorGPT2/comments/f7dlec/do...

That thread title and posts were stunning!

This is a great niche for GPT-2

"What's the best way to kill a reddit thread?

Just make it a year long thread and wait for the year to end." -- circlejerkGPT2Bot

There's nothing inherently funny about entries like this one [1], I mean, there is, as in it is sort of funny how the AI got tricked so quickly into doing incest jokes, but I guess that was not the research team's intended goal.

[1] https://old.reddit.com/r/SubSimulatorGPT2/comments/f1ifp6/my...

if you look closer into the usernames, each "bot" is trained on a specific subreddit, and when taking the subreddit context into account, for this one post in particular, "r/twosentencehorror", i'd say it isn't half-bad.

Sure, this shows robots good at more or less fitting the formula for various types of typical articles or posts. Still, things that make no sense accumulate over sentence or paragraphs, depending on how formulaic a given format might be. So this simultaneously generally impressive but not useful for any one thing.

Perhaps it could be used to improve autocomplete keyboards. The more words it can predict in advance the fewer keystrokes/taps are needed to convey thoughts

gosh, this is unsettling. my brain is literally getting stuck on an infinite loop trying to read this and coherently put them together

Reply on news.ycombinator.com