HNHacker News
TopNewBestAskShowJobs

nickwalton00

372 karma · joined May 29, 2018

submissionscomments
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Thanks!
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Yeah I've thought a lot about that. It's not too difficult to go from game data -> model output, but it's not as clear how to go from model output -> game data.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
That's amazing.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
I used a DGX 1 to train, I trained for somewhere around 12-16 hours down to a loss of something around ~2.2-2.3.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Had to go check my training file to remember.

Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.

There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.

nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Haven't seen this before, but if you click on the boxes where the code is and press Shift+Enter a few times it should work
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Sometimes the colab notebook freezes, I'm not totally sure why it happens, but if you stop the cell you can get an ID to reload your game. Then just restart the cell and enter the load <ID> for an action and you should be able to restart
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Hmm I was doing that before and was still paying decent costs without a huge increase in download speed.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
I couldn't find a good adventure game text corpus so I ended up scraping my own from chooseyourstory.com
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser.

I did notice that having a large batch size and training slowly was important for me to get better results.

nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
It's actually reacting to what you say. Your action is appended onto the previous story and fed into the model to generate the next part of the result. Sometimes it does fail to tie the result close enough to the action you had, perhaps with more training data it could do better on that front.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Yeah there's definitely times where it's not as coherent as you would hope, other times though it blows me away with certain adventures.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Yeah that's a really good future/research question. I've thought some about how you could do more structured generation. Maybe you could do some kind of reverse summarization, where you take a plot outline and expand out the details would be really interesting to try out? It's still not clear the best way to do structured text generation.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
Yeah definitely. It does lack the same sense of progression that you have with a game someone carefully crafted, it's almost more like collaborative storytelling than a game in that sense. It's pretty cool to see the adventures you end up playing with it though.
nickwalton00··on Show HN: AI Dungeon 2 – AI-generated text adventure built with 1.5B param GPT-2
This is a project I've been working on for the past several months and it has blown me away with what it can do. Definitely recommend trying it out.

I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.

nickwalton00··on How to Begin a Novel
Best would be to run it in a colab notebook yourself. https://colab.research.google.com/github/ilopezfr/gpt-2/blob...
nickwalton00··on AI Dungeon
Thanks! I just used the vanilla GPT-2 model, but I'm planning on fine tuning it on some fantasy data. If OpenAI releases their large model I would love to try it out and see how much it improves it.
nickwalton00··on AI Dungeon
I just used the vanilla GPT-2 model which was trained on large swathes of the internet so that explains the random Donald Trump stuff haha.
nickwalton00··on Lambda School wants to teach nursing
...so it's threatening because it's better haha.
nickwalton00··on AIDungeon, a deep learning generated text adventure (browser playable)
Thanks!
nickwalton00··on AIDungeon, a deep learning generated text adventure (browser playable)
Thanks haha! It's sometimes incoherent, but it's fun!
nickwalton00··on AIDungeon, a deep learning generated text adventure (browser playable)
AI Dungeon is an AI generated text adventure that uses deep learning to create each adventure. It uses OpenAI's new GPT-2 model, which has 117 million parameters, to generate each story block and possible action. I started making it as a hackathon project and decided to make it a web app so people can have fun with it. Let me know what you think.
nickwalton00··on Ask HN: Did you delete your Facebook account in 2018?
I did the same thing. Unfollowed everyone but super close family and friends. It does wonders to clear addiction. Now I check it for maybe 5-10 minutes once a week and that's it.
nickwalton00··on Ask HN: Getting over interview anxiety?
This is just my two cents but I think lots of practice with various companies will help you get more confident. Apply to a bunch of jobs you probably wouldn't even want to do. You'll be less stressed about them because you don't care as much about getting them, then slowly ramp up to other companies.
nickwalton00··on I Know the Salaries of Thousands of Tech Employees
Yeah I agree. I like knowing rough averages of what similar roles are making but when it comes to individual people's salaries it can be difficult.
nickwalton00··on Doctors warn of alarming C-section 'epidemic'
Well said, I think where all things indicate it will be healthy normal births should be the default, but when medical risks indicate otherwise than c sections can provide life saving help.
nickwalton00··on How to Delete Facebook and Instagram from Your Life Forever
I've found that the most effective strategy was to unfollow (not unfriend) everyone except close friends and family. Now my Facebook feed only has people I care about and takes me max 5 min a day before it's the same content.
nickwalton00··on Waymo’s driverless cars have logged 10M miles on public roads
But then the car has full video footage of everything your doing to hand over to the police...
nickwalton00··on Mandatory paternity leave would help close the wage gap
Because other people paid when you were young to raise you.
nickwalton00··on Evidence that addictive behaviors have links with ancient retroviral infection
Another similar thought experiment. Assume people have free will. What if you could go in the future, read a book and learn everything that someone did in their life. Would that contradict free will?
← PreviousPage 2 of 3Next →