372 karma · joined May 29, 2018
Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.
There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.
I did notice that having a large batch size and training slowly was important for me to get better results.
I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.