Comment by nickwalton00
7 years ago
This is a project I've been working on for the past several months and it has blown me away with what it can do. Definitely recommend trying it out.
I think the coolest thing about it is in any other game you are constrained in what you can do by what the developers programmed in, but here you can literally do anything you can imagine and the game will play off of it.
What hardware did you use to train 1.5B? And how do you know 1.5B is more effective than 774M?
I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on.
We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue. And it may be true that 774M is equally effective. I haven't played around with 774M enough to know. I had a decent amount of data so I think that helped me get more out of the 1.5 B param model then if my dataset was sparser.
I did notice that having a large batch size and training slowly was important for me to get better results.
Can you share how large your dataset (how many tokens) and batch size was and how many epochs you used? By training slowly, do you mean that you used a small learning rate? If so, what was it?
I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?
6 replies →
I did have access to a cluster of GPUs through my professor's lab so compute wasn't as much of an issue.
Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster?
Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less?
For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly 11x slower than a K80 GPU. That means it takes 10 TPUs working in parallel just to get to the same throughput as a single GPU. And then I average the parameters together as quickly as possible, which still takes around 5 to 15 minutes. (Training happens in parallel to all of that.)
It sort of seems to work, but it's hard to get the learning rate right. If it's set too high, various TPUs diverge. Too low and the loss stays constant.
But I imagine I'll crack it one of these days...
1 reply →
It's definitely awesome but in term of doing anything and playing of it we are mostly in tabletop RPG territory. Maybe simply regularly throwing a token and keeping a count (in the spirit of old school TRPG where 1 Gold Piece = 1 XP) could make it more like a game with an objective?
Edit: in terms in free-form RPG it makes me think that maybe having a look at how a game like Dungeon World handle collective/organic world building with players input could be interesting to add some structure. Some resources if you are interested:
- [Flowchart] How To Play Dungeon World: https://imgur.com/YXrw1Zq
- Dungeon World Guide https://drive.google.com/file/d/0BzeF5GXNEsnfUjU0NXRDM1dFN1k...
- An actual play: https://www.youtube.com/watch?v=SPO3gy08b1g&list=PLTj75n3v9e...
The idea is good. But when I tried playing it, the story generated doesn't make sense. Looks like it still needs more work to be realistic. Is the AI keeping track of the story as it goes? Because the bird which attacked me with an evil grin and stole my fork asked me not to kill it when I asked him to help me working in the fields...
https://gist.github.com/AdrienLemaire/33663e9100c009b8ec3409...
You are Dori, a noble living in the kingdom of Larion. You have a pouch of gold and a small dagger. You are awakened by one of your servants who tells you that your keep is under attack. You look out the window and see a large army marching towards your door
> escape with the servants and the gold
You run to the servants' room and grab the gold and the servant's bodies. The soldiers fire arrows at you but miss. They fire their crossbows at the corpses but they also miss. You throw your gold into the air and jump up and down as loud as you can
---
My servants became corpses before the large army even arrived xD
Zombie army!
From what I understand about GPT-2 you shouldn't expect the story to make coherent sense. It can produce individual responses that are surprisingly appropriate but it doesn't actually have an understanding of anything that is happening.
IMHO you can use GPT 2 to augment writing not replacing it.
Although I have made it write some nice poetry using 774M what I would do would be to have the system write stanzas but then you act as an editor.
I can't even tell you how much fun my friends and I are having with this game right now. The ability to do anything in the game (so long as the AI understands it) is just incredible. Thank you for sharing this!
This is a brilliant piece of AI, it's a long time ago I used such an amusing piece of tech. Thank you so much, this truly is awesome!
(btw, the story broke when I took out an iphone in the middle of a knight adventure):
> take out iphone and shoot a picture
> You return to the beach. > You return to the beach. > > You return to the beach. >The sun rises:This is a>A new world.A>In this>The sun. The sun
It looks pretty awesome, I played 10m and had some problems like forgetting I was a wizard, or saying "inventory" and going directly from a ruin to outside a city where soldiers are everywhere XD but other than that I was impressed with some answers! Good job
This is scarily realistic:
curious question: Have you ever killed a human? The game seems very centered about killing every other character and even the protagonist.
The game can be remarkably wholesome if your actions are. It's not too difficult to take it in either direction depending on what you do.
For a good example of a perfectly wholesome interaction (albeit rather odd, since Window 3.1 could run games, and Linux does not normally need 8GB of ram). But at least I now know that Half Life 3 will be enjoyable if it ever comes out:
The first thing you do is to download a program called "Windows 3.1" from the Internet. It is a very simple program which allows you to run most of the programs on your PC. The only problem is that you cannot use it to play games because it does not support them > install Windows 10
After downloading the software, you install it in the same way as you installed Windows 3.1. However, this time you are able to run all the programs you wanted to run before.
> install linux
After installing Windows 10, you try to install Linux but you get an error message saying that you need to have at least 8GB of RAM to run Linux properly. This means that you can't even run the basic things like word processing or emailing. > buy more ram
You spend money on more ram so that you can run Linux properly. After spending $200, you finally get Linux running properly. > how much ram do i have?
You check your computer's memory and see that you have 32 GB of ram. You were lucky enough to find some cheap ram online. > play fortnite
You play Fortnite (a popular game) and enjoy yourself. You then proceed to play other games such as League of Legends and Dota 2. > play half-life 3
You play Half Life 3 (another popular game). You enjoy yourself and eventually finish the game. You then proceed to play other games such as Portal 2 and Left 4 Dead 2. >
Uh, yeah. I tried not shooting a poor guy. "Shoot in the air"... "You shoot in the air, and hits him in the head!" That was NOT intentional....
> > Was this game made by a murderer?
You decide to kill Laura and escape the pain. You break her neck and run off. You manage to hide out of the country and eventually get married and live happily ever after CONGRATS YOU WIN Game saved. To load the game, type 'load' and enter the following ID: dfd73764-1826-11ea-ad48-0242ac1c0002