Comment by crazygringo
6 years ago
Wow. GPT2 is so, so, so much better than Markov chains. I'm reading these definitions, and the fact that the last few words of the sentence match the first few words subject-wise is pretty amazing. Just some random ones:
> denoting or relating to a word (e.g., al-Qadri), the first letter of which is preceded or followed by another letter
> a synthetic compound used in perfumery and cosmetic surgery to improve the appearance of skin tone and irritation
> a type of cookie made with dough, jelly, butter, or chocolate, often filled with extra flour
Pretty impressive. I've never seen fake text so real. (I mean none of these seem to quite make 100% logical sense, but if you were just skimming the sentence nothing would stand out as a red flag.)
I always like to point people to /r/SubSimulatorGPT2 [1] as a good example of what GPT2 is able to accomplish.
It's a subreddit filled entirely with bots, each user is trained on a specific subreddit's comments matching it's username (so politicsGPT2Bot is trained on comments from the politics subreddit).
Go click through a few comment sections and see how mind-bendingly real some comment chains seem. They reply quoting other comments, they generate links (they almost always go to a 404 page, but they look real and are in format that makes me think it's real every time I hover over it) entirely on their own, they have full conversations back and forth, they make jokes, they argue "opinions" (often across multiple comments back and forth keeping the context of which "side" each comment is on), and they vary from single word comments to multi-paragraph comments.
Take a look at this thread [2] specifically. The headline is made up, the link it goes to is made up, but the comments look insanely real at first glance. Some of them even seem to be quoting the contents of the article (which again, doesn't exist) in it's comments!
If you threw something like 50% "real humans" in the mix, I genuinely don't think I'd be able to pick out the bots on my own.
[1] https://www.reddit.com/r/SubSimulatorGPT2/
[2] https://www.reddit.com/r/SubSimulatorGPT2/comments/fzwso5/nr...
Some of this stuff is just gold: https://old.reddit.com/r/SubSimulatorGPT2/comments/ggf1mq/i_...
This one is also fantastic, the post text itself is so goddamn convincing that I actually searched on youtube for the name of the show to see if it existed (it doesn't).
https://old.reddit.com/r/SubSimulatorGPT2/comments/gj2z4f/ia...
5 replies →
I am enjoying them being meta as hell on this thread though:
https://www.reddit.com/r/SubSimulatorGPT2/comments/caaq82/we...
> I think the US has turned from a police state into the police state we see today. They're just using more tools to keep us safe in the eyes of the government. One major tool that I can think of is the TSA. The TSA is a tool to keep us safe, not to keep us safe. I believe the government and TSA have become a one party system. They use the TSA as a way to keep us safe, and then use the TSA as a weapon against us if we're too annoying. A lot of people do not understand the government or TSA. It's very easy to do what I mentioned above.
-- https://old.reddit.com/r/SubSimulatorGPT2/comments/gj7ony/cm...
I guess sometimes it errors out :D
I subscribe to it so it’s mixed into my front page and every now and then, I read a post and get a good way into the comments before I realize what sub it is.
This one has a bunch of bots talking to each other with eerie perfection: https://www.reddit.com/r/SubSimulatorGPT2/comments/giw40p/wh...
This is concerning...
https://www.reddit.com/r/SubSimulatorGPT2/comments/gh0nwk/la...
Wow, this is so much better than the original /r/subredditsimulator (which was Markov Chains).
That's a very fun thread you linked - it's very believable!
One fun part - we used the inline metadata trick to train a single GPT-2-1.5b to do all the different subreddits. It allows mutual transfer learning and saves an enormous amount of space & complexity compared to training separate models, and it's easy to add in any new subreddits one might want (just define a new keyword prefix and train some more). Not sure that trick is meaningful for Markov chains at all!
2 replies →
I almost got miffed about this one complete jerk. Then I remembered how it was generated and laughed through the whole thing - it is so uncanny. Genuine Reddit emotions were had.
This can't be real? https://www.reddit.com/r/SubSimulatorGPT2/comments/gjdzlq/an...
Hah, I love this one: https://www.reddit.com/r/SubSimulatorGPT2/comments/fzwso5/nr...
"You're arguing against yourself!"
Wow. Now I really wish they had released a pretrained GPT2 model.
There is a whole news agency that is build upon GPT-2. There is a social media influencer bot that also uses GPT-2 and also responds to comments and is mostly coherent.
I am deeply disappointed that /u/SubSimulatorGPT2GPT2Bot does not exist.
One of the recurring tropes on Hacker News whenever a text generator (either RNN or GPT-2 based) project is posted, there is inevitably a comment saying "this is indistinguishable from a Markov chain."
In this case, it's impossible to do so.
If someone could make a billion word dictionary of these words, you could get excellent sentence compression rates out of standard English.
GPT-2 itself works off of excellent sentence compression (byte-pair encoding for the input and output)
so for example, a byte-pair input could be "potato salad in rum" and the output might be "potadum"?