OpenAI Finally Releases GPT2 1.5 Billion Parameter Model

On Tuesday, OpenAI shared its final staged release of the 1.5 billion parameter GPT-2 language model. All associated code and model weights were included in the blog post. 

When OpenAI first released GPT-2 in February this year, concerns around its potential misuse were voiced. At that point, the company decided to hold back on sharing the language model's code and training dataset. 

After months of looking into GPT-2's details, OpenAI has now decided the time is right to allow further research and understanding into synthetic text. Thus, their decision to share the model's parameters. 

RELATED: OPENAI GPT2 NOW WRITES SCIENTIFIC PAPER ABSTRACTS

What has OpenAI noticed about GPT-2?

It's quite clear as to how a system that creates algorithms to complete an entire text or sentence could be misused. Fake news stories come to mind, among other nefarious uses of AI-generated text.

We're releasing the 1.5billion parameter GPT-2 model as part of our staged release publication strategy.
- GPT-2 output detection model: https://t.co/PX3tbOOOTy
- Research from partners on potential malicious uses: https://t.co/om28yMULL5
- More details: https://t.co/d2JzaENiks pic.twitter.com/O3k28rrE5l

— OpenAI (@OpenAI) November 5, 2019

Withholding their language model's code and training dataset went against what OpenAI believed in. However, for security reasons, they decided to do just that, and they still wanted to find a way of sharing the information.

So, over the past nine months, OpenAI observed GPT-2 to ensure they could share their code and model weights safely. The company made the following observations: 

1. Humans find GPT-2 outputs convincing.
2. GPT-2 can be fine-tuned for misuse.
3. Detection is challenging.
4. We’ve seen no strong evidence of misuse so far.
5. We need standards for studying bias.

Regardless of whether or not a task-specific training dataset was shared, GPT-2 was still able to generate mind-blowingly good machine translation, question answering, reading comprehension, and summarization. 

Upon sharing their GPT-2 model, OpenAI has said they are doing so to "aid the study of research into the detection of synthetic text, although this does let adversaries with access better evade detection."

Can you tell which is an AI-generated text and which isn't?

We trialed the 'Talk to Transfomer' text generator to see for ourselves and were impressed!

The 1.5 billion parameter GPT-2 (aka gpt2-xl) is up:
✅ in the transformers repo: https://t.co/KvUK5V7owl
✅ try it out live in Write With Transformer? https://t.co/R0WHn2WMQt

Coming next:
? Detector model based on RoBERTa

Thanks @OpenAI @Miles_Brundage @jackclarkSF and all

— Hugging Face (@huggingface) November 5, 2019

Can you tell where GPT-2 starts generating text in the following paragraph?

"Pushing any technological fears aside, the newly released full-sized GPT-2 transformer will undoubtedly be a hit with the tech crowd. The new transformer will also be available in a smaller capacity to make space in the larger GPT-1."

We'll give you a hint. It's somewhere on the second line. Astonishingly, without much prompting, GPT-2 offered an entire, and quite specific text, which carried on as follows:

"The new GPT-2 transformer has 4 pins (instead of 5), and a lower resistance at 1,400 ohms vs. 1,500 ohms, making it compatible with all standard audio gear. I have heard whispers that the newer GPT-2 will be able to provide the same amount of current as the GPT-1. I do not have any info on the new GPT-2 transformer, so I cannot confirm or deny that statement."

See for yourself what the GPT-2 transformer can do here.