OpenAI's GPT2 Now Writes Scientific Paper Abstracts

A string of tweets demonstrates the transformer neural network's incredible capabilities.
Fabienne Lang

OpenAI created GPT2 earlier this year — which is a "large scale unsupervised language model which generates coherent paragraphs of text," as per OpenAI's blog page. This incredible transformer neural network generates whole paragraphs one word at a time. 

In a fun and intriguing turn of events, Dr. James Howard, a cardiology trainee at Imperial College in London, U.K., decided to test GPT2's scientific abilities. 

Dr. Howard decided to prompt GPT2 with random scientific titles and watched as it wrote abstracts right in front of his eyes. Dr. Howard then shared his responses via Twitter.


Here's what GPT2 was able to create

Dr. Howard re-trained GPT2 on the Pubmed/MEDLINE database — a scientific database with over 30 million biomedical literature citations. This means that when Dr. Howard provided his scientific titles, the transformer neural network was able to respond in scientific terms. 

It took Dr. Howard around 24 hours to re-train GPT2 in this way. 

What Dr. Howard, unbelievably, received in response were concise medical abstracts that were well structured and thought-provoking. Below are some of the abstracts for your perusal. 

A word of warning from Dr. Howard:

The first abstract: 

The second one:

The third one: 

GPT2 just keeps giving: 

It's quite incredible — if perhaps a little worrying — what OpenAI's transformer was able to do, and how it could be re-trained in such a short time.

Dr. Howard was very open and transparent about how this intriguing exchange happened, and you can try for yourselves here where you 'talk' to the transformer yourselves. 

Most Popular
message circleSHOW COMMENT (1)chevron