ChatGPT knows 25 jokes and will repeat them like a broken record

Researchers say ChatGPT cannot create "funny original content."
Sejal Sharma
Representational image
Representational image


ChatGPT is fun but not funny, says a team of researchers who sought to understand the OpenAI chatbot’s ability to grasp and reproduce human humor through prompt-based experiments.

The team noticed that ChatGPT tends to repeat the same joke frequently. They were able to identify a set of about 25 jokes which were reproduced from either the training data or were hard coded in a predefined list. These observations by the German researchers led to the hypothesis that jokes weren’t originally generated by the model.

The researchers wrote in their study: “It cannot yet confidently create intentionally funny original content. The observations of this study illustrate how ChatGPT rather learned a specific joke pattern instead of being able to be actually funny.”

The jokes were strikingly sophisticated

ChatGPT isn’t open-sourced, so the team had to resort to prompt-based experiments. To test how rich the variety of ChatGPT’s jokes was, they asked it to tell a joke a thousand times. Almost all the output contained exactly one joke. But, the prompt ‘Do you know any good jokes?’ led to multiple responses, leading the team to record 1,008 responded jokes in total. All responses were grammatically correct.

Here’s a list of the top most frequent jokes ChatGPT spouted out:

  • Why did the scarecrow win an award? Because he was outstanding in his field.
  • Why did the tomato turn red? Because it saw the salad dressing.
  • Why was the math book sad? Because it had too many problems.
  • Why don’t scientists trust atoms? Because they make up everything.
  • Why did the cookie go to the doctor? Because it was feeling crumbly.

The study comes with limitations

The researchers noted that there were two major limitations to the study. The first is that humor is subjective. Hence, a reliable valuation is hard. Say, if ChatGPT were to generate a joke about Bill Gates, it’s totally up to the reader’s interpretation whether they find it funny or not. One could argue that the chatbot has a different humor than the reader.

The second limitation is that ChatGPT’s output cannot be traced back to specific input data. Since ChatGPT isn’t an open-sourced large language model, the observations of the team are solely based on system outputs.

The researchers acknowledged that ChatGPT has garnered much attention ever since it was launched in November 2022. However, the team wrote in their study, “The system fails to meet the requirements of open science, as data, training details, and model characteristics are kept private. We, therefore, consider our work an important contribution to understanding ChatGPT’s capabilities and objectively highlight its potential and limitations.”

The study was published on the pre-print server arXiv.

Study abstract:

Humor is a central aspect of human communication that has not been solved for artificial agents so far. Large language models (LLMs) are increasingly able to capture implicit and contextual information. Especially, OpenAI’s ChatGPT recently gained immense public attention. The GPT3-based model almost seems to communicate on a human level and can even tell jokes. Humor is an essential component of human communication. But is ChatGPT really funny?

We put ChatGPT’s sense of humor to the test. In a series of exploratory experiments around jokes, i.e., generation, explanation, and detection, we seek to understand ChatGPT’s capability to grasp and reproduce human humor. Since the model itself is not accessible, we applied prompt-based experiments.

Our empirical evidence indicates that jokes are not hard-coded but mostly also not newly generated by the model. Over 90% of 1008 generated jokes were the same 25 Jokes. The system accurately explains valid jokes but also comes up with fictional explanations for invalid jokes. Joke-typical characteristics can mislead Chat-GPT in the classification of jokes. ChatGPT has not solved computational humor yet but it can be a big leap toward “funny” machines.

Add Interesting Engineering to your Google News feed.
Add Interesting Engineering to your Google News feed.
message circleSHOW COMMENT (1)chevron
Job Board