A robot is taught to laugh at jokes by scientists

Robots continue to embrace more human characteristics day by day. Remember Sophia, for example, which was produced by Hong Kong-based Hanson Robotics and impressed us all with her answers.

Now, a group of scientists from Kyoto University are developing an AI system that enacts humor- by laughing at the right place and at the right time.

It is stated that scientists who created Erica the laughing robot claim that the method might enhance regular talks between humans and AI systems.

The results were published in Frontiers in Robotics and AI today.

“We think that one of the important functions of conversational AI is empathy,” said Dr. Koji Inoue of Kyoto University, the lead author of the research.

“So we decided that one way a robot can empathize with users is to share their laughter," Inoue also added.

How was the process?

More than 80 speed-dating conversations took place between male university students and the robot, in which four female amateur actresses initially teleoperated. This provided the data to train Erica.

The duration of each dialogue was 10 to 15 minutes, and 82 dialogue sessions were conducted. Scientists recorded these dialogues with multi-modal sensors, including a 16-channel microphone array and HD cameras.

In the shared-laughter model, an AI system will laugh as an empathetic response after hearing a human laugh at first. To implement this strategy, three subsystems had to be designed: one to recognize laughter, another to choose whether to laugh, and a third to select the right kind of laughter.

“Our biggest challenge in this work was identifying the actual cases of shared laughter, which isn’t easy, because as you know, most laughter is actually not shared at all,” Inoue said.

“We had to carefully categorize exactly which laughs we could use for our analysis and not just assume that any laugh can be responded to.”

"Not funny at all..."

The team evaluated Erica's "sense of humor." When all the conversations were examined, it was observed that Erica did not find every joke funny and did not laugh.

In addition, the team is planning to enhance the frame of the study. “We think that they can show this through their conversational behaviours, such as laughing, eye gaze, gestures and speaking style,” said Inoue.

Abstract:

Spoken dialogue systems must be able to express empathy to achieve natural interaction with human users. However, laughter generation requires a high level of dialogue understanding. Thus, implementing laughter in existing systems, such as in conversational robots, has been challenging. As a first step toward solving this problem, rather than generating laughter from user dialogue, we focus on “shared laughter,” where a user laughs using either solo or speech laughs (initial laugh), and the system laughs in turn (response laugh). The proposed system consists of three models: 1) initial laugh detection, 2) shared laughter prediction, and 3) laugh type selection. We trained each model using a human-robot speed dating dialogue corpus. For the first model, a recurrent neural network was applied, and the detection performance achieved an F1 score of 82.6%. The second model used the acoustic and prosodic features of the initial laugh and achieved a prediction accuracy above that of the random prediction. The third model selects the type of system’s response to laughter as a social or mirthful laugh based on the same features as the initial laugh. We then implemented the full shared laughter generation system in an attentive listening dialogue system and conducted a dialogue listening experiment. The proposed system improved the impression of the dialogue system such as empathy perception compared to a naive baseline without laughter and a reactive system that always responded with only social laughs. We propose that our system can be used for situated robot interaction and also emphasize the need for integrating proper empathetic laughs into conversational robots and agents.