In the past, robots, particularly industrial robots, were required to enhance work efficiency and increase productivity by means of processing capacity that surpasses that of human operators.
Today, robots can do much more than simply performing tasks as commanded by a human operator. Communication robots used for serving customers or care-giving as well as the artificial intelligence incorporated into smartphones have made our lifestyles more convenient while letting us enjoy communication with machines.
What will the future human-machine relationship beyond this collaborative relationship look like? OMRON perceives that the future relationship will embody the harmony between humans and machines that will allow people to bring out their full potential and put their capabilities to the fullest use by getting support from machines. One example of such harmony is OMRON's table tennis robot FORPHEUS.
Since its birth in 2014, this robot has undergone continuing evolutions as a result of OMRON's untiring R&D efforts. These efforts include incorporating feedback from people who have experienced interactions with the robot each year at exhibitions. The fourth generation FORPHEUS developed in 2017 enabled an even greater range of people to experience rallies with the robot than before. Only when it becomes possible to bring out the best in as many people as possible, will the future in which robots support people become a reality. Here is the story that traces the evolution of FORPHEUS and the challenges it has continued to take on.

Aiming to fulfill people's desire for greater enjoyment

Masayuki Koizumi, the leader of the fourth generation FORPHEUS development team

To enjoy a table tennis rally, we must meet various requirements. These include the ability to serve, the ability to return any shots including a smash shot, and keeping the motivation to continue a rally. With the new generation FORPHEUS, the team aimed for meeting all these requirements to enable more people to better enjoy a rally.
Masayuki Koizumi, development leader, explains, "We asked a variety of people to experience a rally with FORPHEUS and discovered that beginners often failed to start a rally because they could not serve properly. By contrast, advanced players easily got bored with a simple rally." With the aim of fulfilling people's desire to better enjoy table tennis rallies, three new functions were developed. These are the serve function, prediction of a smash shot, and sound-based communication.
"We knew that 30% of beginners fail in making a serve, so we thought they may be able to better enjoy a rally if the rally starts with the robot's service. Advanced players, who are able to continue a rally, often want to try hitting a smash shot against the robot. We thought we might be able to amaze people if the robot could predict a smash shot and return it correctly," Koizumi explains. The team also came up with the idea of playing music to accompany the rally to allow players to better enjoy the rally with the robot. These ideas were developed based on analysis of the behaviors of people who had played a rally with FORPHEUS in the past.

Having the robot simulate human body movements

To allow anyone, from beginners to advanced players, to improve their skills while enjoying a rally--making this possible required considerable effort and creativity from developers.
To have the robot toss the ball up or predict a smash shot in fact was not an easy task, although these are all things that human players do naturally. To make these possible for a robot, the development team focused on the movement of the player's body.
Daichi Ueki, who was in charge of the robot's serve function, remembers their observations. "So, we video-recorded the ball toss motion of a human player and played it back in slow motion many times. Finally, we found the series of hand motions that the player made. We noticed that the player lowers the hand once as if trying to drop the ball. The player then makes a twist of the wrist to make the ball leave his hand," he explains. He made improvement after improvement of the robot's hand shape in a short period of time, finally perfecting the ball toss on a trajectory similar to that of a human player's.

Newly added serve function

Yu Maruyama and Kyohei Asai of the development team were responsible for the smash prediction function.
FORPHEUS originally was not very good at responding to fast shots. The team realized that if FORPHEUS could detect the ball only after a fast shot was hit by the opponent, there was no way it could hit it back properly. There had to be a way to predict that a player was about to make a smash. So Maruyama and Asai paid attention to the opponent's posture when hitting a smash.
"When hitting a smash, many people tend to move their shoulders as they raise their arm. By rotating their body sideways for a larger backswing, their shoulders become widely apart. We thought that by recognizing this movement with the built-in camera, FORPHEUS would be able to predict a smash shot," says Maruyama.
Based on this prediction, Asai incorporated a motion into FORPHEUS that would allow it to return a smash.
"Once FORPHEUS predicts a smash shot, he then has to respond to it. We did this by having FORPHEUS simulate a human player's block returns. FORPHEUS sets up his racket at a position where a smash is most likely to come, reaches the ball with minimum motion, and blocks the smash." Masamune Nakayama, in his first year at OMRON, was in charge of strengthening the motion control in response to a ball with an arch-shaped trajectory, which FORPHEUS found most difficult to return. He also worked on improving the camera used for checking the return position. These improvements helped the team overcome the problem of frequent missed shots when returning a ball with an arch-shaped trajectory.

Kyohei Asai (left) and Masamune Nakayama (right) testing the smash prediction function

The sound-based communication function is designed to play music during a rally and change the sound depending on the speed of a return and the position that the ball drops. This allows the player to enjoy a rally while getting into the rhythm to match his/her skill. The speech function was also added for FORPHEUS to react to how the rally is going.
According to team leader Koizumi, the team focused their attention on the relationship between the auditory information people receive and their ability to concentrate. "We learned from a member who was working on research into the human ability to concentrate. The research found that sound becomes an important factor to promote concentration. In fact, table tennis players can never maximize their performance if their hearing is blocked," explains Koizumi. "We receive a surprising amount of information through our ears, such as rhythm, sound, and voice, and we use that information. Based on this knowledge, we thought music may be a way to raise the player's concentration level."
After repeated trial and error, FORPHEUS has further evolved to enable more people to experience the harmony between humans and machines.

At CEATEC JAPAN 2017, more than 15,000 visitors watched rallies between human players and FORPHEUS

How was it possible for OMRON to create a table tennis robot?

FORPHEUS is a robot capable of table tennis rallies. But inside the robot are OMRON's industrial devices and its core technology of "Sensing & Control + Think."
"Sensing to detect the ball's trajectory, accurate control of the robot's movements, adjusting the robot's motions to adapt to each player, OMRON can do all these at one time." explains Masato Kawade, who supervised the development of the fourth generation FORPHEUS. "As we combine our products and technologies to develop FORPHEUS, we constantly look at the needs of our customers. For example, the recognition technology to determine the type of shot can be applied by any manufacturing site that requires 3D recognition, such as robots for pick-and-place applications. FORPHEUS's built-in deep learning technology for time series forecasting will help enhance speed and accuracy for a machine's real time judgment, for use in crisis prediction in self-driving cars," he points out.

Masato Kawade, who supervised the development of FORPHEUS

As it makes a return, FORPEHUS can now say to the opponent, "Let's enjoy continuing a rally!" Far beyond the table tennis table, an exciting future awaits us to enable each person to bring out his/her full potential and put his/her capability to the fullest use in daily living with the support of robots. This is exactly what OMRON seeks to realize through the "harmony between humans and machines."