Linking people to the future with OMRON窶冱 perspective on cutting edge technology
FORPHEUS, the table tennis playing robot, has evolved every year, showcasing OMRON's flagship core technologies. The newest fifth generation FORPHEUS is packed with advanced technologies to enable additional features and functions. These include the ability to cut and drive return balls, predict the human player's movements with higher accuracy, and provide coaching skills tailored to each human player. Once again, it has approached closer to the "robot that can bring out the full potential of people" that is essential for realizing "harmony between humans and machines" that OMRON aims for. For the development of this robot, five engineers/researchers including two new employees got together. The development period allowed for them was a mere nine months. During the course of this development, they faced many challenges and made many accomplishments. In the following interview, the team members share their personal experience of working in an uncharted area of robot technology, and the many roadblocks they encountered before achieving success.
Members of the Fifth Generation FORPHEUS Development Team:
Leader: Satoshi Yase, Wireless and Embedded Lab, Embedded System Research Center
Hardware/Control: Kyohei Asai, AI Control Lab, Intelligent System Research Center
Control: Xiaojun Liu, AI Control Lab, Intelligent System Research Center
Vision: Masamune Nakayama, Vision Sensing Lab, Sensing Technology Research Center
Vision: Yuki Sasaki, Vision Sensing Lab, Sensing Technology Research Center
Interviewer: We understand that the development concept for the fifth generation FORPHEUS was "KURO-OBI: Expert Player and Coach." Could you tell us why you chose this concept?
Yase: The purpose of developing FORPHEUS is to have people understand our vision of "harmony between humans and machines" that OMRON aims to realize in the future, and relate to this through experience. In other words, it is a future in which people and machines learn from each other and grow together. Personalized coaching is essential for bringing out the full potential of individual human players and putting their capabilities to the fullest use. To make this happen, we thought that FORPHEUS had to be an expert table tennis player and an expert coach as well.
Interviewer: Specifically, what kind of evolution did you aim for to make FORPHEUS an expert?
Asai: We wanted to add the ability to cut and drive shots returned by a human player. To make this possible, we had to revise the hardware itself. The pick-and-place robot* in use could push the racket forward to hit the ball, but it was not good at swinging the racket upward. To cope with this weakness, we thought we should probably increase the number of control axes. So, we added an extra axis for the robot's joint to simulate the movement of a human elbow.
Yase: The ultimate goal for a pick-and-place robot is to give it the ability to handle precise and delicate work that humans perform with their hands. It involved a challenge to approaching the functions of the human arm and hand from the shoulder down to the finger tips, as closely as possible.
Interviewer: Was there a hardware specialist on your team?
Asai: A new member, Liu, studied hardware at his university, but he was not a member of our team when the development began. So, I was assigned mostly to deal with hardware. But my specialty is in information processing. So, I had hardly even touched hardware, much less worked with its design. Thus, the entire process was nothing but hardship for me at first.
Interviewer: That must have been very difficult. What was the most difficult part you had to deal with?
Asai: Well, take a motor or decelerator, for example. There are a countless number of varieties, so to begin with, I had to spend lots of time choosing the ideal one. The control of the motor was also difficult. When I started, even a simple movement caused vibration in a control axis, and it sometimes took one or two weeks to identify the cause of the problem. There was no robotic arm of that shape in the world at that time. There were also no documents I could refer to, so I had to conduct trial and error day after day.
Yase: In addition to building the hardware, controlling the motion of the hardware was another big challenge. Table tennis requires a very fast racket swing. The robot has to swing the racket at a speed of 1/1000th of a second or so to correctly hit the ball. We had to create hardware that is fast and accurate, and also had to embed an algorithm that could maximize its performance. We had to spend a considerable amount of time and work to enable these.
Asai: For example, in table tennis, a drive shot adds a spin to the ball by hitting it with an upward path of the racket. What kind of a command should we give to the robot and what kind of a move should we have the robot perform? There was no research conducted for this anywhere in the world, and the multi-axis robotic arm that swings the racket was the first of its kind in the world, as well as the algorithm to control the racket's movement. It was like moving forward in a world with no right answer. There might have been some other way, but I think that what we achieved within the limited timeframe is evidence that we took one of the right approaches.
Liu: My case was the direct opposite of Asai's. While I was in the university, I studied mechanical systems, so I had no experience with devising or embedding an algorithm before joining OMRON. So, it took me a lot of time to develop the algorithm. But I still had to meet the development schedule. So, I was under a lot of pressure. But at that time, I was given an opportunity to attend an exhibition held in my home town Shanghai where the previous fourth generation table tennis robot was displayed.
Interviewer: China is famous for table tennis. Customers there must have been severe critics.
Liu: Yes, they were. They were very demanding, saying "Can't it hit a faster shot?" or "Why not hit a smash?" But hearing these comments, I felt the need to work much harder to embed an effective algorithm, in order to have them understand how great our table tennis robot is. This gave me encouragement. I'm sure the fifth generation FORPHEUS will satisfy the customer who was complaining about the slow shots of the previous model.
Asai: By contrast, at CES in January 2019, where we unveiled the fifth generation table tennis robot, many people noticed the finely detailed improvements we made, such as the ability to return with a topspin or some changes in arm movement. At Hannover Messe in April, even German Chancellor Merkel watched FORPHEUS play in close proximity, which turned our hardships into joy.
Development team at CES 2019縲Interviewer: In addition to the multi-axis robotic arm, the fifth generation also employs an increased number of cameras.
Nakayama: Table tennis involves extremely dynamic movements, so a single camera is not enough to capture a desired movement. For example, when the player holds his hand high above his head, the player's body may get in the way, and make the hand motion invisible. So, we decided to use two cameras instead of one to record the movement of the human player. To predict the direction the player will return the ball, we thought it would be necessary to detect not only the player's body motion but also the position of the racket and the direction it faces. So, we added a new camera to track the racket. We relied on new member Sasaki to handle image processing for that part.
An additional camera to record the racket movementsInterviewer: Mr. Sasaki, did you have any experience in image processing?
Sasaki: No, it was the first time for me to deal with image processing, although I did programming while in the university. The first difficulty I encountered was capturing the image of the racket. This alone took a considerable amount of time.
Yase: The racket is very small compared to the size of the robot. Moreover, the environment changes from exhibition to exhibition, and the lighting varies as well. Visitors' motions and colors of their clothing vary greatly. Under such conditions, it is extremely difficult to identify the racket and make necessary adjustments for the camera system.
Interviewer: Just by listening to what you say, I can see how hard that was.
Sasaki: For me, the pursuit of accuracy itself was a new challenge too. While in the university, it was enough to create something decent and submit it. But here, if the accuracy of data I provide is poor, it would be impossible to control the robot. I hadn't imagined it would take that much time to raise the accuracy of the data I made.
Sasaki conducting camera calibrationNakayama: In determining the racket's orientation based on its marker, the initial algorithm was not accurate enough in terms of judging the direction in which the racket faces. So, while Sasaki was striving for greater accuracy, I concentrated on building an algorithm that can determine the direction the player is about to return the ball, even when using low-accuracy data. At the moment that the ball makes contact with the racket, the ball and marker overlap, so sensing the racket is impossible. So, the data we have is just before the ball hits the racket. I tried various algorithms in search of one that could predict the movements of the racket and ball based on that data. As a result of repeated trial-and-error experiments, I was able to configure a system that could predict how the racket and ball move in a certain degree, without regard to the player, and no matter how the player holds the racket. This made it possible to accurately predict the direction the player will return the ball before it is actually hit. I take pride in the fact that this enabled the robot to return the shot to an optimum location like an expert player does, by taking the proper stance in advance.
Interviewer: That is amazing! Because you have reached that high of a level, you must be very interested with feedback from those who actually experienced a rally with FORPHEUS, aren't you?
Sasaki: Absolutely! Outcomes of development related to the vision technology are easily noticeable to viewers. At CES, we got many questions from visitors regarding the vision system and there were also visitors who took photos. I was very happy with the favorable response from customers.
Nakayama: What made me most glad was hearing visitors say something like, "It's so different from last year's model," or "This is more than a simple improvement; it's an evolution!" I was very happy to see that the technologies and functions we had developed were fully understood by people, and that they really enjoyed them.
-- After the team had overcome all the difficulties and troubles involved, the fifth generation FORPHEUS had come into existence. The future the developers envision beyond this development is one that will make people happy. What will that future look like? Find out in Part II!
