“To Further Human Creativity”
- Developing a Novel Robot that Closely Assists Humans -

In April 2022, the Technology and Intellectual Property HQ, OMRON’s corporate R&D division, launched a new department named「Voyager PJ」o set out on a new endeavor to create robots of a new dimension. Two development leaders of the project that focuses on cutting-edge R&D in robotics, Yoshiya Shibata and Kazuma Tabuchi, are working to reinvent a robot system architecture from scratch. We invited them for an interview to ask how they have built up their skills through their experiences from past development projects and what led them to aspire to develop brand-new robots that no one has ever seen.

Part 1 of this story will focus on Shibata’s aspiration to realize robots that execute complex movements with simple teaching.

The Challenging Task of Developing a Robot that You Can Teach as You Would a Human

Brief Profile of Yoshiya Shibata

Experiences in Developing Recognition Technology and Control Technology Led to the Realization of the Importance of Finding Issues from the Customer’s Perspective

I had longed to create a machine that moves autonomously like organisms. And so, it was natural that I chose OMRON as my career path after completing my graduate course because the company was presenting Sensing & Control as its core technologies at the time. After joining the company, I was assigned to a team working on the development of a solder inspection device for electronic circuit substrates by applying image processing technology. There, I was involved in the development of technology for recognizing electronic components to be mounted, which was intended to reduce the burden of starting up the device. Because so many parameters needed to be set initially for inspection, the process of inputting data into the device was a huge burden. The automatic teaching function we developed was implemented as an assist feature, thus significantly saving labor for making settings for substrates.

Then, in 2011 I was charged with developing motion simulators and others as application tools for programmable logic controllers (PLCs). A motion simulator is a tool that visualizes how a program developed to operate multiple motors simultaneously behaves. Truthfully, I had no knowledge about PLCs when I was given this task. In retrospect, however, I was able to deepen my understanding of control thanks to cooperation from my counterparts from business divisions and other members of the team.

If I look back on these experiences as a developer, which encompassed recognition technology, control technology, and application software development, I was lucky enough to learn about the functions that robots need from a hands-on approach. I was also made aware of the importance of developing technologies that enhance operability for users. All in all, I think that they were amazing experiences.

Getting to Know What It Takes to Develop a Robot System While Leading the Development Team of the First Generation of FORPHEUS

In 2013, I was appointed as the leader of the project to develop the first-generation FORPHEUS. OMRON is developing this table tennis playing robot as the embodiment of its core technologies to represent the “relationships between people and machines” that we strive for. Its research and development began with a plan to unveil the creation at our trade show held outside of Japan. The concept of that exhibition was two-fold: “have visitors understand OMRON’s vision of the future of humans and machines through experiences” and, because the trade show was hosted by China, “produce a robot to play table tennis with.” The concept was determined in April of the year the trade show was scheduled for. We had to start from scratch and complete the project in a very short period as the trade show was to be held in the fall. I think I was singled out for the position as the development leader because I had been involved in the development of technology for picking and placing components immediately before. The pick-and-place technology uses a camera to recognize components on factory automation (FA) conveyor lines and performs the real-time computation of the travel distance and a movement plan for picking up a component based on how fast it is moving. I think they expected me to further advance the technology I had been working on for the previous development project.

The table tennis playing robot sequentially recognizes the ping-pong ball, decides where to hit the ball back after computing where it moves how many seconds later, and creates a model of what angle and speed at which the racket should pass the position. If we could build a system for repeating three steps of movements—take-back, swing, and follow-up—the robot would be ready. It didn’t take much time before I hit upon a plan. For the pick-and-place technology, the coordinate system for recognition is two-dimensional, but for the table tennis playing robot, it is three-dimensional. This makes it difficult to create a prediction model and plan movements, but I was reasonably sure it was doable so long as we could expand each function. Once I got the plan fixed, I could dissect the issue into different tasks, like how much we could increase the accuracy of predictions and synchronization timing.

I was beginning to feel that I had an aptitude for designing a concept, that is, combining functions for which I had prior development experience to realize a target system, although we didn’t initially have a function for predicting the ball’s spin and such and all the robot did was simply hit the ball with the racket.

Nevertheless, I had a lot of trouble going through the development with the time of delivery set for the trade show. The robot was billed as “capable of rallying with human players,” but you wouldn’t call it a “rally” if the robot cannot return the ball ten or so consecutive times. Also, because not all the players are good, some return an arching ball or may not return a ball where the robot can easily hit it back. This made it extremely difficult to determine the number of varieties of returns we should consider.

Until then, I tended to think within a narrow range of ideas, like recognition technology, control technology, or from the perspective of a specific application. Through the experience of working on FORPHEUS, I realized that it would be very complicated to develop the system as a whole and that it would be quite challenging to handle the system.

It was also around this time that I began thinking it was no use in evolving our robot linearly if we wished to realize “harmony,” which OMRON advocates in its vision for the future of human-machine relationships.

Moving to the U.S. by Himself to Take a Position as a Researcher in Residence at MIT with the Aim of Developing Robots that Can Be Used by Anyone Who Wants to Use Them

Those who want to use a robot are not necessarily robotics experts but are people who hope to solve their issues by using a robot. This made me consider what a robot for such users would be like. As I searched for prior art of this kind, it came to my attention that Professor Kamal Youcef-Toumi of MIT, who is engaged in research of a multi-agent system(1), was working on ways to dynamically recreate plans and schedules by taking the dynamics corresponding to ever-changing situations into account. Because the development of intelligent robots that anyone can use was among the focus topics at the Technology and Intellectual Property HQ, and I had experience developing table tennis playing robots there, I was invited to do research at MIT.

There, I got to research how to dynamically avoid obstacles or generate a work schedule for multiple robots vis-à-vis target production volumes. Through these experiences, I learned how to conduct research, namely, conceive an overarching plan, move it, find issues, and redo it.

(1)Composed of multiple agents, a multi-agent system solves problems that are difficult for an individual agent or a monolithic system to solve. For details, please see the entry on Wikipedia (https://en.wikipedia.org/wiki/Multi-agent_system).

Participated in the World Robot Summit 2018 (WRS2018) to Debut a New Robot System

After returning home, I worked on the development of element technologies for robots for anyone to use, including grasping skills and fitting/engagement (2) using AI technology. To see what robots equipped with these technologies can do, I signed up for an assembly task challenge(3), which was organized as part of the World Robot Challenge 2018 (WRC2018). Participants were expected to assemble a robust product without the need for dedicated jigs, with moderate room for positional errors.

Concerning “fitting/engagement,” one of the challenge tasks, I had assumed that I could use conventional techniques for controlling positioning and force, but I was wrong: Because you never know if you have succeeded before different components make contact, it was necessary to recognize and control positions both rapidly and accurately before contact occurs in order to increase the success probability, which made it necessary to shorten control frequencies. The drawback of this is that it would demand high throughput from hardware, which adds to the cost of the system, thus making it harder to popularize the robot. So I decided to take the opposite approach. Namely, if a human did the job, he or she wouldn’t try to fit components carefully from a distance. Rather, the objects to be fitted are brought into contact first, and then a “sweet spot” is sought until they are fitted together nicely. Compared to humans, robots can move with greater accuracy and speed, but humans have the unique advantage of moving with dexterity while tolerating errors. This being the case, our challenge was to create a robot combining the best features of robotics and humans.

The robot that we assembled under the concept of “no dedicated jigs” at the WRS challenge ended up winning fourth place overall and being honored with the Society of Instrument and Control Engineers Award. This gave me confidence that my goal of robot development was not wrong, but at the same time, I was keenly aware of how difficult it would be to make it happen.

(2) Getting a screw or pin through a hole

(3) The World Robot Challenge (WRC) is a competition staged as part of the World Robot Summit (WRS).https://wrs.nedo.go.jp/wrs2018/

In 2021, a different team, O2AC, comprised of members from OMRON SINIC X Corporation, Osaka University, National Institute of Advanced Industrial Science and Technology (AIST), and Chukyo University, competed in WRC to earn third place and the Artificial Intelligence Society Award.

News release (https://www.omron.com/jp/ja/news/2021/09/c0928.html) (in Japanese)

Related paper (https://www.omron.com/jp/ja/technology/omrontechnics/2021/20211119-felix.html)

Envisioning the Future That You Wish to Bring About and Striving to Realize the Vision as You Keep Solving Issues

You can assemble a robot if you give shape to what you want the robot to perform in the form of concept design and combine the necessary element technologies; however, a common approach today is to develop single-purpose robots individually, such as table tennis playing robots and robots that play shogi. That way, we have to launch a project for each given application, which requires so much labor to tweak that the resultant robots will never become widespread. To popularize robots, I want to develop robots that people can use with simple teaching.

I have come to believe that, for this to happen, we need to tap into our past development experiences and build a new architecture of robots that touch, notice, and correct like humans.

To build such an architecture, first and foremost, we need to design a grand concept based on what we have learned from past development projects, and apply whatever components and software that are available and necessary. Alternatively, we could begin by making a robot first, then operate and evaluate it until we get to know the functions and performance that are essentially necessary. Then we could build another to see what is lacking and take one step forward. As you go through this process, you will feel in your bones that you are getting closer to the future that you wish to bring about. I have no doubt whatsoever that we will finally be able to take on the challenge of developing robots that help to enhance human creativity, stand by and assist humans if we follow this process repeatedly.

On days off, Shibata loves going to sea to do lure fishing. He continues to address one new self-set issue after another to move toward the development of a novel robot. Like a migratory fish, he never ceases to move forward in his endeavors.

In Part 2 of this story, we will bring Tabuchi’s thoughts on how robots and humans should interact with each other.

You may also like:

“To Further Human Creativity” - Developing a Novel Robot that Closely Assists Humans -
https://www.omron.com/global/en/technology/activities/07/

Careers
https://www.omron.com/global/en/about/job_search/