Artificial Intelligence breathes life into "digital sardines"
- Learning from nature and giving back to global society –

World trends have finally caught up to manufacturing in Japan
Demonstration of "digital sardines"

What you see here is a video image of some 15,000 sardines swimming in a large group in the boundless expanse of the ocean. Moving to the right or left in a well-coordinated manner, sardines avoid bumping into each other, with the shoal constantly changing its shape. It's hard to look at this image and not be impressed by the beautiful movements of the sardines.

At a first glance, what is shown in the video can be easily mistaken for a real shoal of sardines. But if you take a closer look, you will notice that it's not a recording of real sardines, but a computer graphics simulation.

You might think that the advancement of digital technology makes it easy to create any kind of movements with computer graphics. But in fact, moving some 15,000 sardines in a harmonious manner as seen in the natural world is not an easy task at all.

Behind this beautiful image is a computer with the latest GPU working at full capacity to process a huge amount of data. Embedded with AI technology based on precise calculations, the CG sardines move in a well-coordinated manner.

What makes this possible is OMRON's machine learning technology, which enables a computer to learn in a similar fashion as humans do naturally in the real world. This involves extracting rules or criteria for decision-making from among large quantities of data, recognizing patterns, and making predictions about new data according to these patterns. OMRON developed this machine learning technology through its development of social infrastructure systems and manufacturing automation technology. Another key technology that makes precisely coordinated movements possible is OMRON's vision sensing technology for 3D measurement and other forms of sensing.
OMRON first unveiled its "digital sardine" technology in 2013 in China. The shoal of "digital sardines" swimming exactly like real ones amazed people in China when they saw the image for the first time.

Members of the "digital sardines" development team

The technology was developed by a team comprising senior researcher Yasuyo Kotake and three other researchers—Tadashi Hyuga, Kennosuke Hayashi, and To Sho—and the team leader, Masato Kawade. With a background in brain science and systems neuroscience, Kotake is devoted to the development of new solutions to enable harmonious interaction between people and machines by drawing on image processing and biosignal processing technologies. Hyuga's specialty includes a wide range of vision sensing technologies, from character recognition to detection and recognition of a person's physical condition. Hayashi specializes in behavior recognition and interaction of people based on 3D measurement. Sho is adept at manipulating the space of digital rays through the employment of optical measurement and computer graphics technologies. Uniting this group of members with different areas of expertise is Masato Kawade, who leads the development of technologies from future-oriented and global perspectives. The "digital sardines" are the result of combining a diverse range of technologies reflecting the strengths of each member.

But it's fair to ask, "Why digital sardines?"

The answer is this: OMRON has been committed to addressing technological challenges to realize the autonomy of individuals, mutual coordination, and total harmonization to offer optimally balanced solutions that are adaptable to changes in the social environment. OMRON's goal is to build a society in which individuals with their own goals and values strive to solve issues while mutually detecting and coordinating the intentions of one another. By so doing, optimal harmonization is achieved for large-scale systems to fulfill unified goals and values.

When the team observed the way that sardines live together as one big group based on unspoken rules to protect themselves from underwater predators, they realized that the technology pursued by OMRON would also require following the laws of nature.

This inspired Kotake and other members of the team to thoroughly study the behavior of sardines.

The purpose of life leads to total harmonization

A sardine run—the phenomenon of thousands of sardines swimming in a big shoal

Occasionally, a sardine run may be seen with more than 50,000 sardines migrating in the ocean. The sardines group together to form a huge ball, and make unique movements to alter the shape of the ball from minute to minute to scare away enemies. Because sardines have many predators, a communal lifestyle is indispensable for this species. A smaller group of sardines, or those separated from the group, would easily become the prey of larger fish. This is why sardines instinctively have a habit of swimming close together, forming large, ball-shaped groups.

But sardines group together not just to escape attack from predatory fish. Sardines have a keen sense of smell, so if one or two of them detect food, it is transferred to others in the group so that they all move toward the food in a big group. Sardines inform others of the presence of predators or food and this is how they survive.

At first, Kotake and the others thought they could reproduce a school of sardines using a CG technique, setting parameters to control CG-created sardines that would make them move to avoid colliding with each other. Parameters included the distance from surrounding entities (mates, food, and predators), and the speed and direction of swimming. But their CG sardines didn't move as intended. They seemed to move randomly without intention.

The team surmised that the cause of the failure may be that they didn't give each sardine a purpose of life—living communally in order to survive.

The team went back to the drawing board.

Sardines leading a communal lifestyle are said to have the following three behavioral patterns:
1. Escaping when an enemy approaches
2. Swimming close together to form a big ball
3. Finding and consuming food

Sardines recognize ever-changing marine conditions, and move while changing their priorities for these three patterns. An AI algorithm to simulate this behavior had to be developed. CG sardines stopped moving when there were too many parameters to control their movements. If a sardine's unique personality was too strong, it went out of control so total optimization could not be achieved. The team underwent repeated trial-and-error adjustments. To faithfully reproduce the movement of real sardines, the team compared the movement of CG sardines with that of real ones. They did this over and over and made repeated improvements.

This process finally resulted in a community of "digital sardines" in which each sardine can think on its own and coordinate its behavior with that of others in the group. In this way, each sardine worked toward a single unified purpose, while maintaining its unique potential. Kotani provides her perspective:

"In the past, I have developed image processing techniques for detecting the flow of people or the characteristics of goods being conveyed on a production line. To allow a machine to interpret the behaviors of a person or those of other machines, the machine must be able to think on its own to discover optimal parameters, and take action according to the parameter data. To allow each sardine to individually change numerous parameters at a high speed to derive an optimal solution for the whole has things in common with how image processing works."

Living style of sardines provides insight into people's lifestyles

The technique used to come up with an optimal solution for the "whole" is similar to the technology that allows a car to detect traffic conditions, come up with the best route, and avoid traffic congestion. This is not only optimal for the individual driver, but it also reduces traffic congestion for the "whole" of the drivers on the road.

There are several other technologies that have things in common with this technique. One example is technology to prevent accidents at intersections, by allowing pedestrians, bicycles, and cars to warn each other of their relative positions. Technology to monitor the electricity consumption of each household in a town will also help the smart grid to supply power generated by solar panels and stored in battery units to households when necessary. All these technologies now under development at OMRON will enable total optimization, which in turn will ensure safe and convenient living for all.

"I want to continue developing new technologies to allow machines to correctly interpret the behavior or intentions of people or other machines. This can provide many benefits to our society," Kotake concluded.

Each one of the sardines leading a communal lifestyle can be related to each one of us as humans. That is why we should learn from nature and give our knowledge back to global society in the form of innovative technology. OMRON will continue to help create an ideal society for the future—a society in which total optimization and harmonious balance will be enjoyed among people, machines, and the earth.

A conceptual illustration of a future street intersection where pedestrians, cars, and bicycles exchange positional information with each other

All story

  • Can the

    MEMS Microscopic eyes to observe the heartbeat of the Earth

    Creating a repeating heartbeat-like vibration, this tiny 2mm-square machine is a MEMS absolute sensor that can detect even a minute change in air pressure, such as when a person climbs up or down a single step of stairs. Committed to giving shape to people’s “impossible wishes,” OMRON developers provide insight into the future that this technology will bring about.

  • Can the

    Can the "eyes" of a machine see through society and look into the future?

    The "eyes" of machines instantly capture and recognize the license plates of all passing vehicles on highways regardless of their types and speeds. OMRON's 2D and 3D vision sensing lets machines "see" with greater precision and speed than human vision. Can machines equipped with five senses and that can "think" look into our future? The answer may be here in the work and accomplishments of OMRON's leading engineer, Masaki Suwa.

  • A factory run by challenged people—maximizing diversity in manufacturing

    A factory run by challenged people—maximizing diversity in manufacturing

    OMRON Kyoto Taiyo was founded 30 years ago, when it was difficult for challenged people to find a job. The company's mission was to provide challenged people with work opportunities that could help them enjoy more rewarding lives. The factory's efforts at creating a workplace environment tailored to each worker's needs and potential have led to optimal diversity in manufacturing.

  • Artificial Intelligence breathes life into digital sardines

    Artificial Intelligence breathes life into "digital sardines"

    Sardines live in large groups to protect themselves from predator fish. Within the group, they move about using unspoken rules of behavior. Observing this shoal of sardines, OMRON researchers realized that the technology pursued by OMRON would also require following the laws of nature. Why did OMRON develop "digital sardines?" And what kind of future will this technology bring about? Here are some answers to these questions.

  • World trends have finally caught up to manufacturing in Japan

    World trends have finally caught up to manufacturing in Japan

    OMRON's next-generation controller development project started when IoT and Industrie 4.0 were still unknown in the industry. What developers pursued was the unprecedented world of 100 microsecond (1/10,000 second) level control. Here's how the development team overcame all the challenges to perfect the industry's fastest controller.

  • Machine safeguards ensure that all workers return home with a smile every day

    Machine Safety ensure that all workers return home with a smile every day

    We are committed to developing globally usable products as we address the needs of manufacturers across the world. Our goal is to enable anyone to use machines safely, as we believe this is the right way to simultaneously maximize workplace safety and productivity.

  • Gemba-centric human resource development determines the state of manufacturing a decade from now

    Gemba-centric human resource development determines the state of manufacturing a decade from now

    Manufacturing starts with human resource development. A decade from now, stable production will need to be maintained no matter how the world's population structure changes. To make this happen, we are committed to fostering manufacturing leaders.

  • Taking on the manufacturing innovation challenge through

    Taking on the manufacturing innovation challenge through "one and only" control technology that is beyond the norm.

    What makes it possible to create such a technology that companies at the forefront of manufacturing are eager to adopt?

  • Establishment of Omron Corporate Governance Policies

    Establishment of OMRON Corporate Governance Policies

    We are committed to sustainably increasing our long-term value by putting Our Mission and Values into practice

    Fumio Tateishi, Chairman
  • Improved traffic flow makes a better world

    Fulfilling the desire of everyone to stay healthy longer

    Living a long, healthy life is a universal desire among people everywhere. So why not start managing your health at home to maintain good health and prevent disease? We will do our utmost to support your efforts.

  • Improved traffic flow makes a better world

    Maximizing energy efficiency for sustainable growth

    With our desire to bring electricity closer to people's lives, we strive to promote more efficient use of electricity and help realize a brighter future for households and production sites alike.

  • Improved traffic flow makes a better world

    A factory that makes a "highly personalized choice" possible

    Through our contribution to manufacturing innovation, we are ready to meet retailers' need to quickly display products that people desire, in turn helping people to enjoy more freedom in their choice of lifestyle.

  • Harmony between people and technology

    Delivering the joy and freedom of owning a car

    A car can be an important partner for the driver, whether it's discovering a new sightseeing spot, or fulfilling a long-held dream of landing a key client. To make this happen, we are determined to make cars safer and more friendly to people and the environment.

  • Improved traffic flow makes a better world

    Aerial Imaging — a guidepost on our path to the future

    Only the one who has taken the lead in driving the advancement of displays can see the society to come and envision the convenience that it will offer.

  • Improved traffic flow makes a better world

    Improved traffic flow makes a better world

    Only a traffic management system suitable for each specific country, region, or city can change the flow of people and goods—that is what we believe.

  • Harmony between hurman and technology

    Harmony between human and machines

    The robot hits the ball so it's easy to return. Why?