[Advanced Technology R&D Center] ― Changing things through AutomationVisualizing the future

Visual Haptics – colors represent force

Up to this point, conveying the sensation of a robot hand grasping an object, has involved equipping the operator’s hand with a glove or other device which contains motors and sensors to convey that sensation from the robot hand to the operator’s hand. Such devices, although effective, are often unwieldy and uncomfortable to use. Visual haptics offers a new and simpler way to convey this feedback, by using color changes on a screen rather than mechanical force. A system of Augmented Reality (AR) overlays colors onto the hand, so that when a person operates it from a remote location, a change of color is recognized as a change in force. This provides the interaction (crossmodality) needed between the human sense of sight and the haptics.

Human Eye Display – mimicking how we see

In the world of remote control, it is important that images and control signals be exchanged in real time. But when people are operating from a remote location, delays are simply unavoidable – and these delays impair the user’s experience. This is why the Human Eye Display has been designed, to give the communication a ‘definition’. It works by only transmitting certain parts of the pictures in color and high resolution, sending the other parts in monochrome and low resolution. This reduces the amount of data needing to be transmitted, minimizing delays and bringing improved functionality and effectiveness. This ability to ‘focus’, is a characteristic of human vision that has high vision in the central region, low vision in the peripheral region, where in reality images are unfocused.

In order to reduce data traffic and to achieve simple, light operability, the Human Eye Display was developed with a focus on human cognitive characteristics.

The compact and highly efficient AI Maisart automates repetitive unconscious tasks and frees them from simple tasks.

Revolutionary Automation

The third of the three technologies that comprise the Remote Control Service Platform is gradual automation. This is when we gradually change to automation, step-by-step, via tasks that are easy to automate. A key benefit of gradual automation lies in the fact that it allows people to only concentrate on the tasks which require their attention. So it reduces the burden on people and it also makes it possible for one person to remotely control multiple robots.

Inspired by Sustainable Development work

Masaki Haruna, Ph.D
New Business Strategy and AVATAR
Technology / Manager
Sect., Mechatronics Dept.

Munetaka Kashiwa, Ph.D
Senior Manager, Mechanical Dynamics
Sect., Mechatronics Dept.

The impetus for developing the Remote Control Service Platform came via Mitsubishi Electric’s Sustainable Development Goals project in Indonesia. As Dr Masaki Haruna, of the R&D Center’s Mechatronics Engineering Department recalls:

“Through a project to popularize refrigerators, I strengthened my desire to support developing countries, but realized it is unrealistic to expect people to leave the place they are used to living in. However, labor shortages are becoming apparent all over the world so with the use of remote control technology, it is possible to work in the area where you were born and want to live.”

The project demonstrates that at a site where there is a manpower shortage, a system can be created in which a robot operator from a remote location can perform part of the work – and be compensated for that work. Another advantage of the platform is that it does not force people in these regions to have to wear traditional feedback devices.

Online shopping possibilities by 2025

Soon it maybe possible for shoppers to hold and examine and choose the produce they like.

The Remote Control Service Platform centered on visual haptics was originally conceived to be used in scenarios where dangerous materials such as explosives were handled, but Mitsubishi Electric’s policy is to always aim for application in the widest possible range of fields.

In this way, the haptics technology will make it possible to shop online and still replicate much of the experience of being in a real, live store. The team is also exploring the possibility of using robots to serve customers, in place of people.

Possibilities for the logistics industry

It is also anticipated that the Remote Control Service Platform will be used in the logistics industry. In tandem with autonomous driving technology, it will be possible to remotely control the transportation truck and also the cargo handling.

“Cargo handling is labor-intensive work, but as long as it is a robot that actually carries the cargo, it doesn’t matter if the robot is operated by a man or a woman, does it?” (Dr Masaki Haruna, Mitsubishi Electric R&D Center).

With the aid of autonomous driving, cargo could be handled and delivered using AR controlled robotics.

Agricultural and manufacturing applications

There are also possibilities for the application of the haptics technology in agriculture where it could be used for harvesting fruit. Now farmers will no longer have to do all of the time consuming harvesting. Remote controlled robots can do some of it for them. At the same time the farmer’s labor shortage problem is alleviated, there could be an opportunity for a short-term part-time job for a remote operator.

A similar system could be employed in the manufacturing industry, where remote workers could carry out transport and product inspection work remotely. Once again, the manufacturing industry could solve its labor shortage problems and remote operators could obtain new sources of income.

Opening up a world of remote control possibilities

While there are of course, some issues still to be resolved (including legal ones) Mitsubishi Electric’s Remote Control Service Platform presents the world with an array of possibilities to address a range of social and economic issues and solve labor-related issues for many of the world’s industries.

Products and Solutions

AC Servo System MELSERVO-J5

As well as providing even higher speed and precision in production equipment, the MELSERVO-J5 is the first servo system to support the next-generation industrial open network protocol, CC-Link IE TSN. This enables factories to harness IIoT and edge computing technologies to help customers reduce their total cost of ownership (TCO).

Show details

3D Simulator MELSOFT Gemini

Mitsubishi Electric's Gemini 3D simulator can verify line equipment in a digital environment.
Additionally, it allows pre-verification worker flow lines and line layouts of the assembly line.
With this capability, you can significantly reduce on-site adjustment time and cut costs.

Show details

Data Science Tool “MELSOFT Mailab”

Mitsubishi Electric’s Data Science Tool MELSOFT MaiLab is a data science tool that further improves manufacturing by replacing “human experience and intuition” with digital technology and enabling it to be easily incorporated into control systems.

Show details