The world's first real-time encoder for ultra-high resolution Super Hi-Vision.
Video coding technology is one of the most important technologies in digital video applications such as terrestrial digital broadcasting, satellite broadcasting, and video distribution on the internet.
Video coding technology compresses digital video signal and thereby enables its transmission—for example, via airwaves—and storage. At present, most systems adopt MPEG-4 AVC/H.264 encoding. However, it is predicted that with the advent of Super Hi-Vision (8K), in the future the amount of information contained in digital videos will continue to increase as their resolution increases. In response, ISO/IEC*1 and ITU-T*2 have pushed ahead with the standardization of HEVC as a next-generation video coding standard with a higher compression performance.
Through initiatives including joint technical proposals with Japan Broadcasting Corporation (NHK), Mitsubishi Electric contributed significantly to the formulation of the final draft of the international HEVC standard in January 2013.
In addition, via a joint development initiative with NHK, we have also developed the world's first HEVC encoder that is able to encode Super Hi-Vision in real time.*3 The era of Super Hi-Vision broadcasting, which features 16 times as much information as Full HD, has moved one step closer.
Digital video contains several tens of frames per second. However, the amount of information involved would be enormous if the images in each frame were transmitted or stored without being compressed. It is essential in particular to significantly reduce the amount of information in order to transmit Super Hi-Vision (7,680×4,320 pixels) via airwaves. To enable this, HEVC transmits and stores only the differences between frames, i.e. the differential signal between areas of the image in motion, in order to make it possible to compress the image while maintaining picture quality.
In the case of MPEG-4 AVC/H.264 encoding, images are uniformly divided into 16x16-pixel blocks, and motion is detected in each block as a method of identifying differences between images in different frames. As for HEVC encoding, this block size is not fixed, but can be flexibly altered in response to the specific characteristics of the image signal.
The blocks in even sections of the image displaying minimal change, such as landscapes, are large, while the blocks in complex sections of the image are small. The use of the appropriate block size means that the differential information between images to be transmitted and stored can be further reduced, and HEVC realizes a compression performance approximately two times greater than MPEG-4 AVC/H.264.
Information Technology R&D Center
Information Technology R&D Center
Encoding the high volumes of data in Super Hi-Vision necessitates an enormous amount of computations. To enable this unprecedented volume of calculations to be conducted in real time, Mitsubishi Electric's newly developed Super Hi-Vision HEVC encoder divides each image into 17 horizontal strips. Each of these strips is then divided into optimally sized blocks; the blocks are large in even sections in which there is little change, and small in complex sections. Parallel processing of the different regions in each strip enables the realization of real-time encoding.
The issue when dividing an image into horizontal strips is the processing of the boundaries between the strips. If you simply divide the image and individually encode the strips, then the boundaries between the strips will be visible. In order to resolve this issue, we developed a technology that conducts processing by overlapping the areas close to the boundaries on adjacent strips.
In concrete terms, the modules processing adjacent strips share the image data close to the boundary that is essential for efficient encoding. The use of this shared image data has made it possible to control the degradation of the image close to the boundaries between strips.
Normally, when parallel processing is applied to an image, the image is divided into blocks of four squares, meaning that when the image is encoded, it is necessary to refer to eight areas: The four neighboring areas (above and below, left and right) and the four areas at 45 degree angles. This makes communication between the processing modules complex, and increases the volume of calculations. Because the newly developed technology divides the image into horizontal strips, it is only necessary for the processing modules to share information with two (above and below) adjacent strips. This reduces the burden on the modules, and enables the realization of high-speed processing.
In order to realize our aim of real-time encoding of Super Hi-Vision it was necessary to develop specialized hardware that is able to instantaneously conduct an enormous volume of calculations. How to incorporate the specifications of the new international standard HEVC in the hardware, and to what extent it would be possible to reduce the computational burden, were the crucial points in realizing the hardware.
Intra prediction is one of the main technologies in video coding. Intra prediction refers to the already-encoded pixels adjacent to the block currently being encoded. It generates a predicted image, and encodes the differences between the predicted image and the original image. In order to significantly reduce the amount of information, it is necessary to generate the predicted image as accurately as possible. The HEVC standard has 35 intra prediction modes. Of these, 33 are what are termed directional prediction modes, making it possible to select optimal reference pixels from among adjacent pixels in 33 directions. Selecting the optimum mode after conducting evaluations of every direction would entail an enormous volume of calculations, so we adopted a two-stage method in which the rough direction is first determined, following which the optimum direction is selected in the vicinity of the rough direction. By this means, we have been able to reduce the number of computations while at the same time balancing compression performance and real-time processing capacity. How would it be possible to simplify the algorithms without producing a decline in performance? This was the question on which the greatest amount of time was spent in development.
Despite the fact that a great deal of attention was focused on basic design such as circuit configurations at the initial stage of development, unexpected problems continued to arise in the course of the development. How flexibly would it be possible to respond at these times? On occasions, drastic changes had to be made. Without being tied to our original concepts, we went back to the drawing board and sought the best method. This was our approach to the development.
The development of the encoder discussed here brings the practical realization of Super Hi-Vision one step closer. Fully-fledged Super Hi-Vision broadcasting is scheduled to commence in 2020, and in the near future we will move to the phase of realizing practical use, for example by reducing the size and the weight of the encoder. However, at present, aspects of picture quality are unsatisfactory, and many problems remain to be resolved. For example, it will be necessary to improve the algorithms while bringing the system to the stage of practical use.
Mitsubishi Electric's strength is the experience and expertise that it has developed over the course of long years of work in this area. In the future as well, we will spare no effort in conducting research and development in order to enable as many people as possible to have the unprecedented experience of Super Hi-Vision.