Aiming to be a DX that creates new content by fusing real world images with digital technology" Toshihiro Matsumoto (NHK Enterprises, Inc.)
-5GMF Millimeter Wave Dissemination Promotion Workshop / Japan's Industrial Technology Frontline Report

The 5th Generation Mobile Promotion Forum (5GMF) held the second workshop on "Japan's Industrial Technology Frontiers" to promote the spread of millimeter wave in January 2024. The aim was to have a forum to interact with people outside of the broad telecommunications industry, and the second session featured speakers from NHK Enterprises, a company in the broadcasting and video production field. Toshihiro Matsumoto, Managing Director of the company, and Tetsuya Fukuhara, Executive Officer and General Manager of the Innovation Strategy Office, explained the relationship between content production and DX. In the first part of the report, Mr. Matsumoto gave a presentation titled "Go beyond the frame! Fusion Evolution of Real and Virtual - DX in Content Creation" by Mr. Matsumoto.

DX, a video production that fuses the real and virtual

NHK Enterprises is a television production company. By our count, we produce 18,000 video contents per year, about 80% of which are NHK programs, including numerous dramas, nature programs such as "Darwin is Here", variety programs such as "Chikochan ni scolded", "The Century of the Image: Butterfly Effect", "Family History", and We are involved in almost all areas of broadcast content, from documentaries to animation. In addition to program production, we are also involved in large-scale events such as "Robocon" and Exhibition & Expo, digital content production such as website creation and SNS transmission, archives business to create content databases, copyright management and licensing, and licensing for publication and stage adaptations.

In the past, each of these businesses was handled separately, but now we are aiming to become a "comprehensive contents company" that creates synergies across all these areas. To achieve this, DX will become indispensable. In many cases, DX is only about the digital world or only inside computers, but NHK Enterprises' DX aims to integrate the real and virtual worlds.

KIKI, a sign language avatar for an inclusive society

In the fall of 2023, we introduced one talent to the world. KIKI" is a sign language avatar. She is positioned as a virtual interpreter who can handle Japanese Sign Language, American Sign Language, and International Sign Language. KIKI's function is not only to represent a sign language, but also to be an icon for an inclusive society. KIKI uses a database of approximately 7,000 Japanese Sign Language words created by NHK's Broadcasting Technology Research Laboratories over many years, and is able to realize smooth sign language using space. KIKI is a sign language system that uses space to achieve smooth sign language.

NHK has a "Weather and Disaster Prevention Sign Language CG" page, which is of great social significance because it provides a service that instantly conveys warnings in sign language when, for example, a special warning for heavy rain is issued. In addition, avatars can be used to express sign language in conjunction with the VTuber system. In this case, we expect communication without delay. In addition to communicating in sign language as a function, avatars can be made into avatars that users can nurture by incorporating personalities. We look forward to its existence as an interpreter to create a society without division.

The Deaflympics (Tokyo 2025 Deaflympics), a world-scale comprehensive sporting event for the hearing impaired, will be held in November 2025. KIKI has been selected as a supporting ambassador for the Tokyo 2025 Deaflympics, along with Neru Nagahama and Ikumi Kawamata. We believe this will be a great activity for us as interpreters to create a society without division.

It is said that there are 300 sign languages in the world. With the Deaflympics being held in Japan for the first time, I will do my best to become a bridge connecting people around the world and to realize a Deaf-friendly society" (from KIKI's demonstration video).

On the other hand, there are still many challenges in sign language DX. It is multilingual and there are differences even within Japan. How can we efficiently create a multilingual dictionary? Since it takes time to accumulate words using the conventional motion capture method, one alternative is to make it possible to capture the skeleton from the video images as well. This would make it possible to produce a dictionary simply by asking people to send us footage of sign languages from around the world. In addition to expressing sign language, there is also a need to develop technology to read sign language. We are looking for collaboration with companies that have the technology to read the signs, as it would be much more convenient if it could be combined with technology to convert the signs into Japanese.

The technology to digitize human gestures and skills from video in three dimensions is useful not only for reading sign language, but also for digitizing acting, dancing, and the craftsmanship of living national treasures for transplantation to avatars and robots. From the past era in which various contents were contained in a two-dimensional frame, if even documentaries can be recorded in three dimensions, their value as new records will increase. I feel that this will change the way content is created.

New contents brought about by the fusion of ultra-high definition 3DCG and simulation

Another thing I would like to talk about is the hybridization of ultra-high definition 3DCG and scientific simulation. This is a technique that was actually used in "Darwin Comes to Dinner." This technology has made it possible to recreate the ecology of dinosaurs as a nature documentary. It is now possible to create images that move as naturally as if the dinosaurs were alive.

The reproduction of dinosaur ecology is one example of how combining 3D data with simulation can expand the range of images. Nowadays, technologies such as scanning entire cities are being developed, which can be combined with simulations of changing weather conditions and disasters such as earthquakes to accurately represent events. This is a technology that can be applied not only as a program, but also in other areas.

One application is the experience of art and cultural assets. Art works and cultural assets are displayed in glass cases and cannot be touched, or the viewing direction is limited depending on the location of the exhibition. However, if it is a tea bowl, for example, you would like to hold it in your hand and look at it from all directions. These requests are realized with ultra-high definition 3DCG and real-time rendering utilizing the Unreal Engine.

The actual application of the technology is a simulation of the national treasure "Aburitake Tenmoku Tea Bowl" housed in the Museum of Oriental Ceramics, Osaka, which was made in China about 900 years ago and is so valuable that even in China, there is no complete form left. It was scanned in 3D, and the simulation made it possible to view it from all directions. It is also possible to see it in close-up, which is difficult with the naked eye. Digitization provides a new perspective. The images can be moved with a game controller, and a teacup-shaped "teacup controller" is also available for the Aburidome Tenmoku Chawan. When the teacup controller is touched and moved, the image in the teacup also moves in the same way, allowing the viewer to look at it from the desired direction.

Ordinary CG can only reproduce what you have created with set camera work. But with this 3DCG, you can view it as you like in real time. Also, light reflection and other factors can be simulated in real time, so you can see changes depending on the light source situation. For example, as a usage, we can reproduce the situation where an oil drop tenmoku tea bowl is placed in a tea room, how it looks when candlelight is lit in the morning, during the day, and in the evening, and how it looks when white tea is served, since there is research that tea was white at that time. These are techniques that can be used in the program.

Such use of 3DCG could also be applied to items that cannot be kept in actual storage, such as in museums. For example, it would be significant to scan and save high-resolution scans of industrial product engines, the streets of Shibuya, or excavation sites of cultural assets. We believe that thoroughly scanning everything from small to large objects and preserving realistic objects as accurately as possible will lead to content creation.

We also focus on reviving content. We are in the business of turning old video material into 4K quality in what can be called DX of the past. Two tasks are required when converting low quality images to high quality here: one is the removal of noise, which we do automatically by employing a video analysis processing program that removes it. The second is the appropriate grading process, which in the age of AI is still done manually by skilled designers. In general, AI processing is often used for high-definition image processing, but it is important to take into account human judgment depending on the content rather than having AI work on a batch basis. A simple example is that AI processing erases snow in the background as noise, whereas a meticulous human workflow can preserve the snow.

The 525 era (the SD era, when the NTSC system with 525 scanning lines, etc. was the mainstream), before the era of high-definition, was an unfortunate time for video content. Films before that time contained a lot of information, and a lot of information is still being recorded in HDTV and beyond. However, if 525 content is projected in today's environment, the quality of the content is not as good as it used to be. Under such circumstances, the enhancement of video content to high quality is an important initiative. One of the contents we have recently provided is a video of the Checkers. You can see that the old video has been revived with modern image quality.

I would like to make it possible to use a variety of visual content not only for viewing on TV, but also for interactive use and to show it in a larger space. I would like to create contents that are unique to people who have never seen anything like it before through DX and digitalization.