Creating new video content by utilizing local 5G and real-time rendering technologies" Tetsuya Fukuhara (NHK Enterprises)
-5GMF Millimeter Wave Dissemination Promotion Workshop / Japan's Industrial Technology Frontline Report

The 5th Generation Mobile Promotion Forum (5GMF) held the second workshop on "Japan's Industrial Technology Frontiers" to promote the spread of millimeter wave in January 2024. The aim was to have a forum to interact with people outside of the broad telecommunications industry, and the second session featured speakers from NHK Enterprises, a company in the broadcasting and video production field. Toshihiro Matsumoto, Managing Director of the company, and Tetsuya Fukuhara, Executive Officer and General Manager of the Innovation Strategy Office, explained the relationship between content production and DX. The second part of this report will report on Mr. Fukuhara's lecture on DX in the field of content production.

Organic connections to create businesses using technology

I would like to present some examples of NHK Enterprises' DX efforts. These are new technologies, but I think the biggest challenge is how to turn them into businesses, and I am sure that is your concern as well. There was a time when it was said, "Technology wins and business loses. I would like to consider the creation of businesses as another issue that cannot be solved by technology alone. In creating a business that utilizes technology, we need projects that are organically connected one by one. We believe that partial optimization does not make a business.

There are two types of DX in content production that will be introduced: one using local 5G and the other utilizing real-time rendering. The former is "DX of Production Workflow with Local 5G: Freeing Production from Cables" and will discuss how local 5G is being utilized at NHK Enterprises. The latter is "Advanced Spatial Production with Real-time Rendering - Beyond The Frame: "Super" Expanding the Video Experience". Real-time rendering in the venue will enable the design of various video displays in the space. Both of these initiatives will not only expand the range of expression of content, but also lead to DX workflows.

Local 5G to expand the range of production and content

Local 5G is already being used to increase the efficiency of field operations in agriculture and industry. At NHK Enterprises, we thought of applying this local 5G to enhance staging and other applications.

The first phase was a demonstration experiment conducted at LINE CUBE SHIBUYA (Shibuya Public Hall) in FY2021. What we did here was to increase the flexibility of the cameras by using wireless cameras connected via local 5G. When we deliver or broadcast live on stage, we use a multi-camera configuration, but usually all cameras have cables attached to them, which restricts the range of movement of the cameras. Without cables, the cameraman can move freely and shoot from angles that are not possible with cables. The director will have various requests for filming, and the trial is to free the camera from the cables by utilizing local 5G to achieve more freedom in the direction of the production.

In the demonstration test, we tried a shooting configuration that took advantage of the increased flexibility afforded by local 5G. On stage, the cameraman used a study cam by himself. Since there were no cables, there was no need for a staff member to handle the camera, and the cameraman was able to shoot while moving freely among the dancers. We also used a remote-controlled camera that could be operated remotely. A rail camera that can move freely between the audience seats and a rope camera that can be suspended from the top were used to achieve angles and camera work that would not be possible with cables. In the demonstration test, we used a single base station to wirelessly send images from these cable-removed multi-cameras and confirmed that they could be switched live and transmitted without delay.

In this experiment, it was confirmed that HD video could be transmitted simultaneously with nine cameras; the goal was to be able to transmit about five cameras at 4K resolution, but at this point, the challenge was to secure even more bandwidth through the uplink.

In the second phase of the 2022 demonstration, in addition to the uplink transmission on the indoor stage in the first year, we added several requirements: one was to increase the information to be placed on the local 5G. in the first year, we only transmitted camera images via uplink. But on set, a variety of information circuits are essential, including a "send-back" that shows how the switcher is selecting screens, and an "intercom" that conveys remote conversations between crew members. We wanted to pull it all together and put it on the local 5G, which demonstrated the expanding range of applications for a cable-less system.

The second is to expand the wireless area. We decided to test whether the system could be utilized outdoors rather than indoors, and demonstrated a low-cost, flexible area design using small base stations and multiple distributed antennas (DAS). Another technical point was to examine methods to expand the uplink bandwidth by using semi-synchronous stations.

While examining these technical issues, we aimed to create new content experiences and value. Specifically, we built a local 5G area using two base stations in Workstation Edo (Tsukubamirai City), a location for period dramas and other productions managed by NHK Enterprises. There, we took the "Oshitama Rantaro" musical stage outdoors and took on the challenge of creating new content.

Two zones were created at the site, and the content is a story that progresses concurrently, eventually converging into a single drama. As part of the "creation of new content value," three types of content were created from a single source: one was a real-time viewing experience that provided live-streamed content at the venue on the day of the event to the smartphone in hand; the second was an event experience to witness the progress of the drama; and the third was the story of the two zones is a fusion of the two, and the complete version of the work is provided at a later date through archived distribution or DVDs. By using local 5G, we have achieved multi-use of one source for three different contents. This is an example of a demonstration project that has advanced the study of commercialization, monetization, and business value creation through the use of technology.

Realizes 16:9 out-of-frame video expression in real time

The second case study is the advancement of spatial presentation through real-time rendering, based on the concept of "Beyond The Frame," which aims to "super" expand the visual experience by stepping out of the conventional 16:9 screen frame. While technological advances have led to higher resolution images and displays, the conventional format for trade fairs and exhibitions is to separate corners within a booth and use monitors to showcase images. We challenged the method of variable spatial presentation of the booth in various modes by rendering in real time within the venue.

That is the case of the "Super Experience, the Future Pioneered by Media Technology," which was exhibited at InterBee in 2023. Multiple LED panels were rendered in real time to create a spatial presentation. In the conventional site, if there were 10 exhibit contents, the booth would be subdivided into 10 booths, each with a monitor and an explanatory staff member providing explanations. With this method, explanations could only be given within the range of a 16:9 screen. In contrast, with this method, multiple screens can be combined in various ways, making it possible to present using the entire space.

For this exhibit, we have created a system that allows the site to change variably into various modes. The idea is that if there are 10 projects you want to introduce, you can explain all of them by changing the time of day and the mode. The system can be changed dynamically, and we have prepared four modes: "PV mode" to play promotional videos, "interactive mode" to experience the Aburidotenmoku chawan and Van Gogh's sunflowers, "show time mode" for presentations, and "explanation mode" to introduce individual projects by utilizing multiple sides of the panel. Each of these can be switched in real time, so that when someone comes in who wants to listen to KIKI, the sign language avatar, you can change only one side to that explanation, or you can use the interactive mode to give a large demonstration of the Aburidotenmoku chawan for a VIP who suddenly arrives. The ability to switch between the two instantly makes it possible to create an exhibit that is both interactive and real-time.

In addition, the workflow can be dx'd using this exhibit mechanism. It changes the way you create. First of all, as a pre-visualization, you can simulate the video display in advance. Instead of not knowing what the exhibit will look like until it is put together at the venue on the day of the event, the contents can be confirmed through simulation. Also, it is easy to replace the contents on the day of the exhibition. The production process can be carried out in parallel at the same time, and there are many chances to try out the simulation, which leads to improved quality of creativity.

As experiences become more spatialized in the future, I believe that the on-site need for this type of workflow DX will increase. In addition, as experiences deepen, relationships with users will become more diverse, and projects will require more complex and organic connections. Both engineers and directors want to meet these user needs. In order to organically connect and manage projects with increasingly diverse stakeholders, it is also becoming increasingly important to have personnel who can facilitate projects while maintaining technological literacy. I believe that the role of project managers in facilitating increasingly sophisticated projects will become increasingly important in business development that utilizes cutting-edge technology.