An unconventional approach that spawns new forms. Possibilities in new live entertainment that fuses traditional performing arts with mixed reality —Report on MR kabuki viewing event—

This blog post is an abridged translation of the English translation of the blog post – “型破りは、やがて新たな型となる。伝統芸能と Mixed Reality が融合した新たなライブエンターテイメントの可能性〜MR 歌舞伎鑑賞会体験レポート〜 – News Center Japan (microsoft.com).”

A groundbreaking experiment using mixed reality (MR) technology was conducted at ”Japanese Ghost Story (J Horror) Kabuki” Sadako x Sarayashiki, a kabuki performance held at the Osaka Shochikuza Theatre from October 3 (Mon.) to 25 (Tues.), 2022. The experiment, which ran from October 17 (Mon.) to 24 (Mon.), involved invited members of the public who sat in special seats in the back of the auditorium to watch the kabuki performance while wearing Microsoft HoloLens 2. This allowed these audience members to experience a new form of theater viewing, in which subtitles and visual effects are superimposed on the actual stage.

Pursuing next-generation content with future entertainment in mind

“Shochiku has been working to expand possibilities in live entertainment, our core business, by utilizing digital technology,” says Masashi Tomita, manager of the New Business Co-Creation Office, Innovation Promotion Department, Business Development Headquarters, Shochiku Co., Ltd. The company is focusing on creating next-generation entertainment that fuses the real and virtual worlds, including developing applications for viewing kabuki using augmented reality (AR) technology and 360-degree video content.

The mixed reality (MR) kabuki viewing event was planned as a part of these efforts. “We examined how we could add a live element to the virtual and metaverse contexts, which have been attracting attention recently, and came to the conclusion that MR was one of the answers,” Tomita explains.

This kabuki performance featured stars such as Ainosuke Kataoka, Tsubasa Imai, Kazutaro Nakamura, and Kangyoku Nakamura. The collaboration between Banshu Sarayashiki, a traditional kabuki performance, and Sadako, a character representing modern Japanese horror, was a hot topic. Tomita himself admits that he loves horror content. “Sadako is a character that I have a strong emotional attachment to because she taught me the fun of horror,” he states. When he learned that Sadako would feature in the MR experience event, Tomita felt a sense of destiny and made up his mind to deliver attractive content to his customers.

The huge challenge of adding new elements to traditional performing arts

Upon hearing about the concept, HERE. Co., Ltd. CEO Masanori Doi, who was in charge of MR design and computer graphics production, says he initially was surprised by the idea of combining kabuki and MR. “But at the same time, I was very excited about the various possibilities that would open up if we could make this happen,” he says. As this was everyone’s first attempt and nobody knew the right answers, HERE. held discussions with Shochiku and partner companies to seek out suitable forms of presentation and expression.

During this process, it became evident that many aspects could not be visualized using the script alone, and the work continued even after the curtain went up on the first day of the show. One of the most difficult challenges was to select and choose the right information. “The brain is better at perceiving visual information than auditory information, so after actually overlaying MR components on the stage, some information was removed, while other information was added. For some parts, we decided that expression would be more effective with images rather than subtitles. We did not simply use MR for visual effects, but also tried to remain aware of the amount of information that would be comfortable for the viewers, and tuned MR components accordingly,” explains Doi.

“This time, we tried to express things that could not happen in the real world by overlaying MR on the scenes where the Onmyoji magician casts a spell or a performer strikes a flamboyant pose,” he continues. “I hope to receive much feedback on how viewers feel about our attempt to add new elements to the complete stylistic beauty of kabuki,” Doi says, feeling both excited and nervous.

The initial struggle to fuse MR and real life

The company responsible for building the MR system is Kadinche Corporation, a firm which develops spatial representation technology. The company has been involved in numerous collaborations with Shochiku, mainly through the activities of Miecle Inc., which is a joint venture between Shochiku and Kadinche.

Kadinche has developed two main systems for this collaboration. One is an application that displays various visual effects and subtitle guides on HoloLens 2. The other is a simultaneous display system that allows HoloLens 2 to play back this information from a management terminal. Production manager and technical director Akinori Inada notes that this system was the key to the success or failure of the project. “Kabuki is a live performance. Pauses can change subtly, even during the same scene. Our most important mission was to construct a system that could send the appropriate information to all the HoloLens 2 devices in the audience, and to ensure stable operation.”

The team also struggled to construct a mechanism that displays objects in the same location regardless of the seat they are viewed from. “Initially, we thought of arranging objects without regard to the positioning of seats, but when we tested out this approach, we realized that the view would be very different depending on the seat, so we changed to a system that displays objects according to seat position,” Inada explains. He praises the high functionality of the HoloLens 2, which allows objects to be freely arranged in space, and believes he would not have been able to realize his desired form of expression without the device.

Utilizing know-how that has supported kabuki for new challenges

To display video and subtitle information on the HoloLens 2 without interfering with the progress of the stage performance, help from experts who are familiar with kabuki was required. Cooperation was requested from Earphoneguide Co., Ltd., which has been involved in producing simultaneous audio commentary guides and visual subtitling guides for kabuki for approximately 50 years since the company’s inception.

“We believe that the use of MR and other digital technologies in theater will gradually become more widespread. Being able to accumulate MR-related theater viewing know-how at this time has great benefits for our company,” says Jungo Kato of Earphoneguide Co., Ltd., who was in charge of operations for the experimental performance. Earphoneguide specializes in technology to display audio and subtitles in time with live performances, but in addition to simultaneous subtitles in multiple locations, this project also included the adjusted timing of visual effects. To maximize the content displayed in the MR space, the company went through a trial-and-error process, making revisions on a daily basis.

“When I first heard about this project, I thought it would be interesting, but the actual work was much more difficult than I had imagined,” recalls Earphoneguide’s Ayumi Tsuchiya, who was in charge of subtitling. While viewing MR theater in a positive light, Tsuchiya says with a wry smile, “When I looked at the displayed text through HoloLens 2, it was smaller than I had expected, and the display location did not fit well either. During the performance, I was concerned about all the issues that needed correcting.” “Subtitles and visual effects appear in various sections of the MR space, making it easier for the audience to get immersed in the performance. We still have other tricks that we can include for added entertainment, so I think these developments can make kabuki more enjoyable even for those people who have not shown interest in the past,” Tsuchiya concludes.

Participants excited about new experience as MR kabuki viewing event kicks off

At first, participants were guided to special seats in the back of the theater to watch the opening act of the three-act kabuki performance without the use of HoloLens 2. This allowed them to better understand the changes that occur with and without the use of mixed reality. At the end of the opening act, each participant was handed a HoloLens 2. As this was the first experience for most of the viewers, staff present at the event helped them put on the devices. Looking slightly restless as they looked around and raised and lowered the visor on their HoloLens 2, some participants commented on the surprisingly light weight of the device, while others noted that their glasses might be a bit of a nuisance.

Next, it was time to adjust the “front” settings. Since theater seats face the stage at different angles, the HoloLens 2 must be set so to face the stage directly. When the participants viewed a printed QR code through their HoloLens 2 and a check mark appeared, they were able to complete the setup by tapping the check mark. For the participants, this setup procedure was their first experience with MR. As they viewed the stage and the superimposed title of the kabuki play appeared on the stage curtain, the level of anticipation regarding the performance that was about to begin was undeniably high.

When the curtain for the second act rose, the MR space displayed a description of the scene, allowing participants to seamlessly enter the world of kabuki. As the performance began, subtitles containing the characters’ names and profiles were superimposed above and below their heads and feet. This portion demonstrated the production team’s efforts to avoid interfering with what is happening on stage, which they said was a tough challenge to overcome.

During parts of the performance where unique kabuki turns of phrase may have been difficult to catch for beginners, explanatory subtitles appeared in a corner of the Holo Len 2 user’s view, allowing the audience to understand the content without losing focus of the story unfolding on stage. Additional subtitled commentary also allowed viewers to fully enjoy the surprises provided by the quick change of roles, one of the highlights of kabuki.

And when takemoto (gidayu-bushi) narration, another unique kabuki component in which the story is told in verse, began, the lyrics were displayed to match the inflection of the verse. This is precisely what HERE.’s Masanori Doi refers to as “visualization of voiced expression.”

©️Shochiku Co., Ltd.

As the story unfolded, the features of MR and the functions of HoloLens 2 were fully utilized to create computer graphics that represent blood sprays and witchcraft in scenes that are laced with ghost story-like terror and bizarre phenomena, and to add the sound of water coming from Okiku’s ears as she is water tortured in one of the climactic scenes. While being innovative, this production did not interfere with the actual performances conducted on stage.

The final curtain eventually descended, and the theater resounded with thunderous applause. As the MR participants removed their HoloLens 2 devices, they shared their impressions of the new theater experience, commenting on aspects such as the clear explanations and the interesting incantation scene.

New entertainment possibilities where professionals combine their strengths

“We took an unconventional stance by superimposing mixed reality on traditional performing arts, but if people show more interest in this out-of-the-box approach, then it will become the norm, and a new form of entertainment will be born. I hope that this experiment has given us a glimpse into a potential new standard for kabuki,” HERE.’s Doi says. It may be that these event participants witnessed the birth of a new type of kabuki and stage expression.

Behind the scenes, each collaborating company exerted plenty of effort in pursuit of a method of expression that was innovative, but which did not destroy the world as seen from the perspective of kabuki. The operators at Earphoneguide ensured maximum effect by displaying all video effects and subtitles at the best possible time in parallel with the progress of the stage performance. “I think our decades of experience are paying off,” Kato says proudly. It goes without saying that, if the best possible timing is to be achieved, Kadinche’s simultaneous playback system must be working stably in the background. HoloLens 2 also provided strong support by being able to clearly place MR objects in precise locations in the special theater setting.

Shochiku’s Tomita concludes the interview by stating his company’s big goals. “This demonstration experiment was subsidized by the Agency for Cultural Affairs under the fiscal year 2022 Japan Cultural Expo innovation-type project, but our goal is to present kabuki and other traditional Japanese performing arts combined with technology mainly to overseas visitors to the Japan International Expo to be held in Osaka and the Kansai region in 2025. Kabuki theater with MR is one of the possibilities, and we would like to continue working on this combination as a way of expanding Japanese content and technology overseas.”

This project was realized through the synergy of highly specialized companies bringing their technology and know-how together. Using this achievement as a benchmark, we will continue to embrace the challenge of creating entertainment that can be enjoyed by people around the world.

All content on this page is accurate as of the date of creation and is subject to change without notice. If formal internal approval or agreements with various companies are required, nothing will be finalized before such approval is given or agreements are concluded. Additionally, some or all of the above may become difficult to realize due to various reasons and background circumstances, and may be changed or completely canceled. Your understanding is appreciated.