Protean Visions: On Mixed Reality Performance

produced by: Yuting Zhu

Mixed Reality Performance: Introduction

In January 2019, The UK government launched an £18-million-worth of new initiative that looked at applying immersive technologies such as Virtual Reality (VR) in creative industries. Performing arts organisations involved in the scheme included the Royal Shakespeare Company, Almeida Theatre, Punchdrunk and The Space [1]. Many saw this as one step further on the forefront of “new theatrical vocabulary” [2] that incorporates digital technologies and interactivity, in which Mixed Reality (MR) performance is among the cutting-edge research fields. Nobody had foreseen then, that 2020 and its global pandemic would ever so abruptly push the demand for remote virtual experience under the limelight.

Putting dazzling technological terms like VR and AR aside, as Joris Weijdom points out in his 2017 research handbook [3], one may easily acknowledge that a theatre play is in many ways already a Mixed Reality experience. In fact, the definitions of Mixed Reality performances are broad. They often overlap with immersive, site-specific and multi-media theatre [1]. Surely it is not always necessary to name names and categorise, but one of Steve Benford and Gabriella Giannachi's opinions in their book on Performing Mixed Reality (2011) has inspired this research project to question and investigate.

Benford and Giannachi identify Mixed Reality performances as performances that mix the real and the virtual and combine live performance and interactivity, through the incorporation of digital technologies [4]. The authors also suggest that Mixed Reality performances can often simultaneously occupy multiple points along the "reality/virtuality continuum". The notion of a scale on the reality/virtuality continumm is inherited from Paul Milgram and Fumio Kishino in 1994 at the tail end of the first VR wave. Given the VR context, the "reality" end of the scale can be interpreted as the space where our physical bodies are situated, whereas the "virtuality" end is the digitally rendered, imaginary, and animated world. However, looking at it from today, it is questionable if a linear scale is comprehensive enough to encompass the hybridity of this emerging theatre form.

Perhaps a better way would be to describe it as a multi-faceted web. In an essay on animation-philosophy, Deborah Levitt references concepts from the pluriverse philosophy and the worlding theory to suggest that animation (broadly, including VR) does not recreate but produces "many worlds" based on their "various material-technical conditions of possibility, as well as their aesthetics" [6]. Same claim applies for theatre, whose various verbal, visual, acoustic, as well as other sensory and interactive languages fabricates a web of parallel realities to encourage its audience or participants to suspend their disbelief, to experience, to empathise, and to speculate.

This research project thus aims to explore: how Mixed Reality technologies and theatre performances complement each other in their potential to open up and bridge multiple realities in the specifities of people and contexts engaged. Particularly, a comparative case study will comment on three recent productions through the lens of parallel worlding, and based on the findings, a research artefact will explore the potential of multiple worlding as a creative model for applied theatre in psychotherapy.

Mixed Reality and Multiple Worlding: Comparative Case Studies

The concept of "many worlds", first developed by decolonist theorists as a way to think beyond the discourses and narratives of Western universality, shares some of the same values with Material Semiotics' research approach. The latter seeks to see things not as "what" they are but "how" they relate and interact with other substances. Each way of seeing is inevitably and intendedly subjective, suggesting our world exists as multiple parallel "realities", denying an objective, impartial, and singular "reality"[8]. This seems an apt methodology for this project's research topic, therefore I will seek to weave thoughts together in the form of case studies as follows.

1. Dots, Lines, Cubes - Mum&Gypsy

To identify in what way the Mixed Reality approach succeed or fail at complementing performances, I'd like to start with a production that does not involve VR technology, but live filming and projection.

This is a 2013 play from the Japanese theatre company Mum&Gyspy, titled Dots and lines, and the cube formed. The many different worlds inside. And light. (hereafter shortened as Dots, Lines, Cubes ) which highly correlates with the idea of fluid, overlapping, and interconnected realities. 

By setting up cameras of various focal views and fluid orientations on the stage, the play weaves together multiple isolated events that would otherwise be dwarfed by the main actions happening on stage (e.g. a tiny toy dinosaur being moved, an irrelevant girl in the background tossing balls, visuals of weather from a country indirect to the story). You can't often tell if some of them are virtual, as in symbols or metaphors; or are these montages from a different time and space.

All of these elements are, quoting the play, "dots" that can in a way be connected into "lines", then formed into "cubes", and eventually woven together as "many different worlds inside". The assemblage of these instances, cooperating with the episodic plots and dialogues that recurs, breaks, and reconnects in the way of poetry, mitigates the notion of "central" on stage. In material semiotics' vocabulary, the play generates multiplied, overlapping realities and thus illustrates a world view of the specificities. This seems a perfect example of technology, not necessarily VR, fitting in a performing arts practice as an indisposable module to embody the theme, which does not distract, but strengthens one of theatre art's core missions: generating new realities.

However, without the VR technology present and thus greatly weakened (in fact none) interactivity, what the audience perceives is a multiplied world view pre-composed by the creator. They're still the traiditonal spectators (as opposed to participants) in theatre, not receiving the best an Mixed Reality performance can offer - free room to construct and interfere with your own perceptions.

2. Anti-Gone - Theo Triantafyllidis

On the contrary, the second production plays with interactivity in a creative way. In particular, online interactivity. Theo Triantafyllou's multimedia theatre/film/performance Anti-gone was live-streamed on YouTube as part of the 2020 BFI film festival. With avatars enacted by actors live and visuals powered by a game engine, the piece exhibits a post-climate change world where environmental catastrophe has become the norm, cities are sunken, and a culture of late-capitalism still runs rife. [9] 

Anti-Gone’s protagonists – Spyda and Lynxa – glide through island life to shopping to movies and to psychedelic drugs, navigating a world on the verge of becoming nilisticly dystopic. Exploring a multi-layered experience which combines live performance with digital content, this performance "shifts between our imminent disastrous future and the constructed virtual present". [10]

The sense of constructed reality permeates the production, from the comically glitchy VR figures, to meta references, and to (in the film version, particularly) the improvised lines picked up from the live chatbox. The audience watching the piece via a livestream platform input their preferences on the character's back story, pet name and shopping choices, while the actors randomly insert the chatbox comments into greeting lines or background tech noises (e.g. the voice assistant in the mall). This practice creates an intimate and relevant medium for parallel worlding, which steers reflections on the notion of 'present makes future', and critically towards the worrisome ecological, technological, and socio-political realities.

At one point in the film, the protagonist Spyda asks Lynxa to close her eyes and imagine a scene for herself. The visuals fade to an empty abstraction of shapes, presenting a blank canvas to the audience. It is particularly impressive, in the sense that even in a world made from illusory rendering of the imagined future, it still encompasses room to invite free and individualised constructions of new realities - an ability that is voluntary, born-with, and transcends any medium (game, VR, theatre, etc.). It resonates well with how Matthew Doyle, producer of the staged version of Anti-Gone, once summed up his intention and interest as "how do we break these technologies down, to create a shared experience among a live audience" [9].

3. Dream  - Royal Shakespeare Compay

RSC's latest production Dream is another a motion capture VR performance, inspired by A Midsummer Night's Dream. Premiered in March 2021, it is the most techonologically sophisticated among the three cases, yet most disappointing. By all means, its animated VR wonderland is a meticulous design and breath-takingly beautiful, but which perhaps accounts for its relatively short duration (30 minutes) compared with a traditional staged play or the YouTube livestreamed Anti-Gone. 

Duration is not the only cost for a virtual spectacle – narrative and interactivity are compromised as well. The plot, about a storm sweeping through the forest where Puck and fairies live, is commented by critics as "paper thin" [11]. The only interaction designed is inviting the audience to drop a "firefly" with their mouse to light up Puck's way, which is minimal. Instead of inviting the audience to jointly fabricate various worlds of metamorphosis and wonder, this painstakingly detailed VR landscape compresses the whole thing in a single visual dimension.

If the point is only to visualise someone's imagination of Shakespeare, then good old theatre itself can do that, too – and with much more freedom. To quote from a critic: "It may be rendered by actors standing on a stage, but the mind's eye makes it something else. The audience suspends its disbelief in order to be enchanted."[11] In Anti-Gone, the undisguised glitches and deformities in the avatars constantly remind its audience that this is a virtual world (not pretending to be "real"), one that invites interference. Wheras in Dream, the picturesque life-like renderings of leaves and woods are only distancing the audience, like the glass shell over Mona Lisa, leaving them no choice but to stand and look. 

To conclude, if Dots, Lines, Cubes proves that an animated digital dimension is not necessary for a performance to create Mixed Reality visions; and Anti-Gone showcases the potential of creativity an MR performance can provoke from its "inter-realities" collaboration with the audience; then Dream reminds us that digital technologies do not always complement but can sometimes overshaow the power of theatre itself to provoke imaginative worlding. 

Refering back to the claim that animation and VR alike does not exactly recreate the physical world we are in – precisely because of this, a Mixed Reality performance cannot be, and functions beyond an "empathy machine". The worlds MR performances generate cannot replace the physical inter-personal interactions and contextual socio-geographical experiences. Instead, they open a doorway to step beyond an individual's bubble of existence – it could mean knowledge, experience, but also the limitations of one's perceptions and reasoning. Essentially, an MR performance outlines and helps you see the things you cannot see, perhaps even about yourself.

So the next question is, with its highly personalised advantages in opening up and bridging multiple realities, do MR performances have potentials in areas other than entertainment?

Multiple worlding as a thinking model: Applied theatre in Psychotherapy

Earlier this year I joined a volunteer trainee programme for a suicide-intervention NGO in China. During the time I learned about a practice called narrative therapy, often used for helping people who suffers from recent traumatic incidents, mild to moderate depression, or compulsory behaviours. According to the rescue experiences in the past, a lot of the dangered people experience a loss of ability to believe or envision that a situation is able to change, progress or develop in an alternative direction. They are often cornered in a self-reinforced narrative about their misadventure or personality, which convinces them that there's no other way out but to hurt themselves. Narrative therapy deals with this mentality by helping a person to reconstruct an alternative reality that serves them in a better way. Common techniques include characterising one's emotions as a third person, or identifying unnoticed values, intentions or social impacts that inform one's dominant story.

Uncoincidentally, applied theatre also involves building scenarios and helping an individual or community to identify certain behaviours for constructing or reconstructing a narrative. Its various fields of work include community campaigns, criminal justice system, and of course psychotherapy. This led me to a speculative attempt at applying Mixed Reality performances to the narrative therapy practice – one that highlights the plural, flexible, and progressive worlding process produced by an MR performance, and dedicated to helping someone create a "protean vision" towards their present and future situations.

Therefore I wanted to develop a research artefact as a tool for creating such a pseudo performance. In this pseudo performance, participants can access and interfere with "worlds" envisioned from a pluriverse of storylines. In order to break out from the humanly limitations of one's perceptions and reasoning, why not ask for an AI's vision? In terms of a generative creative approach, AI has advantages in learnability, self-evolution, and controlled unpredictablity. Given the right design and training, an AI can theoretically cover infinite possibilities of scenarios.

Within the constraints of time and conditions, I chose the form of a 2D text-and-image-based generative storyboard to demonstrate the design idea of my pseudo performance. Below is the technical implementation of the artefact:

- Research Artefact: A pseudo pluriverse storyboard generator

To make an automated generator of scenarios driven by AI's visions, I applied Machine Learning techniques to create a text-image-text feedback loop.

First step, the model charRNN is for used for text generation. Unlike a simple next-word-predictor on the keyboard of our phones, charRNN used a Long-and-Short-Term-Memory (LSTM) model for neural network training. Simply put, it can "memorise" a much longer paragragh of words, and is able to refer to the previous contexts when needed. The longer it writes, the more complex the soil is from which its seed grows. Therefore charRNN is good for continuing the writing of a short story or dialogues, which is what I needed.

Then I wanted to visualise the evolution of the pseudo "storyline", making a storyboard for the performance. The second model attnGAN is perfect for the job. It analyses a given text (e.g. This is a red bird with a very short beak.) and generates an image based on that description. Because this model is trained on one of the largest free image dataset of daily objects, it performs well on key words like cars, trees, cats. But when it gets to more abstract subjects or when the sentence doesn't make full grammartical sense, the output "sketches" start to appear eriee, fluid, and surreal – and I fully embrace this aesthetics.

Now that I have a text-image generator of "scenarios" that evolve on its own, it's time to think about interactivity. In a pluriverse or many worlds, a person is never just a spectator. Every minute reaction one takes in the current scenario affects the next, and every decision made splits up new branches on the possibility tree. I needed a simulator of "reaction", and im2txt model was a good fit - an image auto captioning model. When I embedded it into my generator, it regularly interprets the current sketch of scenario into text and feeds it back into the soil of the text generator. Thus a text-image-text feedback loop is formed.

This is a demo of a trial run. I input the programme with the first sentence from Franz Kafka's The Metamorphosis, and set off the text generator trained on Roberto Bolaño's writing style. Here goes:

Here is the result video that highlights the plurality of possible "scenarios": 

Ideally with further development, I hope I'll be able to take these AI's visions into a VR world, embody them with human performances, and incorporate actual interactions with the audience. Again, an MR performance may never be able to recreate the physical reality or replace necessary medical treatments and actions taken in the material world. But I believe its potential in creative worlding works as a catalyst for a shift in ideas and thus cognitive choices. Within this emerging and dynamic thearical genre, there are still "many worlds" to explore.

Biliography

[1] "RSC, Almeida and Punchdrunk participate in government-funded immersive entertainment project". WhatsOnStage. 2019. (https://www.whatsonstage.com/london-theatre/news/rsc-almeida-punchdrunk-virtual-reality-government_48278.html)

[2] "Transmedia Storytelling In The Performing Arts, A New Grammar To Be Learned: An Interview With BelÉn Santa-olalla". (https://thetheatretimes.com/transmedia-storytelling-performing-arts-new-grammar-learned-interview-belen-santa-olalla/)

[3] Weijdom, Joris. "Mixed Reality and the Theatre of the Future". International Network for Contemporary Performing Arts, Brussels. 2017. (https://www.ietm.org/en/system/files/publications/ietm_fp_mixed-reality_march2017_1.pdf)

[4] Benford, Steve, and Gabriella Giannachi. Performing mixed reality. Mit Press, 2011.

[5] Milgram, Paul, et al. "Augmented Reality: A class of displays on the reality-virtuality continuum". Telemanipulator and Telepresence Technologies. Vol. 2351. International Society for Optics and Photonics, 1994.

[6] Levitt, Deborah. "Rendering Worlds: Animation-Philosophy, Cosmotechnics, Conviviality". https://blog.animationstudies.org/?p=2746&fbclid=IwAR1KN5xr-GurjWQArlUaMrhIaENG8ASxXqYBC7jMnbFXrBLHoP9-5fgezB4

[7] Buchan, Suzanne, ed. Animated Worlds. Indiana University Press, 2007.

[8] John Law. "Material Semiotics". (https://learn.gold.ac.uk/mod/resource/view.php?id=915772)

[9] "2020 Sundance Film Festival: New Frontier Line-up Announced". (https://www.sundance.org/blogs/news/2020-new-frontier-announced-festival)

[10] "Anti-Gone is a mixed reality performance set in a post-climate change world". (https://www.itsnicethat.com/articles/theo-triantafyllidis-matthew-doyle-anti-gone-digital-180919)

[11] Crompton, Sarah. "Review: Dream (RSC, online)". (https://www.whatsonstage.com/stratford-upon-avon-theatre/reviews/dream-rsc_53590.html)

[12] Monk, Gerald Ed, et al. Narrative therapy in practice: The archaeology of hope. Jossey-Bass, 1997.