This project is a further develop project for creative coding workshop term2 'SPYOO'. It is aiming to create an immersive storytelling live performance by face hacking technology. Based on the experience of Schizophrenia patient's family, evidencing the patient suffering between reality and illusion becomes the narrative of the performance.
Pantomimus was a masked dancer that performed in the ancient Roman theatre, his performance lack speech and the communication during the scenes was made through dance and body movements. Based on that, in this performance, the audience will be immersed in the storytelling about schizophrenia, the performer’s dance will be enhanced with face hacking technology and projections.
Schizophrenia is still a very misunderstood disease, people tend to discredit their realities or even show no interest in knowing it. However, it is still a very sad illness that affects schizophrenics but all of their friends and families. People that suffer from mental disorder interpret reality abnormally. It may cause delusions, disordered thinking or even hallucinations, impairing day to day chores. People who do not have the experience with schizophrenia patient might think that they want to get attention or might be possessed. However, it is not what they can control. Schizophrenia is a severe mental disorder in which people interpret reality abnormally. It might cause some combination of hallucinations, delusions, and extremely disordered thinking and behaviour that impairs daily functioning.
The experience integrates a computational instrument with the performer recreating the original story through the non-speaking performance. The goal of this project is to bring the audience to experience what is the point of view of a person who is going through this kind of moment. I want to bring awareness to the suffering of those patients and the ones surrounding them.
Drawing the pipeline for the project, I started from the small one and transfer into a larger one. Eventually, it becomes a collaborate project with mask design, performance and music composing. I reference a lot of contemporary performance artists such as Gillian Wearing, Chris Burden and Franko B. Base on those amazing artist, this project is tempting to grab the fear and the painful images brings the awareness of the mental disorder.
The Face Hacking code that I reference from DBraun’s TD-FaceCHOP in touchDesigner. By using this code, need to hack into the installation of the OpenCV, dlib and cmake. However, the process of installation dlib and cmake wasn’t really well.
For pedromartins face tracking algorithm, it only supports 68 points of the face landmarks which start from the eyebrows to the chin. To create a bigger canvas of the face, I create a few points base on the original vertices and add the new order of the face meshes. This not only allows me to have more canvas to project visuals but gives the audience more easily to see what is on the face projection.
In this project’s visual reference from some face projection mapping project and surrealism artist. There are 5 phases of the visual base on the different feeling of the character been through and building up the emotion of the narrative.
1. Opening - Telling the audience to prepare the show, slightly show a few images.
2. Refraction – Refract the true feeling of the patient, showing the other part of them.
3. Projection – Start to project others on themselves.
4. Madness – Out of control, having a delusion.
5. Confusion – Feeling stable sometimes but still get confusing of reality and illusion.
The setting of the visual scene is reference form my experience of seeing the patient from all different phases of the symptoms. For the Refraction part, I chose to use a more harmonious way to express something that is more hidden behind the face. In this effect, you can see many combinations of different expression refracting. Furthermore, creating a more surrealistic image of the illusion of feelings.
Jump to the projection part, I used the generated photos images which they are using AI to generated human faces. Furthermore, I used these image as input data for the styledGAN in runwayML with my performance face. This present that the patient is using their perspective to project others identity on themselves. Therefore, these fake faces express they are twisting the truth into their imagination and pretend the perfection life that they think others have.
The Madness is trying to recreate their paranoia feeling when the symptoms get worse. They suffer from losing control of their thought, even with the hallucination. They are hearing the voices which are not real. Many of the scenes, I used the black jelly material effects to present the fear of the patient. During the performance, when it shows the black part not only give the audience a scary feeling but the patient is also scared and fear of losing control.
For the last part, Confusion is illustrating the most painful process that I see from the patient. Sometimes the patient will get more stable when they are taking pills on time. However, even with pills they can still out of control and go back to the madness. They know there is something wrong with them but they just cannot control.
Cover by hand as a protective mechanism images, shows we all want to hide our true feelings. Although even cover by hand, the lunatic expression still exposes to the public.
2. Mask Design
Mask using different visual convergence and materials to develop the characteristics of the main role who suffered in disorders and delusions in a mechanical vision. The components of this role have been extracted and revised through the following testing into several versions.
The portrayal that we set first is mainly about a mental illness patient with abnormal acts. Artificial treatment sometimes can be cold but humans have been holding temperature or warmth both physically and mentally, which means the outfit should work with the natural body itself to resonate among the audience. Also, following the fact that her personal ideation is influenced by others expectations and social norms, the face mask is designed as a platform to project those conflict gears. Therefore, the performer's face is repeated by moulding with preserving details as an artificial shell but leaving the true self in vague, wrapped, even breathless.
For making the process more flexible, me and my performer chose to combine with choreography and improvisation. The way we start for the perform, I tried to brings the narrative of the character to the performer. Then the performer starts to project the character to herself and base on the references we found to improvise. After all, we look back to some recording discuss which part of performing is fit with the effects and the narratives.
It is a challenge for my performer to do this performance. During the show she ha to wear the mask that we made, so she cannot see and have to make sure her face is facing in front of the Kinect, fixed her feet in the specific space. This means that she can only react with the audio and the memory of the visuals.
The bondage fixed the mask is a bit tight hurt a bit of her face.
For the rehearsal, we use zoom meeting to remotely rehearse. To let the performer know what is the visual going to project and let her know how the face hacking work, I screengrab the zoom meeting screen and make the screen as an input to track the face and put the filter on her and spout out the output to the zoom.
I have a lack of experience of music composed, so I found many references to the audio such as 404.zero and many different samples of the audio. I wrote a draft of the timelines and cues also the transition of the different scene to music composer.
It is also a difficult part that matching the visual with the audio, piece different part together.
I keep testing different visual effect and the images with similar feeling audio that I’m looking for then sent it to music composer. Base on the reference we got and the visual, timelines draft, music composer create the audio that fits the narrative. For some parts of the music, I gave the concept of the visuals might be and what I want to her, and she creates the audio first then I base on the audio creates the images.
Basically can add many effects through 3D model and the 2D images, one of the tricky parts is the computational expense. Face tracking takes extremely expensive GPU. Originally, I want to create the full real-time render face hacking visuals. However, for the stability of the whole show and the frame rate I choice to do the half real-time effects and half pre-render visuals.
After all, these visual effects still made with real-time performance.
Mask and the computer vision
The trickiest part is to make sure the computer can still recognize the mask as a human face. I tried many devices and the illuminator but the algorithm that I used still not that sensitive for the detection. We made many version of the mask, but some idea is being cancelled due to the detection.
Two critical part of this project is the IR camera and the projector. I tested many types of equipment such as realsence F200 D435 and the Kinect V2 to see which one is the best in the dark.
The realsence can actually get more clear images in the dark and it works really well with the infrared illuminator. I found out that two wired things in realsence D435 and KinectV2.
The D435 is designed to do the motion capture and the depth images, so when I turned the infrared function, it will shot many points of dot to interrupt the images. To grab more clear images in dark it will also absorb the visible light which is a big problem when projecting on the face.
Kinect V2 works perfectly, but it can only absorb the infrared light from the Kinect itself, it will not be impacted by the illuminator that I use. This means the object needs to be more close to the equipment to be detected.
In this project, I think it is really fascinating to see how technology gives a more immersive experience to the live-performance. Face Hacking creates more dynamic expression to the audience. This brings the big white screen to the performer’s face like people can tell the character’s story through their face.
Meanwhile, I’m collaborating with a fashion photographer that experimenting the face projection mapping on the model for a collection. This project is using 2D UV mapping for the texture for most visuals. However, I would love to jump into the next level for 3D tracking, I already create some 3D face rigging to experiment with the 3D facial motion capture through the FaceTraker. I would also want to try different program tools for face hacking such as SparkAR, Blender, Unity and Unreal4.