The immersive space morphs and alters with light and becomes solid, its pressure composing and decomposing the self-awareness of skin. The dancer’s body is extended and manipulated as a conscious entity, exceeding the physiological object. The constant feedback between the body's trajectory and interaction with the environment changes the nature of the object itself.
produced by: Friendred
The embodied, symbiotic relationship of light, sound, and space is central to this performance. The performer uses their body as a self-aware agent to adapt to the surrounding sensory environment. The piece crosses the intersection of technological interventions, architectural fabrication, and contemporary dance performance. Forty laser beams controlled by machine learning algorithms read and encode choreography. Synergies of futuristic lighting and surrounding screens create a brief detachment from reality. Computer vision is utilised with computational precision along with the control of body movement.
Concept and background research
Phenomenology and embodiment
Key to understanding the context of Skin-awareness is the nuanced relationship between neomaterialism, phenomenology and embodiment in HCI. Paul Dourish incorporated Merleau Ponty’s philosophic perspective into human computing interaction and early interactive performance. Dourish was not the first to apply phenomenology to HCI but was the first to reach a wider audience. When we incorporate embodiment into performance, another fundamental aspect that should be considered is information selection. Information selection, refers to the information that is produced during a performance, for example, when a human subject’s body physically interacts with a material and there is information feedback between the two entities. However, the gap between phenomenal embodiment and information expressed from the performance can be filled with improvisation. It includes the redundancy and performativity of the machine and creativity from the performer’s sensorium.
Merleau-Ponty, the French philosopher, makes the distinction of two perspectives of the human body, the third person perspective versus the first person perspective. The third person perspective describes the body as a physiological object in the world, whilst the first person perspective represents the body as a self-aware entity, capable of subjective cognition and perception through interacting with surrounding tangible objects and environments. This means that individuals perceive objects by interacting with them for what they actually are but that the complete experience depends on the individual’s own perspective. There is constant feedback between the individual and the object and each individual’s unique approach to interaction with the object may change the nature of the object itself.
Merleau-Ponty said “my body is transparent in use”, people are not fully aware that the body actually exists until it stops functioning correctly. This phenomenological framework had fundamental influence in early HCI and has been extended to include multiple aspects of exploring the relationship between users and artefacts. Following on from Merleau-Ponty’s argument of “lived bodies”, Toombs described another example of the phenomenal and objective bodies.
“For the person who routinely uses a wheelchair the device becomes a part of the body. One intuitively allows for the width of the wheels when going through a doorway; one performs the necessary hand/arm movements to move forwards and backwards without thinking about it. With habitual use the wheelchair becomes an extension of one’s bodily range. Thus, when a stranger pushes my wheelchair without my permission, it is invading my personal bodily space.” [Toombs 2002, page 256]. [27,32]
People construct their own version of perception when their mind is trying to convert the information from the reality reflecting the scenes. The process of what information the human mind projects can be distorted by past experiences, education, cultural values as well as the specificity of the occasion and information itself.  When considering this notion in relation to haptic feedback and embodiment in the performance, the contradiction of the sense of somatic encounters and the complexity of human perception will be processed by the mind.
Most recently two experiences caught my eye. One was an installation I experienced at Uncanny Valleys of a Possible Future, Physical Mind, created by Teun Vonk (also exhibited in 2006 Ars Electronica). When the body encounters pressure, via gradually being squeezed between two inflatable objects, the experiencer feels more released on the contrary. The experiencer’s body is surrounded by a relatively compressed and secure space, giving rise to the feeling that the inflexibility of the body in fact increases the flexibility of the mind.
Most of the information has been processed and filtered through habituation. Our haptic and visible feedback or other sensations will be perceived in the context of habituation. Consequently, we will instead receive more information when the irrelevant data has been reduced, in this case with the exclusion of sound and visuals, and introduction of enhanced haptic sensation.
This piece has spanned an over 6 month development period. It includes literature review, background research, 3D rendering, concept design, modelling, prototyping, fabrication, software constructing, dance performance, score design and rehearsals, amongst other things.
Drawing | 3D rendering | 3D model in Fusion 360 |
Before I started fabricating the installation and in order to make sure the size and appearance are close to the design I engeneered in the Fusion and Cinema4D, the best way to do is the prototype. I, hence, made a 1:1 ratio mockup made by cardboard (12 pieces) and sliced by Fusion Slicer.
Fabrication in the metal workshop
I fabricated the metal structure consisted of two rings as well as the conjunctions, and top circle made by metal. via rolling machine, cutting, grinding, Plasma arc welding (PAW).
During the process of making 40 laser modules, I was planning to make each laser controlled by a MOSFET module to switch them on and off. Each one of the laser could also have the dimming effect as long as I could find a good way to communicate between two teensy boards. Because the dimming effect need to be pulse width modulation (PWM). Making 40 MOSFET module is quite a challenge as picture depicted below.
Laser & micro servo simulation
I found a way to deconstruct and reconstruct the data from Open Frameworks to Arduino via osc and byte transfering seperatly. This is a technical challenge as I had to communicate with 40 lasers, 80 server motors, two stepper motors and two LED lights strips witch has 600 pixels on it. Each laser module has three pins, which means I need a micro controller have 40*3=120 pinouts, so maybe I need another two MEGA? That was the question I asked myself, but lucky enough I changed plan in the end, I used servo drivers instead of connecting each servo to the pinout directly. Moreover, I made a laser simulation programme to visualise the movements of the lasers while sending the tilt and pan data to the each module. For the simulation programme screen recording, you can find in here.
Skeleton tracking & machine learning
I managed to finish 5 stages of laser patterns and controlled by the skeleton detected by the Kinect. You can see the GUI on the left screen, skeleton torso position, pan & tilt value as well as laser stages. The white mask in the video is me detected by kinect camera via openNI2 library. The skeleton has been using the NiTE library for this project.
This link shows how I was training different data set by NN (Neuro network algorithm) in Openframworks. This is a demonstration of how I am going to control all the laser patterns by training the body gesture data from dancers.
I spent a whole week to assemble 40 tilt pan modules, each module has five parts including connection part (pictures below), foundation, pan, tilt and laser holder. Each foundation needs to be drilled a whole for the wire from micro servo motors. It roughly took 2 month to print out all the module I needed, and some backups too.
It is also intricate when I was trying to allocate the position of the laser modules. First of all I will need to cut out the cardboard ring shape with the precise circle location on it, then I need to customise the individual shape of the connection part. Because I used cable tie for binding the light strips inside so that made the top cover has a uneven surface, which caused me to cut the individual shape for the connection of the laser modules.
Installing light strips
It is not just easy as connecting two 5 meter led light strips to the power and micro controller. I did spend some time on calculating the amperage and voltage usage in order to spread the power evenly over the 40 A 5V power supply. Otherwise the end of LED strip will have the dimming effect from the beginning.
5 screens projection mapping
Thanks Atau for loaning his small scale simulator and mac pro we set up in the research lab. It was really important to do some test before I actually can run the test on site. Thanks Blanca for the mad mapper induction. I got a fresh idea of how I supposed to control 5 projectors in the mad mapper, including the blending, mapping and masking.
Last but not least, I need to calibrate the origin accordination for each laser modules in order to match the same coordination as my OF sketch.
There will be abundant potentials of this piece, for example the 40 robots made by 80 servo motors can be further developed. The Neural network machine learning algorithm can be used in a way to communicate with the laser patterns and dancer’s body, the performance can be extended on the stage and theatre.
In my original plan, I made two stepper motors to control objects hanging underneath, but due to time issues I didn’t use them. Like the concept design, this can be a really powerful part of the performance in the future. The patterns on the programable light strips I installed inside can be changed at any time any point according future development.
Learning by doing is an essential principle of making a project. I made a very thorough plan for this project, however, there will always be more problems than expected during the process. Half of the time was used to try different ways to solve or to optimise the same issue, which is also a very precious learning process. For example, I was struggling to make XBee work so that I can control all the data from open framework wirelessly. Nevertheless, It worked as good as wireless when I used a 10 meter USB extension cable with power hub.
Another huge benefit I learned from my mistakes during this process is to never give up trying different ways of solving problems and to always try to view things from a different perspective to find solutions. This is illustrated by the fact that I was spending a huge amount of time to run the test of communicating between Arduino Mega in order to meet the demand of 80 data pins from 80 servo motors, but in fact after a week of trying, I eventually changed the plan to use 5 PWM servo drivers and using Teensy board instead of Arduino Mega.
Thanks everyone who helped with this project in the past 6 months.
Thank you Alex for proofreading and great support.
Thank you Luigi Ambrosio for choreography and perform in this piece.
Thank you Bianca Vrcan for performing.
Thank you Joy Lee, John Xie for producing the mucic.
Thank you Atau and Blanca for the knowledge of immersive research project and soft ware study.
Thank you Rita-Josy Haddoub for helping with calibrating the servo motors.
Thank you Roy, Valerio, Christopher Baker and Elias for the help with code issues.
Thank you Arturas Bond, Joseph Rodrigues Marsh and Darel Digregorio for cinema photography.
Thank you Nicolas and Ben for the help with metal work.
Thank you Laurie, Annie, Chris, Ben, Eevi, Petros and Howard for helping with installing
Thank you Amy, Laura for the help.
Thank you Dianiah for helping with making the mirrors.
Thanks my family.
Open frameworks Addon:
Thingiverse open souse: https://www.thingiverse.com/thing:2467743
Merleau-Ponty, M. 1962. Phenomenology of Perception. Routledge & Kegan Paul, London, UK.
Dag Svanæs, Interaction design for and with the lived body: Some implications of merleau-ponty's phenomenology, ACM Transactions on Computer-Human Interaction (TOCHI), v.20 n.1, p.1-30, March 2013 [doi>10.1145/2442106.2442114].
Smelik, A. (2010). The scientific imaginary in visual culture. Goettingen: V & R Unipress, pp.69-88.
Salter, C. (2015). Alien agency. Cambridge, Massachusetts: The MIT Press.
Toombs, S. K. 2002. Reflections on bodily change: The lived experience of disability. In Handbook of Phenomenology and Medicine, S. K. Toombs, Ed. Kluwer Acedemic Press, The Netherlands, 245--260.