Sensory Response Systems:
Embodied performance

Ryan Jordan
Module: Creative Technologies and Arts Practice
Supervisor: Janis Jefferies

MFA Computational Arts
Goldsmiths Digital Studios
Department of Computing
Goldsmiths College
University of London

ma701rj@gold.ac.uk
http://doc.gold.ac.uk/~ma701rj

07/07/2008

Abstract

Sensory Response Systems is an exploration into audio-visual performance using an array of sensors responsive to physical movements in order to control the audio-visual output in programmes such as pd and Max/MSP. It also looks at reshaping and replicating the body through the use of fabric, textiles and technology. We draw from inspiration and definition through reference to embodiment [1, 11], phenomenology [13], post-modernism [7], post-humanism [2, 3, 10], performance [4, 5, 6, 8, 9], physical computing [14, 18, 19] and computer sound [15, 20]. This essay establishes the context for Sensory Response Systems.

Introduction

There is a need in computer music and audio-visual performance for a more direct and immediate relationship and control over the sound and images being generated because at present the laptop performer is generally restricted to the computer interface. As far as I am aware the majority of interfaces available are the standard keyboard, mouse, and MIDI controller. More recently the Nintendo Wii has gained popularity. Although these interfaces are usable they leave a gap and are insufficient for full body expression. They lack intimacy between performer and the instrument (computer). As computers and components are becoming smaller and cheaper there is a growing interest in designing new interfaces and controllers for musicians, artists and performers. Sensory Response Systems seeks to create an alternative interface by treating the whole body as an instrument for audio-visual expression which will create an impact on the audience. The performer has a direct, full body contact with the sound and visual output through their movements and gestures. I also seek to reshape and replicate the bodily form with costumes and computing technologies in order for the performer to fully embody and become the instrument.

Becoming the instrument: Musical instruments and performance tools

What is the difference between a musical instrument and a performance tool? A musical instrument restricts the freedom of movement and theatrical expression. A performance tool enhances it but it still has its own limitations and restrictions. A performance tool can be seen more accurately as an extension and progression of a musical instrument.
Musical instruments are mainly related to breathing, plucking, strumming or hitting but what happens when we have ‘…some way of mapping bodily or vocal gestures into the flow properties of a sound’ [19]? We extend the instrument, via sensor technology, to encompass the body. Performance with new interfaces employs physiological-intellectual behaviour similar to performance with traditional instruments. With practice the performer will gain greater control over this new interface, as in traditional practice.
Wishart states ‘…We may group particular parameters and types of articulation into fields governed by rules. These operational fields may themselves be articulated through other inputs (physiological-intellectual performance behaviour or higher level rules) given by the composer’ [19]. The operational fields for a traditional instrument such as a piano will include rules such as the material it’s made from (i.e. wood), the ratio of tension between the strings, etc for the parameter of sound; and the operational fields for performance on a piano will include rules such as play louder, softer, quicker, etc for the physiological-intellectual interpretation of the piece. This may also be applied to new interfaces.
As well as taking into account the operational fields and physiological-intellectual behaviour it is important to acknowledge the perceptual or sensory feedback interaction between the performer, instrument and listener/audience. We will not look at it in too much depth as there is insufficient breadth for it in this paper.
Roederers’ [15] chain system of Instrument à Air à Listener, or Source à Medium à Receptor, is a very basic description of the flow of energy from the instrument to the listener and has been better described by Leman [14] in figure 1.

[Fig.1 Leman, 2007]
Here we enter into multimodal perception by incorporating haptic, visual and sonic feedback. We can see from the diagram the physiological-intellectual behaviour described by Wishart in the performers’ articulations and imitations of the musical goal transferred into bio-mechanic energy in order to operate the mediator, which is the instrument or interface. There is haptic, visual and sonic feedback received from the mediator with each carrying different bits of information back to the performer so they can make the next decision for action. The mediator also emits the sonic and visual output to the listener, who is also receiving sonic and visual information directly from the performer. This information forms the listeners’ perception of the performance. We shall now look at, rather briefly and within the word constraints of this essay, theories of embodiment to help us better understand this situation

Incorporating Practice and Embodied Performance

N. Katherine Hayles states in her book How We Became Posthuman [10] that incorporating practice is “an action that is encoded into bodily memory by repeated performances until it becomes habitual”. She uses the example of learning to type: “this person has repeatedly performed certain actions until the keys seem to be extensions of his or her fingers”. This extension forms part of an extended body-image for the user or performer as they have repeated actions so much that it has been encoded into their muscles and memory. This example is true when applied to learning a musical instrument, for example the piano. This notion of incorporated practice is supported from the previously mentioned operational fields of Wishart and the interaction and feedback diagram of Leman as there is physiological-intellectual behaviour and feedback with the instrument allowing the performer to learn the instrument. If we apply these techniques to new interfaces and wearable technologies for performance we can see how incorporating practice leads to the user extending their body-image in order to incorporate, interact and learn the interface. The learning of a new interface or instrument has repercussions at various levels as McLuhan describes in his book Understanding Media: The extensions of man [12]. “Any extension, whether of skin, hand, or foot, affects the whole psychic and social complex”. To connect this with ideas of embodiment we again refer to Hayles; “Experiences of embodiment are in continual interaction with constructions of the body”. The learning of new interfaces, especially when applied to full body gesture tracking and sensing, form new constructions of the body which expand the body-image. The experiences of embodiment of new interfaces lead to enhanced modalities of perception and experience for the performer as they must build new cognitive ‘tools (maps?)’ in order to fully explore and master the interface. In short they expand their consciousness.
Hayles highlights how the body and embodiment are separate as the body is “always normative relative to some set of criteria” and embodiment is “contextual, enmeshed within the specifics of place, time, physiology, and culture”. Hayles approach to embodiment comes from the construction of subjectivity and informational processes and not from a literal cyborgian approach. “Embodiment is therefore immaterial, and … is defined by pattern rather than presence” [8]. In Thecla Shiphorsts’ essay Body Matters: The Palpability of Invisible Computing [16] she describes in detail approaching design of wearable computing through Somatics – studies of the soma. This is where the body is perceived from within, through a first person perspective; the lived experience of the moving body. The human soma, which is self-sensing and self-moving, cannot be immaterial as it requires the body in order to complete the task. The embodiment and body are not separate here but are interacting to create a sense of perception, consciousness and awareness. Consciousness is a learned sensory-motor function; awareness is the function of isolating new sensory-motor phenomena and being able to learn it. Through this learning we are expanding our consciousness as was explained previously when dealing with extensions of the body and body-image.
The following section will now look at several artists work in relation to the extension of the body through fabrics, garments and technology.

Embodiment in performance, fashion and design

The idea of reshaping and replicating the bodily form is not new and the use of modern technology to achieve it is not either. In the early 1900’s dancer Loie Fuller [8, 9] was transforming her bodily form into images of “pure energy and light, which conjured metaphors of clouds, butterflies, and dancing fire”. Leaving no natural body on stage she cloaked herself in yards of semitransparent gauze robes which she weaved around her body. With the use of wooden batons of her own design attached to her arms she could transform her form beyond the natural limits of the body [fig2]. For her performances she
[fig2]
used the new technology of the day – electricity. Much of the technological props used in her performances were her own designs of multidirectional, multicoloured lighting boxes, and mirror and glass contraptions used to give illusions of floating in the air and replicating multiple dancers. The sheets of fabric she performed in acted like a screen for the projections which further enhanced the effect her reconfigured bodily form. She was originally considered to old and her body not the correct shape for dance so she concealed her bodily form and “her art lay in the technological reshaping and replication of the body” [9]. We can see from Fullers early performances intertwining technology, costume and dance how she extended her body-image and transformed her into moving shapes and colours. A century later we can still see reconfiguring and reshaping of the body through wearable technologies in projects such as Whisper [17] , FoAM , Kondition Pluriel and Walking City (2007) [fig.3] by Ying Gao .  Gao, based at
[fig.3]
Hexagram in Canada, explores interactive clothing, new materials, and fashion. The clothes react to human presence, either breath or motion, with the pattern and design of the garments folding and unfolding in slow motion. Schiphorst [16] draws attention to the importance of moving in slow motion as it brings about an increase in awareness of the embodied state. This technique is practiced in T’ai Chi and Butoh traditions as well as movement therapies. Moving in slow motion allows the body to enter into an “immersive state in relation to its environment”. The effect of the cloth moving slowly in relation to human presence is calm and mesmerising . Another artist to employ clothes which move is fashion designer Hussein Chalayan . His spring/summer 2007 collection, called One Hundred and Eleven, was made up of, among others, five animatronic dresses [fig.4].
 [fig.4]
The dresses used an array of electronic components in the garments altering their form and shape. The garments retract over and weave around the body leaving the wearer in a different outfit. In this case it is not the human body which is reconfigured but the body of the fabric instead. In Symbolic Exchange and Death [2] Baudrillard explains how fashion is both ‘neo-’ and ‘retro-’ at the same time and Chalayans’ designs reflect this as they morph classical styles in contemporary ways. Chalayans’ designs also reflect what Baudrillard goes on to say: “Fashion is only a simulation of the innocence of becoming, the cycle of appearances is just its recycling”.
The Emergent Objects [4] project looks at ways of designing through performance. Working with a robotics company, Shadow Robots , and dancers they have been exploring embodied performance as a way to improve interaction and robotics design with their SpiderCrab (2007) sub-project [4]. They used the mask technique, “The performer contemplates the mask so deeply that it can ‘posses’ – fully inhabit or inform – their body when worn”, so the dancers could embody the robot and in return the robot embodied the dancers. They claim that “new perspectives on the robot, its design and potential, emerged with a palpably exciting rapidity”.
Emergent Objects approach embodiment from a distance. By this I mean that the dancer always has a physical distance between their bodies and the robotic bodies. Stelarc’s Scanning Robot/Automatic Arm (1995) and Exoskeleton (1998), and Marcel.lí Antúnez Roca’s Afasia (1998)and Epizoo (1994) see the artist wearing the robotic technology. In my view, these significant cyborgian artworks explore mechanical embodiment in a somewhat ‘macho-camp’ way but they are truly inspirational for possible designs for new musical interfaces and full body expression. In the ten or so years since these performances computers, machines and communication devices have become smaller and more discrete. Even Stelarc’s latest piece Extra Ear (2007) is more subtle as the technology is smaller and embedded into his flesh. Celeritas (2007) [18] is a wearable wireless system which enables performers to play a virtual 3-D instrument around their body by attaching a 10mm WIMU (Wireless Inertial Measurement Unit) module to themselves. Once a movement constraint, technology is now small enough to be little hindrance for performers and to allow them to move freely. In this sense technology can be seen as getting closer to embodying us, we are no longer embodying technology.

Situating my work

My work is taking two possible approaches at the time of writing and is seeking to create a balance, or hybrid, of the two. The first is Loie Fuller and Deleuze’s The Fold [7]. The second draws on architect Richard Rogers , and performance artists Stelarc and Marcel.lí Antúnez Roca.
As explained previously in this essay Fuller’s costumes were made from yards of semitransparent gauze robes weaved around her body. The patterns which emerged were made up of many folds in the fabric. The audience is faced with the swirling shape of the fabric with no (human) bodily form but all the movements are originating from her body. This split from what the audience sees to the invisibility of how it is being produced is described by Deleuze when he discusses the pleats of matter and the folds in the soul. He states that there is:
“a correspondence and even communication between the two levels, between the two labyrinths, between the pleats of matter and the folds in the soul … the veins are the pleats of matter that surround living beings held in the mass … the veins are innate ideas of the soul…”
It could be said that the images of “pure energy and light … clouds, butterflies, and dancing fire” conjured by Fuller’s costume are the innate ideas and expressions of her soul. In other words she has fully embodied the costume and it is now an extension of her self.
If we now look at the work of Rogers, Stelarc and Roca, the most important difference is the revealing of the technology and inner workings of the system. The importance here is to reveal what is there and not hide or conceal the machine. Rogers’ Lloyds’ Building [fig.5], situated in Lime Street, London, has been designed with no outer shell.

[fig.5]
Pedestrians walking past can see the ventilation ducts, pipes, tubes, etc, protruding, winding and going back into the building. It leaves the physical workings of the building on display. Similarly in their performances of Scanning Robot/Automatic Arm and Epizoo, Stelarc and Roca display the technologies attached to their almost naked bodies. For me I like to see the technology as it has its own visual aesthetic and I feel it is important for the audience to see this as they can then grasp more quickly that there is a computer or machine presence as well as a human one.
My Design for the costume in Sensory Response Systems will balance somewhere between reshaping and reconfiguring the body with fabrics whilst leaving the technology and the ‘inner’ workings of the system in view. The costume will be more discrete than Fullers’, with electronic components in the fabric which will reconfigure its form in choreography with the performers own physical movements. Sensors attached to the performers’ body will be on view but again will be discrete with the wires and connectors weaving in and out of the fabric and around the performers’ body. Similar to the Celeritas project I aim to create an environment in which the performer can play an instrument but it differs in that I am not aiming to create a 3-D virtual instrument, but a more tangible, body based one.

Conclusion

I approached Sensory Response Systems as exploration into audio-visual performance using sensors responsive to physical movements and fabrics to reshape and reconfigure the body-image and form: to turn the body into a new interface for audio-visual expression. Placing this in the context of embodiment, post-modernism, post-humanism, physical computing, performance, and computer sound, and looking at references in fashion, architecture and interactive clothing it provides a strong and stable structure in which to place the work. It has been demonstrated that through the application of sensors and fabrics on to and around the human body we extend our body-image and conscious processes to explore new ways of learning, moving and performing. We have also demonstrated that physiological-intellectual behaviour, and sensory feedback and interaction are important for learning new instruments and interfaces.
Now that Sensory Response Systems has been established within the contexts detailed in this essay it can progress onto further work and exploration. I plan to work closely through the design process with technicians, dancers, performers and musicians in order to make an efficient and effective body-instrument.

Bibliography

1.   Bartenieff, I., Lewis, D. 1980. Body Movement: Coping with the Environment. New York: Gordon and Breach Science Publishers.

2.   Baudraillard, J. 2006. Symbolic Exchange and Death. London: Sage.

3.   Baudraillard, J.1994. Simulacra and Simulation. Ann Arbor: University of Michigan Press.

4.   Baylis, A., McKinney, J., Popat, S., Wallis, M. 2007. Emergent Objects: Designing through performance. International Journal of Performing Arts and Digital Media (in press).

5.   Birringer, J. 1998. Media and Performance: Along the Border. Baltimore: The John Hopkins University Press.

6.   Boal, A. 1992. Games for Actors and Non-Actors. London: Routledge. Jackson, A. (trans).

7.   Deleuze, G. 1993. The Fold: Leibniz and the Baroque. London: The Athlone Press.

8.   Dixon, S. 2007. Digital Performance: A History of New Media in Theatre, Dance, Performance Art, and Installation. Cambridge: MIT Press.

9.   Garelick, R. K. Electric Salome: Loie Fuller at the Exposition Universelle of 1900. 1995. In: Imperialism and Theatre: Essays on World Theatre, Drama and Performance. Gainore, J. E. (ed), pp. 85-103. London: Routledge.

10.  Hayles, N. K. 1999. How we Became Post-Human. London: University of Chicago Press.

11.  Johnson, D. H. (ed). 1995. Bone, Breath, and Gesture: Practices of Embodiment. Berkeley : North Atlantic Books.
 
12.  McLuhan, M. 1994. Understanding Media: The Extensions of Man. Cambridge: MIT Press.

13.  Merleau-Ponty, M. 2006. Phenomenology of Perception. London: Routledge.

14.  O’Sullivan, D., Igoe, T. 2004. Physical Computing: Sensing and Controlling the Physical World with Computers. Boston: Thompson Course Technology.

15.  Polotti, P., Rocchesso,  D. (ed). 2008. Sound to Sense, Sense to Sound: A State of the Art in Sound and Music Computing. http://smcnetwork.org

16.  Roederer, J. G. 1995. The Physics and Psychophysics of Music. 3rd Ed. New York: Springer-Verlag .

17.  Schiphorst, T. 2008. Body Matters: The Palpability of Invisible Computing. Canada: Simon Fraser University. In press?

18.  Schiphorst, T., Andersen, K. 2006. Between Bodies: using Experience Modeling to Create Gestural Protocols for Physiological Data Transfer. http://whisper.surrey.sfu.ca/

19.  Torre, G., Fernström, M., O’Flynn, B., Angove, P. 2007. Celeritas: Wearable Wireless System. Proceedings from the 2007 Conference on New Interfaces for Musical Expression (NIME07), New York, NY, USA.

20.  Wishart, T. 1985. On Sonic Art. York: Imagineering Press.

21.  Zimmer, R., Jefferies, J., Srinivasan, M. A. 2008. Touch Technologies and Museum Access. In press, Berg Publishing.