This project is an exploration of using facial expression as a performing tool. We usually don't focus on, but a face always producing rhythms, through movement of muscles. Combined with a concept of angle of view, I wanted to make an experimental audiovisual performance only using face tracking technology.
produced by: Philip Liu
Core concept of this work is facial expression controls visual, and the visual controls sound. Connections between the elements of each stage are clear, for example left or right blink draws bright square on the half of screen. Width and height of mouth determines amplitude and amount of deformation (phase of deforming signal) of two primitives, and dots in the primitives determines amplitude of each bins of spectral filter/filterbank. Current position of face determines orbital angle of a virtual camera, which also alters the look of primitives projected on the viewports. The combination of openCV and visual to audio mapping is the important idea of this work.
Each part, face detection, generative visual and sound synthesis had coded separately and merged into one project. The concept of blink was already fixed from the conceptual phase, but the concept of viewpoint had been changed. Initially, I changed shape of waveform itself using the face position data, but changing one dimensional data using 3d transformation was ambiguous and I decided to embed 3d visual shape between the facial expression and sound.
This project uses two program, openFrameworks(oF) and Supercollider(SC). Communication between two program is done via OSC. There was optimization issue when applying x coordinate data of the dots to the filter bank. Actually each primitives has more than 1400 dots, and it is too large number to run filter bank in real time, so I decided to send only 90 samples to SC and use them to determine the filterbank amplitudes. There are ghost faces from the half point of video, and those were made using shift register like method. There is struct and it contains every related data from blink on/off, mouth width/height and shape of jaw. Then I made a vector of struct, and save current information each frame and draw them after specific amount of video frame. As a result it yields delayed blink, so density of blink is increasing from the half of video as the number of ghost faces are increasing.