Holographic Audio Visualizations
Holographic Audio Visualizations explores the world of audio visualizations. It strives to rethink the various ways in which audio can be visualized while also taking into consideration the human side of how we interact with sound. Instead of translating audio inputs into binary data, Holographic Audio Visualizations uses the human body as a means for sensor data. Holographic Audio Visualizations was conceived through observing how we, as human beings, move and interact with sound. People tend to move their arms and hands to the rhythm of the music. Depending on the severity of the beat, some may thrust their arms in the air or lightly tap their hand to the music. Through a combination of code and physical computing, Holographic Audio Visualizations uses holograms to visually represent sound in real time and sensor data from the bracelet worn by users to record this human interaction. It is a fun and engaging way for participants to see the sounds they are hearing.
Produced By: Julianne Rahimi
Holographic Audio Visualizations was born out of a curiosity for the world of audio visualizations. I was fascinated by how artists could visually show something that has no visual properties; sound is heard and not seen. This curiosity evolved into a personal exploration through the means of art. I wanted to understand how humans interact with audio and the ways in which they display it; how do we become the visualizations? Exploring this curiosity beyond the medium of dance is where Holographic Audio Visualizations was born.
I started my exploration by making audio visual synthesizers. As I programed these, I realized I did not understand what went into making a sound, and through my research, I discovered that the elements that went into creating this medium were things I did not work well with. Instead of abandoning the topic, I thought about the different ways we see and interact with audio and how to compute it without any audio inputs. As a former dancer, I knew we could see it through the body; through the movements we make when we hear and feel each beat. Combining this with my past in photography, in particular, cinematic photography, I thought of ways in which I could tell the story of sound. For me, that story was the beat; how hard the beat was, the rhythm of the beat, and the way in which each beat made people move.
Physical Computing and Wearable Technology
In order to combine interactions, sounds, and visuals, I needed to connect the human body and the computer; I needed to think about physical computing, a course I took from Professor Phoenix Perry. My first step was to find the right sensor. I researched the different sensors used to track movement and found two that could potentially provide me with the data I needed: accelerometers, which tracked the speed of movement, and gyroscopes, which tracked how much a location changed. Since I wanted to tell the story of the beat, I would need to gather how fast and hard someone was moving; I would need to use an accelerometer.
My next step was to think about how to connect an accelerometer to the human body. I thought about my former professor at the University of California, Santa Barbara, Professor Xárene Eskandar. In her class she introduced the genre of wearable technology. Wearables would become my answer for connecting the human body with an accelerometer; connecting it with the computer.
As I brainstormed about wearables and music, I thought about music festivals and how attendees are usually given bracelets as their tickets. This led me to think, if I were to be making a piece for a festival, how could I get each attendee to wear an accelerometer? The answer: the bracelets. An accelerometer bracelet would provide the data about how people moved to a beat, since people have a tendency to move their hands when they dance or hear music.
Both Adafruit and LilyPad are used for wearable technology and have boards and sensors that can be sewn into fabric using conductive thread, which builds connections the same way wires do. For my bracelets I used Adafruit’s Flora V2, LilyPad’s Accelerometer ADXL335, and three ply conductive thread. The X, Y, Z, power (+), and ground (-) pins on the accelerometer connect to the D12, D6, D9, 3.3V, and Ground pins, respectively, on the Flora. Since the Flora is not a board made by Arduino, I had to download the Adafruit AVR Boards package. Once the right board was selected, Arduino could successfully upload code to the Flora. I uploaded Arduino’s Standard Firmata code, which can be found in the examples that come with the downloadable Firmata Library. This allows Arduino to send serial data and communicate with Processing, which I planned to use to create my visualizations.
The Visualizations
In order to understand how the accelerometer data would work with Processing, I created a series of simple sketches that included pulsating ellipses, rotating grids, and various other moving shapes. To get these working, I needed to download and import the Arduino Library for Processing, call the serial port, set the Arduino sensor data inputs, and translate the input data from digital to analog. I could then use the accelerometer’s data to control my sketches.
As my understanding grew, so did the complexity and creativity of my sketches. I remembered seeing videos of iPhone holograms and became inspired. Essentially, these holograms are made by cutting the clear cover of CD cases into an upside-down pyramid that has been cut flat just above the apex and placed on top of an iPhone screen in the center of a video of four mirrored moving objects. Instead of playing a video, I would run a Processing sketch that used the sensor data to manipulate the hologram’s movements. Essentially, this would allow users to interact with and see audio in a physical manifestation.
My inspiration for designing the first hologram came from Generative Design and its chapter about Randomness and Noise. In it, it shows how to make a noisy landscape. This reminded me of a vintage radio and how they used to display sound levels with a line that would spike up and down. Instead of a line, the noise corresponds to the sensor data, which determines how high or low each noise pattern is.
After creating my first hologram in Processing, I wanted to find a way to display it on an iPad for the exhibition. I also wanted to find a way to enable Bluetooth so that the user did not need to be tethered to send data. However, this is where problems arose.
iOS and Bluetooth
First I tried using p5.js and translated the code from Processing’s Java to p5.js’ JavaScript, a language that is compatible with iOS. I found and tried a few labs from a Physical Computing course at New York University’s Interactive Telecommunication Program that used Arduino with p5.js, but I was unsuccessful because of the type of program I was trying to create and because I was not using an Arduino brand board.
I then found someone in Japan who appeared to have successfully connected Arduino to an iPad and ran a Processing sketch using HTML, CSS, and processing.js. I tried following his technique and ordered an iPad Camera Connector but still could not run my sketch because of the three-dimensional images and the type of board I was using.
My next attempt would be to use one of the two Processing apps I found on the App Store, Procoding and Processing iCompiler. Processing iCompiler seemed to work fine with the three-dimensional images, but I could not call the serial port that connected the iPad with the Flora. I tried using the Flora Bluefruit LE to send serial data to my program through Bluetooth, however, the Flora Bluefruit LE is mainly made to control a component remotely as opposed to sending sensor data. I was able to get the Bluetooth on and see the connection through the apps Bluefruit, nRF Toolbox, nRF UART, and A Manager, but could not receive any data. I then tried using Adafruit’s Bluefruit LE Friend to bridge the connection through my computer, but still could not receive any data. If I were to continue this project in the future, the ability to remotely send data to a program would be something I would like to achieve.
However, for the exhibition I decided to return back to tethering the wearable. I tried connecting the Flora to the iPad with a Micro-B USB cable and the Camera Connector. When I opened my code in Processing iCompiler, I found that the Flora V2 used too much power. I connected a 105mAh and a 2000mAh LiPo Battery Pack and received an error message saying the Flora was not compatible with the iPad. I tried troubleshooting using the app Get Console, but I was only able to find the serial port and not read the data.
After I watched Gary Bennett’s YouTube video explaining how he connected Arduino to his iPad, I realized why the Camera Connector was not working and that I would not be able to use iPads for this project. Apparently, Apple does not freely allow external electronics to work with iOS. They have an elite program, called the MFi Program, that allows people to experiment with this technology. If one were to be accepted, they would receive a cable that has a chip inside which allows one to connect external electronics to iOS devices. As an alternative to getting the cable through the MFi Program, there is the Redpark TTL Serial Cable (C2-TTL), which has the chip that is needed to make the connection. On one end, there is a 30-pin connector and on the other end the TTL headers, which allows an Arduino to connect to the power, ground, TX, and RX pins, just like the Adafruit Bluefruit LE. According to Bennett, this chip is vital in order for the iPad to receive serial data, which explains why I was unable to do so with the Camera Connector. However, this cable becomes incompatible when used with apps from the App Store.
Acceptance
Since iPads were no longer a viable option, I returned to using monitors and began programming my second hologram. For this one, I got my inspiration from Professor Theodoros Papatheodorou. He had mentioned using a particle system, which inspired me to create particles that were in constant motion but changed their movement along the X and Y-Axis because of the sensor data from the accelerometer. Since I would be placing “pyramids” on top of the screen to hold the holograms inside, I thought about having these particles come out of a pyramid so it would appear to be a pyramid in a “pyramid”. Since object-oriented programming was taught in both Professor Tim Blackwell and Professor Theodoros Papatheodorou classes, the first program employs the use of functions while this one uses both classes and functions.
Although Holographic Audio Visualizations has been developed for Metasis, if I were to continue it, I would refine the bracelets to be cordless, as mentioned. I would also make bracelets that were more fashionable, in addition to the ones that I have for the exhibition, which are designed to be conveniently put on in a gallery. This way users could have options. Lastly, I would want to develop even more holograms and see the different ways in which I could potentially visualize audio in the real world; in a way that appears tangible to the user.
Source code:
HolographicAudioVisualizations_Pyramid
HolographicAudioVisualizations_NoisyLandscape
References:
- http://www.generative-gestaltung.de
- https://github.com/generative-design/Code-Package-Processing-3.x/releases/tag/latest
- https://www.arduino.cc
- https://processing.org
- https://github.com/firmata/processing
- http://natureofcode.com
- https://processing.org/tutorials/p3d/
- http://playground.arduino.cc/Interfacing/Processing
- https://processing.org/reference/curveVertex_.html
- https://processing.org/reference/curvePoint_.html
- https://processing.org/examples/simpleparticlesystem.html
- http://www.telegraph.co.uk/technology/mobile-phones/11780393/How-to-turn-your-phone-into-a-3D-hologram-projector.html
- https://learn.adafruit.com/getting-started-with-flora?view=all
- http://lilypadarduino.org/?p=384
- https://learn.sparkfun.com/tutorials/sewing-with-conductive-thread
- https://www.adafruit.com/product/641
Research:
Physical Computing and Wearable Technology
- https://www.arduino.cc/en/Reference/Serial
- https://processing.org/tutorials/electronics/
- https://learn.sparkfun.com/tutorials/e-textile-basics
Wearables
- http://archive.fabacademy.org/archives/2016/fablabbcn2016/students/139/htm/week_11.htm
- http://www.instructables.com/id/Wearable-Toy-Piano/?ALLSTEPS
- http://www.instructables.com/id/Color-Mixing-LED-Bracelet/?ALLSTEPS
- http://www.instructables.com/id/Fabric-amp-Bead-Tilt-Sensing-Bracelet/?ALLSTEPS
- http://duino4projects.com/tilt-sensing-bracelet-using-arduino/
- https://blog.arduino.cc/category/wearable-computing/
Conductive Thread
- https://learn.sparkfun.com/tutorials/sewing-with-conductive-thread
- https://learn.adafruit.com/conductive-thread/connecting-components
- https://cdn-learn.adafruit.com/downloads/pdf/conductive-thread.pdf
- https://www.adafruit.com/product/641
Conductive Fabric
- http://www.instructables.com/id/Conductive-Fabric/
- http://materia.nl/material/electrolycra/
- http://forum.arduino.cc/index.php?topic=12593.0
- https://books.google.co.uk/books?id=_wLWDAAAQBAJ&pg=PA76&lpg=PA76&dq=how+to+use+electrolycra&source=bl&ots=P8fCSWTh2b&sig=-OM6g-ea8GTX3CqJxYc3DfsFj7o&hl=en&sa=X&ved=0ahUKEwjBhuetndrOAhVMCsAKHeayDPwQ6AEIVzAJ#v=onepage&q=how%20to%20use%20electrolycra&f=false
- http://www.lessemf.com/321.pdf
- http://www.kobakant.at/DIY/?p=2789
Components and Sensors
- https://www.adafruit.com/product/2487
- https://cdn-shop.adafruit.com/product-files/2267/MDBT40-P256R.pdf
- https://www.sparkfun.com/products/13633
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/introduction
- https://learn.adafruit.com/adafruit-flora-bluefruit-le/overview
- https://learn.adafruit.com/getting-started-with-flora/overview
- https://cdn-learn.adafruit.com/downloads/pdf/getting-started-with-flora.pdf
- https://learn.adafruit.com/getting-started-with-flora/flora-pinout-diagram
- https://cdn-learn.adafruit.com/downloads/pdf/adafruit-flora-bluefruit-le.pdf
- https://learn.adafruit.com/adafruit-flora-bluefruit-le/example-code
- https://www.adafruit.com/product/2267
- https://www.adafruit.com/product/284
- https://learn.adafruit.com/adafruit-flora-bluefruit-le/data-mode
- https://www.adafruit.com/products/2479
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/introduction
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/configuration
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/uart-service
- https://www.sparkfun.com/datasheets/DevTools/LilyPad/ADXL335.pdf
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/software-resources
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/faq
- https://learn.adafruit.com/introducing-the-adafruit-bluefruit-le-uart-friend/general-purpose
- https://learn.adafruit.com/getting-started-with-the-nrf8001-bluefruit-le-breakout/software-uart-service
- http://store-usa.arduino.cc/products/abx00001
Programming and Connections: iOS, Bluetooth, Arduino, Processing, Other Softwares, Apps, Etc.
- http://processingjs.org/download/
- https://p5js.org
- https://github.com/processing/p5.js/wiki/Processing-transition
- https://github.com/processing/p5.js/wiki/Getting-started-with-WebGL-in-p5
- https://github.com/processing-js/processing-js
- https://nodejs.org/docs/v0.12.7/api/
- https://github.com/processing/p5.js/wiki/p5.js,-node.js,-socket.io
- https://learn.adafruit.com/bluefruit-le-connect-for-ios
- https://forum.processing.org/one/topic/processing-and-ipad-beginner-s-guide.html
- http://www.instructables.com/id/Cheap-2-Way-Bluetooth-Connection-Between-Arduino-a/?ALLSTEPS
- http://blog.roman-mueller.ch/index.php/2013/04/24/connecting-to-arduino-using-processing-and-bluetooth-hc-05-hc06/
- http://playground.arduino.cc/Learning/Tutorial01
- https://www.element14.com/community/groups/pumping-station-one/blog/2011/04/05/arduino-bluetooth-interfacing
- https://forum.processing.org/two/discussion/10482/arduino-processing-and-bluetooth
- https://github.com/NordicSemiconductor
- https://github.com/NordicSemiconductor/nrf51-UART-examples/blob/master/app_uart_library_example_with_ble/main.c
- https://devzone.nordicsemi.com/documentation/nrf51/6.0.0/s110/html/a00066.html
- https://learn.sparkfun.com/tutorials/serial-communication/uarts
- https://www.arduino.cc/en/Reference/SoftwareSerial
- https://www.freebsd.org/doc/en/articles/serial-uart/
- https://github.com/vanevery/p5.serialport
- https://forum.processing.org/two/discussion/17724/how-to-receive-data-from-android-in-arduino-over-bluetooth
- http://d.hatena.ne.jp/kougaku-navi/20110423/1303589571
- http://d.hatena.ne.jp/kougaku-navi/20110410/1302392313
- http://procoding.audiocommander.de/
- http://thenextweb.com/dd/2012/01/12/creative-coding-is-coming-to-an-ios-device-near-you-but-can-you-wait/#gref
- https://www.youtube.com/watch?v=WK3zc8XDxuI
- https://github.com/NordicSemiconductor/Android-nRF-Toolbox
- https://developer.nordicsemi.com/nRF5_SDK/nRF51_SDK_v8.x.x/doc/8.0.0/s110/html/a00072.html
- https://github.com/processing/p5.js-editor
- https://github.com/adafruit/Adafruit_BluefruitLE_OSX/blob/master/README.md
- https://github.com/adafruit/Bluefruit_LE_Connect
- https://learn.sparkfun.com/tutorials/terminal-basics/tips-and-tricks
- http://stackoverflow.com/questions/8632586/macos-whats-the-difference-between-dev-tty-and-dev-cu
- https://github.com/firmata/processing
- https://learn.sparkfun.com/tutorials/connecting-arduino-to-processing
- https://github.com/generative-design/Code-Package-Processing-3.x/releases/tag/latest
- https://www.mathsisfun.com/algebra/trig-finding-angle-right-triangle.html
- https://processing.org/tutorials/trig/
- https://processing.org/tutorials/p3d/
- https://github.com/processing/processing/wiki/OpenGL-Issues
- https://processing.org/discourse/beta/num_1259409632.html
- http://playground.arduino.cc/Interfacing/Processing
- https://processing.org/tutorials/objects/
- https://forum.processing.org/one/topic/how-to-mirror-the-left-half-of-the-canvas.html
- https://processing.org/examples/bounce.html
- https://processing.org/examples/movingoncurves.html
- https://forum.processing.org/one/topic/moving-circle-along-a-path.html
- https://processing.org/examples/arrayobjects.html
- https://processing.org/tutorials/pshape/
- https://processing.org/reference/PI.html
- https://processing.org/examples/flocking.html
- https://processing.org/examples/arraylistclass.html
- http://cs.smith.edu/dftwiki/index.php/Creating_a_trail_of_moving_object_in_Processing
- https://www.youtube.com/watch?v=NXcppUf7vl8
- http://makezine.com/projects/connect-an-iphone-ipad-or-ipod-touch-to-arduino-with-the-redpark-serial-cable/
- http://stackoverflow.com/questions/7770973/connect-an-iphone-to-arduino-over-bluetooth
- https://processing.org/examples/simpleparticlesystem.html
- https://processing.org/tutorials/drawing/
- http://teachmetomake.com/wordpress/adafruit-bluefruit-arduino
- http://redpark.com/products/developers/serial-cable-sdk-faq-download/
- http://redpark.com/products/developers/#sdk
- http://redpark.com/ttl-serial-cable-c2-ttl/
- http://redpark.com/redpark-ttl-serial-cable-simplifies-connections-to-arduino/
- http://stackoverflow.com/questions/5930645/how-to-transfer-data-over-usb-with-an-ipad
- http://forum.arduino.cc/index.php?topic=144560.0
- http://www.oreilly.com/pub/e/2731
- https://processing.org/discourse/beta/num_1235616643.html
- https://processing.org/tutorials/pshape/
- https://processing.org/reference/PShape.html
- file:///Applications/Developer%20Tools/Processing.app/Contents/Java/modes/java/reference/PShape_getVertexCount_.html
- https://processing.org/examples/functions.html
- https://processing.org/reference/Array.html
- https://processing.org/reference/PShape_getVertex_.html
- https://processing.org/tutorials/pvector/
- http://redpark.com/serial-cable-115-2-kbps-c2-db9v/
- https://www.nordicsemi.com/eng/Products/Nordic-mobile-Apps/nRF-UART-App2
- https://www.nordicsemi.com/eng/Products/Nordic-mobile-Apps/nRF-Toolbox-App
NYU's ITP Labs
- https://itp.nyu.edu/physcomp/labs/labs-serial-communication/lab-serial-input-to-the-p5-js-ide/
- https://itp.nyu.edu/physcomp/lessons/serial-communication/serial-communication-the-basics/
- https://itp.nyu.edu/physcomp/lessons/microcontrollers/microcontrollers-the-basics/
- https://github.com/ITPNYU/physcomp
- https://itp.nyu.edu/physcomp/videos/videos-serial-communication/
- https://itp.nyu.edu/physcomp/labs/labs-serial-communication/lab-serial-output-from-p5-js/
- https://itp.nyu.edu/physcomp/labs/labs-serial-communication/serial-output-from-an-arduino/
- https://itp.nyu.edu/physcomp/labs/labs-serial-communication/lab-serial-communication-with-node-js/
- https://itp.nyu.edu/physcomp/lessons/serial-communication/interpreting-serial-data/
- https://itp.nyu.edu/physcomp/lessons/serial-communication/synchronous-serial-communication-the-basics/
- https://itp.nyu.edu/physcomp/
- http://www.gal-nissim.com/2015/10/13/first-contact-the-arduino-p5-js-summit/
- http://www.techaboom.com/combining-p5-and-arduino/
- http://www.ella-dagan.com/labs-serial-input-output-arduino-p5-js/
Display
- http://www.apple.com/imac/specs/
- https://www.ifixit.com/Guide/Repair/Installing-iMac-Intel-21-5-Inch-Hard-Drive-Fan/1853/1
- https://discussions.apple.com/thread/3681707?tstart=0
- https://www.youtube.com/watch?v=uQurKC7z4II
- http://davidchumac.blogspot.co.uk/2009/11/how-to-remove-imac-stand.html
- https://www.quora.com/Can-you-remove-the-stand-from-the-21-5-inch-iMac-by-removing-and-reinserting-parts-of-the-computer
- http://imacmount.com
- https://www.youtube.com/watch?v=Cmm7jWU-g2s&feature=player_embedded
- https://www.ifixit.com/Answers/View/51635/Any+easier+to+remove+stand
- http://forums.macrumors.com/threads/remove-stand-21-5.1084575/