The original concept for the project was to create a program which could act as an audio analysis tool and as an art piece for VJing. The program would be a landscape created purely from FFT values of the music track, creating a mountain effect. It would use granular synthesis would be used to enhance both the audio analysis and the art aspects of the program. It would allow the user to be more precise with how fast they want to move through the track.
What we ended up creating was a program in which the user can explore the mountain-scape of a particular music track. This landscape has been created using FFT values of that track. The FFT values are stored in the program so the user is able to move to any point of the track and explore those particular parts of the landscape. The user also has the ability to change the rotation of their view so they can see the landscape at different angles and the ability to change the colour of the landscape by pressing a key. When analysing the audio, the program comes with overlays which the user can use to see where they are in the track.
Considering that there have been a few, but coherent, adjustments to our initial idea, and therefore we’ve not spent all the time working on the final one, after all we believe that the result is enjoyable. Visual is surprisingly immersive, in fact especially if the camera is animated sometimes you spot your head tilted on some unusual angle. Handling animations is pretty easy, though not exactly user-friendly since by now it is in part set with keys instead of being entirely set with trackpad drag and drop or other gestures.
However, there were some concepts in the original idea which did not make it through. An example is the granular synthesis. Although we had research for it and started to prototype it in order to understand how it worked, we struggled on getting the basis of the project into the live code. This meant that we were unable to develop granular synthesis into the program. Another aspect we had to find a workaround for was collecting the FFT data for the music track in order for the user to explore the landscape. With the current issue of the program, the user has to wait until track has finished playing for the landscape to be stored in memory. Ideally we wanted the user to explore the landscape as soon as they select a track, but due to limitations of minim we had to use the workaround. An unavoidable issue which we encountered was the framerate of the program would reduce to around 15 frames per second. This was when we used the perspective function, which we needed to use so that users can see more of the mountain-scape and feel as if the mountains are part of a never ending landscape.
If we did the project again, we would take what we had learned from here and improve the program further. To start, we would already have a clear idea on what we would want to build and not change our ideas constantly, which is what ended up happening. Since we would have a clear idea on what the project was about, we would focus on creating the landscape from FFT values and quickly move onto implementing granular synthesis. Our focus would then shift to user customisation and extracting more data which would be analysed and could be represented in a landscape. We would also look at other programming languages, like open frameworks, because we understand now that java is very resource intensive and doesn't manage memory well. This would reduce the latency and increase the framerate of the program. Another area we would focus on would be allowing the user to playback the music as they were analysing the landscape.