DataGlobe Update

After the deadline i decided to post the project on some sub-reddits as well as tweeting some interesting people. The response was very positive, and I was very lucky to post to /r/dataisbeautiful on the day it was made a default sub.

I’ve had 5,169 visits, from all corners of the world, from New York City to Beijing. I feel extremely proud to have built a project focusing on global themes and actually captured a global audience.

Another update is that Goldsmiths have used the Dataglobe as the image on the Undergraduate Innovation Show poster. I’m very happy indeed to have been accepted to demo my work at the show let alone be used on the poster.

Where: Amersham Arms, 388 New Cross Road, London SE14 6TY
When: 3pm-11pm Wednesday 4 June 2014
Register for free tickets:

6 – About DataGlobe


DataGlobe is a web based 3D data visualisation. It visualises data indexes and measures from Numbeo, “the world’s largest database of user contributed data about cities and countries worldwide”. All the data visualised is from Q1 of 2014.

The aim of the project was to question how big data can become meaningful information for citizens and how openly available data is for current worldwide issues and trends. I’ve explored this under the bracket of interactive 3D web graphics, and data visualisation. The main Javascript library implemented is three.js, along with JQuery for data manipulation, and Bootstrap for styling. It best viewed in an up-to-date version of Google Chrome.

Visit DataGlobe.

DataGlobe was developed as my Year 2 Creative Project for my BSc in Creative Computing at Goldsmiths, UoL. My full development log posts and reflections can be found below.

5- DataGlobe, Reflections.

As I read back over my initial project proposal, I have accomplished my main goal of creating a 3D data visualisation on the web, which is good. The data I first wanted to visualise was to do with the environment and ecology. I succeeded in this, by visualising pollution and emissions data on a global level. I always went over and above this, by being able to visualise even more meaningful data, such as rent indexes, safety as well as economic indicators. Technically I achieved most of my goals. I got to grips with three.js and the camera libraries it had to offer, rendered the globe, added a nice glow texture as well as successfully implementing all the data handling and steps needed to go from numbers to lines in 3D space. I did not include one technical proposal, which was to have additional heads-up display style graphs to back up the globe. I feel that as I progressed with the project, my focus shifted onto making the globe look as striking as possible, and ensuring user interaction was sleek. More and more throughout the course of development did I think that graphs would actually be quite a boring, dated way to look at data. I loved the idea of a 3D intractable, changing graphic of the world could be used, rather than basic lines. The idea of visualising location based data needs to have a solid context. The user experience and overall knowledge gained by looking at and interacting with the 3D globe is much more advanced and interesting than seeing data on a chart.

I did not manage to have functionality to visualise historic data sets, where the user could see data from a number of years. This is one major failing in the project. I simply ran out of time to do this. I could have achieved this, it would have just taken more time to format all the data correctly. This would allow the user to be able to gain even more insight into the current data set, why is it like this now? what changes have occurred in the past? are questions that come to mind when i use the final version.

I wanted by data visualisation to be able to appeal to people who live in cities. A user in New York City should be able to see data for their city and be able to compare it other cities on a global level. I have achieved this. A young person in Jakarta can use the visualisation to learn more about their city in terms of global measures, as well as get insight into global trends. This is also applicable to an older user in Vancouver. I’m very proud of achieving a user based aim, as this project’s worth hinges on the fluidity of the user experience achieved by a considered, minimalistic design approach.

I have learnt that data sets with brilliant meaning and worth are hard to find and the best way to acquire these is to collect them yourself. Numbeo was a brilliant discovery, and potentially the best data sets I have ever seen. I did feel let down by the government API’s and world bank, as their data sets were very very lacking, choosing to focus on strange economic indexes rather than information that is truly useful to it’s citizens. Of course it’s up for debate as to why a government would do this anyway. I feel that I had underestimated this aspect of the project and it did take me quite some time to find relevant data, and once I found it, the formatting and transforming did take quite some time and work-around to get it fully working.

I feel that my data visualisation is quite fair. I am not massaging any numbers, nor do I have any sort of agenda at play. The only editing I did was to remove one or two data values that massively peaked above the rest. The reason being that this would totally ruin the scaling of all other data spiked due it being, perhaps falsely, very high. I put a focus on this in my original proposal and presentation of ideas. It’s key that the visualisation gives true insight and is not falsified.

I could develop this project much further with more time and resources. Numbeo is all user-supplied data. I feel that it would be incredible to be able to collect data en-masse, and get users to give me data and then be able to visualise an ever growing and changing data set. I also the type of data that could be collected could be very diverse, from traditional economic and geographic data, to zeitgeist data such as the 2013 floods in the UK, and mapping this against short term economic impacts. I could develop the graphics further, to have more sophisticated interaction in terms of the camera control, as well as using more high res images to texture the globe, or perhaps opting for a flatter texture. This could allow for focus on single countries/ continents as well as just the whole world.

When considering my project in the early development stages, I really loved the idea of treating the browser as a simple canvas, which the user can interact with, and as I progressed through the project I came to realise this even further. This whole project has taught me a great deal. Not only about the ins and outs of jquery syntax and data structures, but about the sort of computing I enjoy. Projects that can be easily accessed by anyone on earth, and educate on very issues. To achieve through intuitive, visually striking graphics is extremely satisfying indeed. I was also keen to learn about ‘big data’ and its technical make up, as it a theme that is swiftly entering mainstream digital culture. My ideas of graphics, web applications and big data sets have been totally reimagined. This has been my largest, most intricate project to date and I’m very happy with the end result, and everything that it took to get there.


Additional Information and Resources used.

three.js library
Orbit Controls Camera Library, slightly amended camera max/min zoom
Planet Pixel Emporium, Earth texture used.
Threex.planets, learnt how to create a globe mesh
WebGL Globe, understood how to convert lat/lng to 3d vector.

4 – Development Progress, NUMBEO, Design Ideas and data formatting.

The data that the guardian had provided wasn’t very reliable. It had measures such as ‘coffee price’. Surely in any city you could spend anything on coffee, from super cheap to very expensive. After some searching, I found Numbeo. A data resource site that has clients such as the BBC and Wired, and operates under a “attribution-ShareAlike 3.0 Unported” Creative Commons license, which allows for remixing, transforming and building the material provided. Their data is very expansive, from health indexes to rent values, they have produced amalgamated indexes that draw from various measures and sources. The data provided is on a city level. I knew that I would have to convert these strings into longitude and latitude values in order to visualise them on the globe through the google geo api. I also had the challenge of going from data tables on a website to JSON.

I copied the whole data table and pasted it into a google spreadsheet. I could then convert this into a JSON file via an online CSV to JSON tool. I then had json like this . I hacked together a little javascript function that took the JSON data feed, and logged it to the console. I carefully formatted the logs to have the correct CSV syntax in place. I logged the result of the longitude and latitude API calls to the log, so I effectively had a big CSV file of the city name, longitude and latitude and all the data. I then used the CSV to JSON tool again to reformat it. I did this for all 4 data sets, and then had 4 expansive JSON files, complete with the correct longitude and latitude co-ordinates for each city.

I could now spend time thinking about the overall design of the globe and data spikes. At first I got all the values onto the globe, drawn as white lines. I did this by first drawing as many lines as needed for that data set at a uniform size, and then pushing all the data into a city values array, which was then scaled, and used as a scalar value for the lines. Some data sets had more entries than others, so I needed to clear these arrays every time I selected a value to ensure they were actually refreshed. A function would then re-size the lines. The processing power is actually quite small as the lines are added to the scene and sized only once. Next, I added some colour to the lines, based on the data value. I carefully went through the datasets and noted which data values were good when high, and bad when low, and vice versa. For example a high crime rate is bad, but a high safety index is good. I could then determine how to colour the lines. For high-values meaning good things, I coloured the highest lines green, and lowest red. I then added a little appendix to each JSON file at the top to hold a description one each data value, outlining what exactly it meant.

Final touches were to polish up the website. I used Bootstrap for this. It allowed me to style the drop down menu, as well as have a generally nice feel to the overall design of the site. This paired with some google fonts allows for a smooth, intuitive user experience, with attention focused on the globe and data rather than the overall website layout. Done. I had developed the underlying functions and major computations earlier. I spent a long time in this stage of development, searching for data and then handling it, to ensure the visualisation was meaningful. The way i scaled the data ensured a uniform comparison for all numbers, whether the range was between 0-10 or 10,000 to 1,000,000. I’m very happy with the end result as it allows me to be able to simply feed the functions with data and they do all the legwork in terms of scaling and displaying, saving me the laborious task of mass-data-editing.

3 – Development Progress, Basic 3D Visualisation

Geo-location data is based around latitude and longitude. I had never appreciated the power of this measure. It has the potential to be incredibly accurate. Having picked through the code of the Web GL globe and read the blog on the arms globe dev blog, I understood I need to write a function that took a latitude and longitude co-ordinates to a scaled vector in 3D space, that would run through the city’s location on my globe. This was daunting.

First I tested what mesh to render and place in three JS. First I tried using the cube geometry by stretching them out to become long thin lines. However, the cube geometry had the very odd functionality of drawing from the middle outwards. I could not declare the ‘start’ at the center of my world and another final point to draw a vector between, without some very laborious transformations. I didn’t want to hack together my data ‘spikes’. They needed to be simple. The Line geometry answered my prayers in that I could declare two points, one at 0,0,0 and one in some given point in 3D space and it would draw a line between them. I then focused on the mission to change lat-lng co-ordinates into a 3D vector again. I researched how the web gl globe did it. Some complex trigonometric functions allowed this transformation assuming the globe was totally spherical. I had experimented by dubbing in London’s longitude and latitude co-ordinates into the function with a simple scalar value. I knew that a vector was easy to scale, a simple multiplication on x y and z would scale the whole thing. Having tinkered with the function I managed to get it working, and retested for other co-ordinates across the world. I hit problems when it came to drawing lines for the lower half of the globe, but negating my phi/ theta values overcame this. I now had a 3D globe, and a way to draw lines through longitude and latitude points correctly. Next I needed some meaningful data.

The globe baseline is done, and now I need to look for data. Due to the amount of data visualisation I had seen, I slightly naively assumed that there was an expanse of usable, open data just waiting to be plugged into a project. The data sources I thought would be brilliant, and the World Bank were strange. They provided a lot of countrywide data sets for obscure measures of things such as the “Annual Percentage added to GPD from agriculture, 2000-2010″. but not too many world-wide data sets centered around cities. Thanks, World Bank.

Data sets from these sources were also provided in heavy formatted PDFs or websites. I needed to learn what data format I wanted the data to be in. I had of course used CSV files, but I was not sure how I could visualize these. After some basic javascript tinkering, I could pick out values from a CSV file through a simple for loop easily. However, the web GL globe had thousands of data points, as well as location data. A CSV file of all that data would be hellish. I discovered JSON, a human readable, JavaScript style data format. After some guidance from a tutor, I got to grips with how to read a JSON data file into javascript, and use the values as data in my visualisation. Through JQuery and AJAX.

When searching for data, I found the Guardian data blog had an impressive array of data visualizations, a lot of which provided the data on a google drive spreadsheet. I finally found some city data, about costs to live in a city . I knew the data set was not going to be my prime final visualisation, but it was good enough to run some tests.

I exported the google spreadsheet as a CSV file, and then used the online CSV to JSON converter to make it a JSON file. There were 20 or so cities on there. I had most of the longitude and latitudes of these cities in an array from my earlier tests, so I reordered the array, and added the cities I was missing. I then ran a for loop to call my lat-lng to 3d vector function, to map these cities coordinates to the ‘price of a coffee’ data set. It worked. I had 20 or so lines rendering in 3D space over the globe, calling at least some values from a JSON file, not just a huge javascript array, and I was delighted. I now needed to explore how to automatically get longitude and latitude from city names. The google Geo Coding API did this. I could make an API request with just a string of a city name, and then get a longitude and latitude value back.


2- Three.js is amazing.

I started getting to grips with Three JS by using tools and online editors such as Three JS Playground and Live Code Lab. They gave me great insight into the fundamentals of three.js, and the creative power of web graphics.

Through the Chrome Experiments web site, I found the Web GL Globe, made by the Google data arts team. Another data visualisation based around a 3D globe. It’s a brilliant project. The user can see general, continental trends can be very swiftly. Despite the fact data can be seen for the whole world, through interaction, it’s possible to extrapolate very specific differences. The data on major cities in the UK can be analysed and compared, despite the fact the UK is a small island nation with a handful of major cities. The data visualised was quite obvious. The population density spiked in huge cities in the far east, and the Google language search was guessable. I learned how my visualisation needed to ensure that the data can provide interesting insight for major global issues on a continent-wide scale as well as quite locally. I know felt that a 3D globe based visualisation of location based data is what I was going to aim for. It’s a powerful concept if designed well and ultimately I felt it was achievable. It did take me longer to decide on what avenue to go down for this project than I had anticipated, but I felt sure this was the way to go.

I started research on making globes in threejs. The threex.planets project was very useful as an introduction to three.js texturing and rendering as a whole. It taught me about the different types of lighting, how to actually texture a 3D shape, as well as the different levels that make up a texture. This project used images from Planet Pixel Emporium, a site that sells high res planet textures, as well as bump map, specular and cloud textures which can be intricately layered to create an extremely life like planet. A lot of examples I researched used the Orbit Controls camera library. This allows user interaction with the camera through clicking and dragging, to dolly in and out form the scene as well as scanning around in 3d space. Through these project and libraries, I made a 3D globe, with user interaction, as well as a bump map with shadows, a specular texture that allows lights to reflect off the sea and a cloud map, that renders quickly on the internet.

three.js globe from Robin Hunter on Vimeo.

This was incredible. I had always conceived high res, user intractable graphics to be reserved for dedicated, stand alone software and apps. The idea of treating the web browser as a canvas that anyone across the world with a good browser client can interact with has changed the way I consider graphics in terms of their use in web projects. The three js library is expansive and lightweight, with massive potential for games, visualisations as well as other advanced projects.

1 – Project ideas

This is post number 1 in a series of reflections about my second year Creative Computing project that i have developed in late 2013 through to mid 2014.

At first, the prospect of being given total freedom to think of then implement a project was exciting but very daunting. I had some ideas that were similar to my first year major group work, an audio visualiser called Super Rotator made with Processing and MaxMSP.

Super Rotator – Sound Reactive Graphics from Super Rotator on Vimeo.

I loved working on that project, as it allowed me to explore how music and graphics can overlap in a live context. Since completing that work, i continued finding interesting audio-visual projects that combined audio visualisation as well as actual music production. An ios app called Thicket is great example.

THICKET, iPad/iphone App. from Didier LAHELY on Vimeo.

I also started tinkering with openFrameworks over the summer time, implementing basic graphics and basic audio analysis. Early year 2 CC lectures covered OF in more detail, and i explored other ideas such as computer vision and projection. Night Lights is one project that really stuck out to me. Interaction takes place through the users shadow casting onto a receptor, which then controls projections.

I started to think more objectively about what i wanted to pursue in this project. As much as i loved the creativity and the innovation in a lot of the fully fledged audio visual installation projects, i felt i would struggle to be able to define an idea and then fully realise it. The temptation and possibility of straying away and being distracted from an initial idea due to the sheer wealth of potential outcomes was what put me off going down this route. I knew that i wanted to explore interactive graphics in some way, maybe moving away from the audio side of things. My major aim in first year was to learn about audio visualisation, which i achieved through the super rotator. More and more i started to become engaged with data visualisation. Finding and researching David McCandless’ Information Is Beautiful and three.js was a turning point for me.

Three JS is a lightweight 3D web graphics library, with some very impressive implementations.
Fully polished, 3D games can run in-browser with no worries. The Information is Beautiful book helped me learn that big data visualisation is quite a simple concept, but can produce massively insightful results when done well. This great web library combined with the concept of data visualisation helped me think more clearly about what i wanted to do.

ArmsGlobe visualises arms and ammunition trade data between 1992 and 2010, and is based on interaction with a 3D globe. This population globe from Google visualises the world population, again on a 3D globe. I was drawn to these projects as they were very striking visually, and the fact data on the whole world can be seen in one place, on the internet, was incredible. It was these projects that made me decide that i wanted to work on a 3D, web-based data visualisation.