Resources for Data Visualisers

I want to visualise

I want data

I want to see more

I want to read more

Image by http://www.flickr.com/photos/41894148532@N01/2517652/

architecture

**Warning heavy tech content**

With the launch of the D-viz toolkit, its come time to describe a little of what happening under the hood (and to an extent why).

Flowchart of the visualisation process in look.iriss.org.uk

Flowchart of the visualisation process in look.iriss.org.uk

First, a note (or two) about our intended audience.
While the d-vis toolkit is freely available to all, our goal was to open up the ability to make visualisations to those who work in the social services, in the public, private and voluntary sectors. in many cases the end users are often not tech-savy and in addition are forced to use ‘legacy’ systems ( for instance internet explorer 6 is still the most prevalent ), and often use of java ( and in some cases even javascript ) is not allowed. This was an important consideration in building the toolkit. We needed to build a system that was easy to use and used technology that would work across the board.
One early decision was to use SVG as our chosen base format, which has some advantages as a resolution independent format (it also allows use the ability to have interactive visualisations in the future), though it is not supported in IE until the latest version (and we’ve received some indications that its support is not complete )

Data input.
Data has been split into 2 types words and numbers.
Words are simply copied and pasted into a text box, he hope to allow the upload of MS word documents at some point in the future.
‘Numbers’ can be uploaded as excel spreadsheets. This is done using the excellent drupal sheetnode module, which handles both the upload of several formats of spreadsheet, but also displays them in a familiar interface for users.

Dashboard
The dashboard is our term for the form which allows you to choose your options. visualisation options are presented depending on the type of data you uploaded, once chosen you are given supplementary options depending on your chosen visualisation type.

Visualisation (aka the fun part).
To build the visualisations we built 3 different (for lack of a better word) ‘engines’. I’ll describe the protovis engine in depth later, but here are the other two described in brief.
wordclouds, The data is parsed and sent off to a remote server, which generates the svg and returns it.
ISOTYPES, this module generates an svg based on preloaded svg paths for icons, changing size, colour and the number of items based on the data provided.

Protovis.

flowchart for the protovis engine

flowchart for the protovis engine

There are several javascript tools in existence that make fantastic visualisations that are rendered in SVG, One of our favourites was protovis ( http://vis.stanford.edu/protovis/ ) which was created by Mike Bostock at standford university. but we had a problem, protovis, being javascript ran client-side, rendered in SVG and there were issues saving the final image as a file. Enter rhino / envjs which allowed us to run javascript server-side. Since this was technology we hadn’t used before ( nor did we have the time or resources to test ) we decided to run this on a separate server.
There were several limitations to this. Firstly all the instructions had to be hard coded into the javascript file, we also had to create a batch file which instructed envjs. Lastly due to the limited DOM, working with test options caused issues. Text issues were solved by appending the title to the end of the SVG document, and font/size options by rewriting the SVG tag at the beginning of the document. the SVG file was then read and sent back to the drupal server as a string, where is is saved as a file once again.

Once drupal has an SVG file through this process it creates a visualisation content type, and saves the SVG to it. the SVG is also rasterised and a standard image (firstly so it is in a familiar format, secondly to ensure all users can view it). and it is also saved as a .pdf.

While a rather brief explanation of the whole process I hope it gives an insight to the architecture of the site, and the many complexities that can be encountered, and overcome in building a project such as this.

Launched!

Our datavis site has launched! http://look.iriss.org.uk

15th March was International Social Work day, and we were very proud to have Alison Petch (Director of IRISS) announce and present the site at the conference in glasgow. We also had a stand allowing delegates to have a first play with the tools on offer. For my own part it was the culmination of nearly 30 hours work over 2 days to finalise some of the tools, and it was a proud moment for me to see some of the tools being put to use for the first time.

Work still continues developing and improving the tool, and I’m looking forward to expanding on the visualisations.

Currently the registration system has a couple of issues, however if you wish to use the site, please email us at look@iriss.org.uk and well sign you up.
Regards,
Ben

Development continues…..


As the new site starts to fall into place, We’ve evaluated several Drupal modules and are working to develop some new ones.

Firstly, We’ve chosen the awesome sheetnode module, which allows users to upload spreadsheet data in various formats ( Excel, mac-excel and google spreadsheets ) into a Drupal node. not only does this offer a simple way for users to get their data into the system, but it also allows them to modify the data via a easy to use editing interface. Lastly ( and best of all ) each cell, row and column can be accessed through Views. All that power held in a single ( and woefully unknown ) module.  check it out here -http://drupal.org/project/sheetnode

We’ve also been working on a couple of new modules, Firstly  a simple api module to integrate Protovis into Drupal, with several simple charts built-in. Protovis is javascript library, and we are currently working on being able to save the Protovis in SVG format.  Development has also started on a standalone visualisation set ( project ‘quickviz’) which created simple stylistic SVG graphics to demonstrate single data points or small sets of data ( mainly percentages for single / small groups). Find out more about Protovis here -> http://vis.stanford.edu/protovis/

You may have noticed that we are concentrating on SVG as our base format. This has been chosen since it is a resolution- independent format ( you can re-size it without any loss in quality ). Our goal is to use imagemagick to allow users to convert the SVG to jpeg for when a user wants to download and use the image in their own site/publication.

Facebook – Visualizing Friendships

Visualizing data is like photography. Instead of starting with a blank canvas, you manipulate the lens used to present the data from a certain angle.

A facinating post from Facebook engineer Paul Butler in geographically visualising friendship connections on Facebook.  The lovely vis was created with the open source stats package R

Read more(be sure to checkout the high res rendering) : http://www.facebook.com/notes/facebook-engineering/visualizing-friendships/469716398919

Now cooking

Chef Danboard

We’ve been setting up a test-bed for the datavis tool in drupal, apart from the usual suspect like cck & views we’ve added a few other to the mix, here’s what we have so far:

Wireframes

We’ve been working on wireframes to show our pilot users so they can get a better understanding of how the website will work for them.  It’s been useful to get us thinking about the tool will work in terms of layout, functions.

These were all sketched out on paper & pencil before being transferred to more finalised versions and inked in with a sharpie.

We’ve attached the wireframes to this post if you would like to share any thoughts.

download datavis wireframes

look@iriss.org.uk

The What

Part of the data visualisation(dvis) work at IRISS is to develop a web based tool for creating & sharing visualisations.

The why

Visualizing data enables social work organisations and practitioners to gain new insights into data and provide new ways of navigating large amounts of data. This prototype will initially benefit the social work department the project partners with, but subsequently will have a bigger impact when the tool is made available across Scotland.

The How

Recently we’ve invested a lot of time developing sites & tools with Drupal, so the decision to use it was easy.  The community surrounding Drupal is rich & diverse with many thousands of people writing & sharing code.

Using shared code (or Modules) allows us to construct the mundane parts like users, roles, storage & import of data, permissions, sharing (social networks, email, embedding) rapidly.  This allows more time to be spent on larger features like generating the visuals themselves, the user interface & testing.

What will it look like?

Below is a sample of the from last years data vis pilot, Ian Phillip created this great visualisation from the eSAY data set provided by The Scottish Consortium for Learning Disability (SCLD).

Based on the positive feedback we received the options for re-creating this from your own data will be one of the first types we’ll be developing.

Employment of adults with learning disabilities 2008

Rapid Research

To quickly gather a snapshot of tools, services, examples & best practice we’ve been using delicious to tag resources scattered across the web. We decided to use the tag iriss_dvis to collect & easily share what each person finds. There’s a list of those link here on the blog so please feel free to explore.