Recently the multi-talented IRISS programmer, Ben Lyons tool part in the BBC’s scraperwiki hacks and hackers day. The event focussed on getting hacks (journalists) and hackers (programmers) to work together on developing a tool which would bridge the two worlds.  Ben and programmer Paul Miller formed a team with theBBC journalist Chris Sleight who reports on the central scotland area (http://www.youtube.com/watch?v=TeH6Ca77coY).

Chris had an interest in obtaining data from the central scotland fire and rescue service(http://www.centralscotlandfire.gov.uk/news/latest-incidents/recorded-incidents.html), which the team agreed to focus on. While this site was useful to Chris, there was also no easy way for him to obtain updates when there is an fire incident, nor a way to use the data in any really meaningful way.

After some intial digging the team found a way to use scraperwiki to scrape limited data from the site (at circa 60 records at a time), and also discovered another 14,460 historical records, which allow them to start building several outputs.

Within one day the team managed to build a dynamic protoviz visualisation from within scraperwiki (http://scraperwikiviews.com/run/firebug_simple_viz_2/), as well as extract some very useful statistics (the one they chose to work with on the day was the percentage of malicious calls per area). They also did some initial work on creating geo-mapping views, which is one of IRISS’s focus for its data visualisation team this year.

For the team’s impressive efforts they were awarded second place on the day, and also won the scraperwiki “best scraper” award.

As a follow up, Paul Miller spent some time the following week finalising some additional features, including notifications on twitter @FireCentralScot and an interactive googlemap (http://central-scotland-fire-reports.paulmiller.it/fire/map/custom ).

You can watch Ben speaking about this great event on youtube: http://www.youtube.com/watch?v=3RQ1f6vUMoM

Well done, Ben!

**Warning heavy tech content**

With the launch of the D-viz toolkit, its come time to describe a little of what happening under the hood (and to an extent why).

Flowchart of the visualisation process in look.iriss.org.uk

First, a note (or two) about our intended audience.
While the d-vis toolkit is freely available to all, our goal was to open up the ability to make visualisations to those who work in the social services, in the public, private and voluntary sectors. in many cases the end users are often not tech-savy and in addition are forced to use ‘legacy’ systems ( for instance internet explorer 6 is still the most prevalent ), and often use of java ( and in some cases even javascript ) is not allowed. This was an important consideration in building the toolkit. We needed to build a system that was easy to use and used technology that would work across the board.
One early decision was to use SVG as our chosen base format, which has some advantages as a resolution independent format (it also allows use the ability to have interactive visualisations in the future), though it is not supported in IE until the latest version (and we’ve received some indications that its support is not complete )

Data input.
Data has been split into 2 types words and numbers.
Words are simply copied and pasted into a text box, he hope to allow the upload of MS word documents at some point in the future.
‘Numbers’ can be uploaded as excel spreadsheets. This is done using the excellent drupal sheetnode module, which handles both the upload of several formats of spreadsheet, but also displays them in a familiar interface for users.

Dashboard
The dashboard is our term for the form which allows you to choose your options. visualisation options are presented depending on the type of data you uploaded, once chosen you are given supplementary options depending on your chosen visualisation type.

Visualisation (aka the fun part).
To build the visualisations we built 3 different (for lack of a better word) ‘engines’. I’ll describe the protovis engine in depth later, but here are the other two described in brief.
wordclouds, The data is parsed and sent off to a remote server, which generates the svg and returns it.
ISOTYPES, this module generates an svg based on preloaded svg paths for icons, changing size, colour and the number of items based on the data provided.

Protovis.

flowchart for the protovis engine

There are several javascript tools in existence that make fantastic visualisations that are rendered in SVG, One of our favourites was protovis ( http://vis.stanford.edu/protovis/ ) which was created by Mike Bostock at standford university. but we had a problem, protovis, being javascript ran client-side, rendered in SVG and there were issues saving the final image as a file. Enter rhino / envjs which allowed us to run javascript server-side. Since this was technology we hadn’t used before ( nor did we have the time or resources to test ) we decided to run this on a separate server.
There were several limitations to this. Firstly all the instructions had to be hard coded into the javascript file, we also had to create a batch file which instructed envjs. Lastly due to the limited DOM, working with test options caused issues. Text issues were solved by appending the title to the end of the SVG document, and font/size options by rewriting the SVG tag at the beginning of the document. the SVG file was then read and sent back to the drupal server as a string, where is is saved as a file once again.

Once drupal has an SVG file through this process it creates a visualisation content type, and saves the SVG to it. the SVG is also rasterised and a standard image (firstly so it is in a familiar format, secondly to ensure all users can view it). and it is also saved as a .pdf.

While a rather brief explanation of the whole process I hope it gives an insight to the architecture of the site, and the many complexities that can be encountered, and overcome in building a project such as this.