Cloud storage and privacy – the dark side of the silver lining

A report of the Senate Environment and Communications Committee released this month may have profound implications for small businesses, councils and non-government organisations that use cloud storage for personal information relating to their clients or customers.

The Committee’s report, titled The adequacy of protections for the privacy of Australians online was prepared in response to recent significant advances in online technology and its use for social media and cloud computing. The committee was particularly interested in the impacts on personal privacy and the protection of data.

The report contains nine recommendations. Key points include:

• Small businesses which hold substantial quantities of personal information, or which transfer personal information offshore, should be subject to the requirements of the Privacy Act 1988.
• All Australian organisations which transfer personal information overseas, including small businesses, must ensure that this information is given the same protection as that provided under Australia’s privacy framework.
• If an organisation overseas collects information from Australia, this information should be protected by the Privacy Act.
• All Australian organisations that transfer personal information offshore are fully accountable for protecting the privacy of that information.

The Privacy Minister Brendan O’Connor has highlighted in an interview with News Ltd that cloud storage was a particular concern.

“While some ‘cloud’ providers are located here in Australia, many more are located overseas,” he said.

“That gives rise to difficult jurisdictional issues, particularly where the laws of two or more countries could potentially apply.”

The Minister has indicated that the recommendations will be implemented.

“This is an important development that will prevent organisations from trying to avoid their obligations under the Act by transferring the handling of personal information to countries with lower privacy protection standards,” Mr O’Connor said.

This point is especially relevant as many countries do not offer the same degree of protection for personal data stored on servers in their jurisdictions against it being accessed by government agencies, or even shared with businesses.

If implemented, the recommendations will have obvious consequences for organisations large and small across all sectors that have made explicit decisions to save costs by moving client data to cloud storage. Even if they have received promises regarding the location of data storage facilities and their security, these assurances are very difficult to check.

However, these are not the only organisations that may be caught up if the recommendations are implemented. The buzz around cloud computing and storage – not to mention the potential cost savings - has encouraged many small organisations to experiment with online applications for their day-to-day computing, such as Google Docs, which rely on cloud storage. The files generated may include a range of personal data which could be covered by the proposals.

In addition, many organisations that continue to use “traditional” PC-based software also rely on cloud storage applications such as Dropbox for data backup or portability. So even if an organisation’s primary storage of personal data is located in its Australian offices, additional copies may be kept on an overseas server and would therefore be caught by the recommendations. Furthermore, the current exemption from the requirements of the Privacy Act enjoyed by small organisations would be removed, even if they do not store data overseas.

While the Senate committee report offers sound arguments for the proposed changes and nobody would deny the fundamental right of people to expect high standards of privacy to protect their personal data, the government needs to work closely with all organisations including small businesses,  NGOs and government agencies as well as cloud storage and software providers to ensure that implementation of these recommendations is not too costly or onerous.

Smartphones – back to the linear – but mashed – future of social media

In my last post I discussed my recent acquisition of a new iPhone 4. It only takes a short while to realise the revolutionary implications of these devices for the future of social media and Web 2.

Facebook iPhone interface

Checking tweets or Facebook and LinkedIn updates is a no-brainer. The linear nature of most social media interfaces also means that there is comparatively little difference in the user’s experience between a smartphone and a PC, despite the latter’s vastly larger screen real estate – and of course you can add comments and updates anytime, anywhere.

The same, linear, slightly retro feel applies to switching from app to app and to opening multiple files within the same application – just like the early versions of Windows, or the first time you fired up Internet Explorer (obviously the experience is a little different on tablets such as the iPad because of their larger screens).

On the other hand, the ability of smartphones to integrate different applications with social media is probably their strongest feature next to their portability and for ease of use beats computers hands-down. The most obvious example is the act of taking a photo to email or to post to a social media website, both of which are accomplished in a few seconds and just two or three steps on most smartphones.

This is a major advantage in terms of convenience and time saving, compared to using a digital camera, uploading the picture to a computer and then sending it. The quality might not be as good and you may not have the range of tools available on a PC to manipulate the image, but for most day-to-day purposes, who cares?

This mashing of applications on the iPhone can be used in less obvious ways. Recently I had to scan a receipt to email as part of an expenses claim. Again, it was far easier and quicker to use one of the many iPhone scanning apps rather than crank up a high-quality but slow scanner and turn on the computer.

 

Dragon Dictate iPhone app interface (recently released in an Australian accent version)

OK, the iPhone’s scan image wasn’t as great as the scanner could have done, but I didn’t need OCR quality and it was certainly good enough for my purposes. The same triumph of convenience over quality applies to a host of other things involving applications and social media which are easy to accomplish on the iPhone, from emailing voice recordings to posting movies to YouTube, or using the Dragon Dictation app to compose your next Facebook post.

 

 

As a result, I think we are at the beginning of the next wave of innovation in social media and the web generally – and as smartphones become ubiquitous and are increasingly the weapon of choice for going online, the potential for all sorts of combinations involving apps and social media seems almost limitless.

Web scraping – the Jekyll and Hyde of Web 2.0 (part 2)?

In the last post I wrote about a positive use of web scraping, the software process used to extract data from the HTML mark-up language used on websites. I highlighted Planningalerts, a web and phone app to deliver real-time information gathered from council websites about development proposals in that may affect a specific property.

Now for a look at a social media application which opens the door to web scraping in a much more controversial manner. Foursquare is described by Wikipedia as a web and mobile application that allows registered users to connect with friends and update their location. Points are awarded for “checking in” at venues.

Foursquare website

Basically, Foursquare broadcasts the user’s location to their friends and if they wish to allow it, to other Foursquare members, based on their smartphone’s global positioning system (GPS).

The process is best described, not on the Foursquare website but in a recent Guardian article. In summary, users “check in” on their phone whenever they arrive at a point of interest so that fellow users know where they are. They can also use their phone to check the names of all the other users in the same area, where exactly they are and if they are with other users

The application is still in its infancy but is already attracting a lot of users. Recently it signed up its two-millionth user, just three months after reached its first million. According to the Guardian:

“Foursquare is now being widely touted as the app which will, after years of anticipation and prediction, mark the beginning of ‘life as a game’ computing. Whatever you do, wherever you go, you will be scoring points, earning ‘medals’, and be in, at the very least, social competition with other users around you.”

However, as the Guardian article points out, this “game” could come with a price – a potentially huge loss of privacy. There are at least three areas of concern. First, by its nature, Foursquare automatically reveals a fundamental item of information, the user’s precise location, which is not disclosed even to the user’s friends by any other social media application. This has implications which are only beginning to be understood.

Second, while they have recently been tightened up, Foursquare privacy settings still require users to actively opt out in relation to key options to share data instead of the other way around. As the recent fracas over Facebook privacy rules demonstrates, this approach can leave users very vulnerable.

This is a particular issue with Foursquare, however, as there is little point to the program unless you choose to release your location information to at least some other users. However, even if you do opt to disclose your location only to your friends, this can still be risky, especially if you haven’t been too discriminating about who your “friends” are.

This risk is also compounded by the way in which the program facilitates the linking of Foursquare’s locational broadcast to a user’s Twitter feed, thus enabling their location to be spread even more widely.

 The biggest concern however, is that Foursquare could be potentially vulnerable to “malicious” web scraping. Unlike the Planningalerts application’s use of web scraping described in my last post, this involves the collecting and collating of private data that users have revealed (intentionally or otherwise) on social media websites rather than gathering public information made available on council or government websites.

Even if a user avoids the temptation to link their own Foursquare, Facebook, Twitter and other social media accounts to reduce the risks described above, someone with the right skills can gather pieces of information from these sites and link it with other publicly available information such as phone directories and electoral rolls to build a detailed picture of that user’s address, employment, lifestyle, friends, associates, shopping preferences etc.

The role of web-scraped Foursquare data could be particularly critical in providing information on the user’s movements on a day-to-day basis. The Guardian sums up the risks:

“The big worry … is who might get to make use of this information. Pick your paranoia. Someone with criminal intent, such as a burglar, identity thief or stalker? Governments, the security services or police? Terrorists? Or a corporation looking to target its products at you with incredible precision?”

This is not to condemn web scraping and similar data-gathering techniques out of hand – as Planningalerts demonstrates, they can provide a particularly effective way of making already publicly-available data even more accessible. Nor is it a criticism of the innovation demonstrated by applications such as Foursquare. It does however provide a strong argument for all social media applications to beef up their privacy measures and to inform users of all the risks involved.

If we are going to march into the brave new world promised by Foursquare and the other locationally-enabled social media apps to follow, we had better do so with our eyes open.

Web scraping – the Jekyll and Hyde of Web 2.0 (part 1)?

Two media reports this week have highlighted the way in which social media and web 2.0 applications can use “web scraping” in very different ways and for very different outcomes.

Web scraping is a specialised software-based process used to extract data from websites, where it is commonly displayed using HTML or similar mark-up languages. When displayed in this way, it is difficult for conventional software trying to “read” the data to make a distinction between relevant information and the surrounding “noise” of the formatting, a situation complicated by the fact that websites may display similar information using very different layouts.

After the data has been “scraped” from the relevant websites it is collated in a database or some other systematic framework and put to new purposes, possibly not ones that were envisaged by the original creators of the information. The most common examples are the websites which provide on-demand price information for a product selected by the user, based on real-time comparisons between the web pages of various online shops.

Web scraping can be viewed as a specific form of data scraping, or the extraction of data from the human-readable output of any computer, but for the purposes of these posts I will regard the two terms as synonymous.

First, a positive example of web scraping at work. The not-for-profit organisation OpenAustralia, which has already made a name for itself in making information about the federal parliament, parliamentary debates and individual MPs more easily accessible, has just released a new app for iPhones and Android smartphones. The app gives users the ability to locate neighbouring development proposals that may affect a property just by pointing thier mobile phones at the property.

Planningalerts phone app.

The app is an extension of OpenAustralia’s already-successful web-based planning alerts service. This allows users log the address of a property and be informed by email of development proposals within either a 200 metre, 800 metre or 2km radius, assuming the property is located within the boundaries of a council that provides this information online in a format that has been scraped by the software.

This is not without its complications. In an interview with the Sydney Morning Herald, OpenAustralia founder Matthew Landauer describes the process as “very painful and error prone”:

“The program clicks on links, fills out forms to do searches, and then when the program finds the web page with the development application it has to extract the unstructured information on the page and turn it into structured information.”

Only 85 of the over 650 councils in Australia are covered by the software, but these include many of the larger ones and others are being added to the system by “crowdsourcing”: members of the community adding the details of the websites of councils which start to place their DA information online.

www.planningalerts.org.au website

So far, so good. There would be widespread agreement that this is an appropriate “repurpose” of information which is publicly available anyway. Some councils may be nervous about the way in which the OpenAustralia phone and online applications raise the bar in terms of who gets notified about DAs, but hopefully most will take a positive attitude and cooperate with this initiative by making information available in a more standardised, software-readable format. In turn, this and other eGov initiatives may lead to councils and other levels of government making a wider range of data available in computer-accessible formats.

However, web scraping can be used in different and less benign ways. In my next post I will look at a new application which raises major questions about the use of web scraping and related techniques in relation to social media websites.

If the iPad is the answer, what is the question?

Recently I extolled the virtues of netbooks as an ideal device for travellers, due to their light weight, long battery life and very low (sub $500) cost. When the iPad was subsequently announced I assumed that it would be a “netbook killer”. However, after reading about them and having a brief play with one, I’m not so sure. In fact, once all the current enthusiasm for the iPad dies down, I don’t know exactly what its role will be.

This is not to knock the iPad, through which Apple has done to the tablet computer what it did through the iPhone to the mobile phone – remake the current crop of disparate poorly-implemented devices as a well-executed package integrated with a great, intuitive interface which can be used as a platform for a range of applications. 

iPad (from Wikipedia)

The difference of course is that the mobile phone was already a ubiquitous device when the iPhone came along. Millions of people already had the things and knew roughly what to do with them, though Apple was certainly successful in making a better interface and extending the range of phone-based activities.

 In geological terms, the mobile phone market was not so much as a niche as a very large depression, one only partly filled by a lake of existing devices which the iPhone successfully swamped. To labour the metaphor, the iPhone then went on to fill up most of the neighbouring niches and creating new ones, courtesy of the vast range of iPhone apps.

By comparison the tablet was only a very small niche before Apple arrived. While the iPad is likely to fill this relatively small hole, this is not that significant an achievement. Apple and the apps providers also need to fill a lot of connected niches  – and create new ones to fill as well – to give the iPad critical mass.

This is complicated by the fact that, because of its size, the iPad is unlikely to become a ubiquitous device like the mobile phone – for a start, you can’t just put one in your pocket, or indeed the average handbag, as you zoom out the front door. This will obviously limit its numbers, unless a great killer app or two can be found.

Such an app is unlikely to come from the ranks of iPhone apps ported to the iPad – after all, the iPad can use all of these already. It will likely have to be something new which is best suited to the iPad form factor. One suggested example is the delivery of newspapers, though electronic paper could well do a better job of displaying these (that is, if anyone ever actually releases electronic paper displays on a commercial basis). It may be that the killer iPad app is yet to be written.

This brings me back to the iPad’s potential as a netbook killer. There is little doubt that the iPad may be the ideal device for some travellers, especially those who just want to send and receive emails, look up websites (provided they don’t use Flash) or upload and post holiday photos (provided they buy the appropriate attachment). It also has almost twice the battery life of a netbook and would no doubt suit many (well-heeled) tourists.

However a netbook can do all of these things, albeit less elegantly, and is easier to use for anything more complex such as word processing or photo editing. It is also a lot cheaper – ironically, your average business traveller who can afford to buy an iPad may prefer to stick with a netbook because it is much easier to use the latter to type up that last-minute report on the early-bird flight to Melbourne. This response is also consistent with the suggestion that iPads are best thought of as content consumption rather than content production devices.

While I can’t predict whether such a killer iPad application will emerge or what that application might be, it’s interesting to consider how the iPad might be used in relation to some of the Web 2.0 applications I have previously reviewed. Those workshop or conference tools such as Poll Everywhere or iMeet which require interactive, real time participant input might benefit the most. MyCommittee also has potential, but only once it can be used to run actual committee meetings themselves. 

Perhaps more intriguing are the web-based mind mapping applications I reviewed last year, such as Comapping, Mindomo and MindMeister. These may also require some adjustment but they would offer the potential for brainstorming or mind mapping for small groups using iPads.

In summary, while the jury is still out on the extent to which the iPad will succeeed, there is a lot of potential to use the device to capture feedback interactively in meetings, workshops and conferences which should be explored by NGOs and small businesses – that is, if you can afford to buy the things in the first place. I’d be interested in any suggestions about other Web 2.0 applications for the NGO or small business sectors that are designed for iPads or which might be particularly useful in the iPad format.

iMEET! – a big step towards 21st century conferencing

A while back I reviewed Poll Everywhere – a Web 2.0-based tool for organisations to conduct polls and surveys at meetings, workshops and conferences. The software enables attendees to participate and vote via SMS text messages from their mobile phones in real time. 

iMEET! is aimed at a broadly similar market and is also web-based, but with a somewhat different direction and emphasis. I was fortunate to get some experience in using iMEET! at a workshop recently (disclaimer: I was also helping to run the workshop, though I had nothing to do with the choice of facilitator or technology). 

Just as Poll Everywhere introduced a modern take on voting in conferences, iMEET! does the same for the traditional process of gathering and processing workshop feedback. 

I think everyone has sat through one variation or another of the seemingly timeless process of workshop brainstorming sessions at conferences. You know the drill – everyone breaks into small groups, debates who is going to be the scribe and then leaves that person with the thankless task of recording the discussions on multiple sheets of butcher’s paper. 

These are then used to report back to the whole group in some sort of plenary session – and then after the meeting some poor soul has the job of transcribing a mass of notes, scrawled in different hands, into some sort of coherent report of the outcomes. The process is time-consuming, and to be frank, unexciting, which means that conference organisers all over the world have to deal with the problem of conference participants nicking off before the dreaded final report-back plenary session. 

Relatively few technological innovations have impacted on this process since, it seems, the beginning of time. One was the introduction of electronic whiteboards, but these are clumsy and relatively expensive and most venues usually have only one or two. They really only lend themselves to being used by a small group holding a plenary workshop session. 

Other innovations such as the use of small networked voting pads have not really caught on because of their proprietary nature. Laptops have also been used, but while these have facilitated the process of recording workshop outcomes, there has not been an effective application to integrate the results in a plenary session-friendly discussion format. For most of the workshops I’ve attended and convened it’s been the good old butcher’s paper, even though butchers themselves have long since moved to plastic! 

Enter iMEET!, which is making a strong claim to being the game-changing application for conferences and workshops. It uses lap-tops, but integrates them in a very innovative way, linking them wirelessly so that information can be entered into a web-based iMEET! database directly. 

To quote the company’s website:

“[This] information is stored on a central server and made available to all other laptops, and projected via a data projector onto a large screen for all to view. All that you need to be able to do is enter content via the keyboard into the intuitive interface on the computer, using a standard web browser.” 

In practice the system is easy to use. Each small group is allocated a laptop and in most cases still has to appoint someone to record the outcomes of the workshop (or “focus session” in iMEET!’s parlance). However, instead of scribbling on large sheets of paper, the scribe enters the comments into an interface on the laptop which is a little like an on-line forum (in theory each member of the group could take turns to enter their own comments, but having a single person as the group’s recorder provides more consistency and helps to ensure that the recorded comments are mediated by the whole group in some way). 

Example of iMEET! screen as seen by a workshop focus group (from the iMEET! website)

Drop-down boxes can be added by the facilitator for groups to categorise their comments as they make them – for example, they could be asked to characterise their comments as either, say, positive or negative, or as short, medium or long term actions. Typically, each group sees only its own comments on its laptop, however, the responses are brought together on a single computer to be reviewed by the workshop facilitator and displayed to the whole workshop on a large screen in real time, with the group making each comment clearly marked. 

During the session itself or immediately afterwards the facilitator can prioritise and categorise the comments, bring together related ideas or identify potential conflicts. The fun really begins in the subsequent plenary session, which is much faster and more interactive than usual – no more succession of people dragging sheets of butcher’s paper up to a podium and reading out their hand-written comments. 

Instead the outcomes are immediately available and the facilitator can lead the whole group in making sense of them, identifying key issues and seeking responses from the audience. The material can be reshaped or expanded as a result of input from the whole group. For example, a set of options for future action can be selected and the participants invited to vote on them. As with Poll Everywhere, the outcomes are displayed in real time, though in iMEET!, the number of laptops available at the workshop is obviously a limitation. 

iMEET! screen set up for a work group to vote on options (from the iMEET! website)

After the meeting the final output can be delivered as a Word or Excel document within 24 hours of the event – or the iMEET! session left open for further online comment by participants (via a secured website) with output provided at a nominated time. 

Verdict: having seen iMEET! in action I can vouch both for its effectiveness and for the speed with which the results were produced. I was very impressed, as were most of the workshop participants. 

There is one important difference between iMEET! and programs such as Poll Everywhere and MyCommittee (which I have also reviewed) – even though it is web-based, iMEET! doesn’t lend itself to being treated just as an off-the-shelf product. It really needs at least a preliminary consultation with the consultancy team behind it (Australian-based Global Learning) to understand its full potential and to set it up properly. 

It also works best when workshop participants can be split up into relatively small groups – say no more than five or six people per laptop. This makes it easier for everyone to see what is being recorded and also allows the group’s recorder to keep up with the input. 

Global Learning can provide facilitation services as well as the laptops, or the client can use their own. Any reasonable PC or Mac laptop with reasonable battery life will do, as long as they can access the internet directly or through a wireless LAN. Even netbooks could be used, though the latest crop of CULV laptops with larger screens might be a better bet. 

As a result of the number of options involved it is probably best to get pricing information directly from iMEET!, though I understand that it is available on a conference-by-conference basis. 

The really interesting thing is how iMEET! rounds out the suite of meeting and workshop-related products which are relevant to small and medium organisations, complementing applications already in this arena such as MyCommittee and Poll Everywhere. 

While there is some degree of overlap between these products – particularly between Poll Everywhere and iMEET! – it’s best to think of how these applications can be used together creatively. For example, MyCommittee could be used to setup a meeting, and iMEET! to run workshops or Poll Everywhere to record votes on important issues during the meeting . Or iMEET! could be used to workshop key options through small groups prior to a major conference at which Poll Everywhere is used by a larger group of participants to vote on these options. 

In summary, iMEET! fills an important, if hitherto largely ignored, niche. It could help you to make your next conference much more productive and who knows – the dreaded end-of-day plenary could even become a lot more interesting for you and your participants.

 As always, you should look at the issues I have raised in previous posts about avoiding problems with Web 2.0 applications in evaluating iMEET!

In praise of netbooks

Sometimes you come across a tool so useful you wonder how you worked without it.

Over the past few months I travelled extensively – and for the first time with a netbook. I had a fairly typical example, an MSI Wind U100+ (where do they get these names?) with an Atom processor, 2GB Ram and a 160GB hard drive running Windows XP. It is a rather natty blue and apart from a slightly-tempremental wireless card, performs perfectly. Its light, easy to pack and can do most things a laptop could do a few years ago. Its certainly capable enough for word processing, web-surfing and email, which make up what 90% of travellers do 90% of the time.

Many ultra-portable laptops also have these virtues, but in my view these aren’t the main reason for buying a netbook. There are two reasons why netbooks make the ultimate traveller’s tool: the first is their phenomenal battery life – around five hours with a 6-cell battery, which is still better than most “conventional” laptops. The second is how cheap these things are – I picked up mine for under $A500. This means that if the thing gets stolen it isn’t the end of the world (except for the data you have on it, of course).

I’ve travelled with laptops before and its great to have a comparatively large screen and a full set of programs, but their comparatively poor battery life and the fact that they are considerably more expensive can make them more of a liability than an asset.

The wider issue, however, is how easy it is to access to social media and online tools is when you are on the road, irrespective of what device you are using. There is of course wireless broadband, but this can be an extremely expensive proposition, especially overseas. Access via wi-fi can be extremely variable – it can be free in hotels, but elsewhere often involves casual user charges, where it is available. This means that you may have to lower your expectations of being able to access key programs if they are available only online – or look at those applications which can be downloaded and used offline.

Update – MyCommittee online meeting tool improved

Back in June I wrote a short review of MyCommittee, an online tool to manage committees and meetings. I concluded that there was a lot to like about MyCommittee and that it would suit a small to medium organisation with reasonably complex agendas.

I also highlighted some issues with the interface and a number of limitations, including the fact that the task management feature was problematic for organisations with a staffed secretariat as actions could only be scheduled for committee members. There was also little ability to change the terminology in the agenda, but my biggest concern was the inability to view resolutions in anything other than chronological order and the lack of any facility to extract them for storage in a separate database.

I took up these shortcomings with the program’s creators, who indicated that they might be considered in the next release of the program – and that’s just what has happened. Paul Cox from MyCommittee has just responded, noting that the latest version addresses some of my concerns. I’ve only had a chance to check it out briefly but I’m pleased to confirm what he’s said – plus more. MyCommittee now has the following:

• More flexibility in customising the agenda template – for example, changing the terminology and order of the standing items at the beginning and end.
• The ability to create custom lists of resolutions, by date, keyword, status, etc.
• A facility to export any list of resolutions in PDF, CSV or XML format to print or import into another application or database.
• Access for committee members to subcommittees (to view and download meeting minutes, resolutions, etc.).
• A greater range of roles for committee members, including “resource”. This could cover non-voting staff attending the meeting, who I assume could be allocated tasks, but I have not yet tested this facility.

Of these improvements the ability to create custom lists of resolutions and export them is the most significant. Not only does this make it easier for organisations to set up their own resolutions databases, it also provides the ability to download and backup resolutions offline. I suggested in my review that organisations needing this facility should hold off until it was implemented – but with its incorporation in the latest version, now is definitely time to have another look at MyCommittee.

Web 2.0 tools unlock statistics 3 – accessing a whole “world of information”

In my last posts I looked at Australian examples of the innovative use of Web 2.0 to provide free, interactive access to census and other statistical information. This time I’ll shift scale and provide a brief overview of two sites which take a Web 2.0 perspective to the presentation of international statistics.

UNdata was recently launched by the United Nations Department of Economic and Social Affairs (DESA)  Under-Secretary-General Sha Zukang, who states:

“The UN System has accumulated over the past 60 years an impressive amount of information. UNdata, developed by the Statistics Division of DESA, is a new powerful tool, which will bring this unique and authoritative set of data not only to the desks of decision-makers and analysts, but also to journalists, to students and to all citizens of the world.”

The site proclaims that it is a “world of information” and this is no exaggeration. Since its foundation, the UN has been collecting statistical data from member states on a huge variety of topics. As the site states, this is a considerable asset but it was stored across a large number of separate databases, each with different access policies.

UNdata overcomes these problems by pooling these databases into a “single internet environment”, allowing access through a keyword search on the organisation’s home page. There is also a facility for advanced search and country profiles are provided.

The site claims there are over 55 million data points covering a wide range of themes including agriculture, education, employment, energy, environment, health, HIV/Aids, human development, industry, IT, national accounts, population, refugees, tourism, trade, as well as the millennium development goals indicators.

UNdata opens with a somewhat understated home page, “data” which centres on a search facility through which databases can be accessed. Type “health” for example, and 57 databases referring to health are listed. This is augmented by a simple menu system with four alternatives – data, glossary (which lists all the terms used in the databases), wiki (which provides access to a linked wiki which describes all aspects of the UNdata project) and advanced (which provides access to country profiles, advanced search and other features).

Below the search facility is a link to the Monthly Bulletin of Statistics (MBS) which provides economic and social statistics for more than 200 countries and territories. Under this is a section divided into three – Databases (which lists the key UN databases by category), Updates (which shows the latest additions to the databases) and Country data services (which provides links to the government data service of each UN member).

Clicking on a database name provides access to a fairly orthodox table, with columns typically for country, subgroup, year, source and unit value. The table can be sorted by any column and unwanted columns can be deleted. The tables can be turned into pivot tables or exported.

Whilst the front-end customisation features are relatively modest by Web 2.0 standards, a huge amount of work was obviously involved in bringing a large number of disparate databases together, integrating them on one site and displaying this volume of information in a consistent and accessible format.

If you want statistics on life expectancy in Africa or education in South America – or to find out how Australia performs in comparison to other countries in the world – this is the place to go. The country profiles are particularly useful, providing a statistical snapshot of almost all UN member states – and my one criticism of UNdata is that these could be better featured on the start page.

The second international site that I want to look at is linked to UNdata and picks up where it leaves off in terms of presentation. As the site states, Gapminder is a non-profit site “promoting sustainable global development and achievement of the UN Millennium Development Goals by increased use and understanding of statistics and other information about social, economic and environmental development at local, national and global levels”.

The site uses Trendalyzer software to convert tables into animated and interactive graphics. The site opens to four boxes – the two on the right devoted to videos of Gapminder in action, the left to the program itself. The main action is in the top left box, “Gapminder world”. Clicking on this leads to a graph which will show dynamically over time the changes within and between two national variables – for example, size of families and infant mortality, or urbanisation and population growth.

Depending on the variables chosen, the story that Gapminder tells can be extraordinarily vivid – and at times very depressing. Regions are depicted by different colours and its possible to tag specific nations to track their progress (or otherwise) over time. Maps can be attached to emails or blogs – I haven’t tested this, but it appears to send html code that links to the Gapminder site and loads the variables that you have used in the map.

It’s definitely worth viewing the tutorial and downloading the pdf document that summarises how the site works. You also need to experiment with the variables you wish to display and to play with the controls scattered around the graph (which could be a bit larger and clearer). Its also important to note that just because there appears to be a correlation between two sets of data does not mean that there is a causal relationship between them – and of course the quality of some of the statistics may be higher than for other data sets.

Verdict: Both UNData and Gapminder are invaluable tools for organisations interested in international aid or issues such as poverty, health or education, as well as those who want to benchmark Australia’s performance on key data sets against that of other countries. The two sites complement each other, so that the UNData site can be used to prvide a detailed snapshot of a particular issue – or through the country profiles, a picture of a specific country – whilst Gapminder can show in a vivid way the story of change over time.

The Trendalyzer software which powers the Gapminder site is available on Google Docs and I will look at it in a future post.

Web 2.0 tools unlock statistics 1 – ABS releases TableBuilder

Web 2.0 tools unlock statistics 1 – ABS releases TableBuilder

So far in this blog I’ve looked mainly at end user applications, but this is not the only area in which Web 2.0 strategies are appearing. Another aspect which is also beneficial to councils, small businesses, NGOs and other organisations is the increasing use of Web 2.0 technologies and interfaces to provide masses of free statistical data and, perhaps more imprtantly, to make this information easier to use.

The provision of statistical information by governments parallels the development of the internet, albeit with a substantial time lag. Up to only a decade or so ago, most government data was available only in hard copy. Some material was on microfiche, or, if you were really lucky, on floppy disk and eventually CD – and you were often charged for the privilege of accessing it.

Eventually and somewhat grudgingly governments started making data available online, but usually in the form of static tables on websites or in PDF documents. Even where it was available in spreadsheet format, the size of the files and download speeds at the time often meant that CDs were still the best format for distribution – and there was little or no customisation.

This started to change with the advent of broadband. The advent of Web 2.0, changing community expectations and government attitudes have also combined with the emergence of third party providers to make this data much more available and accessible. There has been a dramatic increase in the data provided online by governments and they have (mostly) given up charging for it.

One of the oldest of these Web 2.0-influenced government sites and the one probably best known to Australian organisations is the Australian Bureau of Statistics (ABS). The ABS has made material available online for a long time but has made increasing use of Web 2.0 interactivity, especially since the 2006 Census, to present census information through a range of products.

This process has greatly increased the accessibility of census data and also brought down the cost, as many of these products are free. Thus small businesses, community organisations and even individuals can develop their own profiles of their potential customer base, service catchment area or local community without having to hire consultants or researchers.

The ABS products which show the most Web 2.0 influence include:

• QuickStats, which provides a summary of key Census data relating to persons, families and dwellings in user-selected locations, compared to national figures. This is the simplest and least interactive of the ABS Web 2.0 products and therefore has limited options – you choose a location and it displays a summary table. You have no choice over the data Quickstats displays and you can also only choose one location at a time.
• MapStats, which provides thematically mapped Census statistics for a chosen location. It is step up from Quickstats, as you get to choose both location and data. In theory you also get to choose the geographic boundaries that will be used for the thematic mapping, but in many cases this is predetermined for you by the type of location you have selected. The maps which can be downloaded are quite effective but the lack of control over the level of boundaries used can be frustrating. You also can’t access the underlying data.
• CData Online, which enables users to create their own tables of census data at levels ranging from a single collection district to the whole country. Unlike the previous products you can choose to start with either locations or topics and you have much greater choice over both. The resulting tables are very detailed and can saved in a workspace on the ABS website or downloaded. CData Online is obviously more influenced by Web 2.0 than the preceding products and is therefore much more flexible but it requires a fair degree of work to produce the result you want.

 The ABS has just added substantially to its Web 2.0-enabled stable with the release of TableBuilder. This new online tool allows users to create their own tables of census data by accessing all variables in the census output record file for all ABS geographic areas. Users are able to prepare their own cross-tabulations and smaller tables may be exported or viewed as a graph or a thematic map within the product, whilst large tables can be downloaded to the desktop for further manipulation.  

 

In effect, TableViewer combines almost all the features of the other ABS Web 2.0 census-based products – but all this comes at a price. TableViewer is a charged subscription service costing $1,655 (including GST). This price is for individual users for a “single Census reference cycle” and includes online training and ongoing help-line support.

TableViewer is a very sophisticated product which allows a much finer grain of manipulation of census data online by the end user than even CData Online does. Whilst a trial version is not available, the online and interactive Tableviewer tutorials provide a good idea of what the product can do and how it performs.

Whilst the ABS products provide a wealth of census information in an accessible format, a Melbourne company, .id (informed decisions), has taken Web 2.0 concepts a step further to produce a range of user-friendly interactive profiles. These I’ll take a look at in the next post.