Posts Tagged ‘open data’
The outbreak of the Ebola virus disease (EVD) in western Africa that started in March this year is the deadliest outbreak of EVD to date. In August the World Health Organization declared the epidemic to be an international public health emergency and in a statement on September 26 said that “The Ebola epidemic ravaging parts of West Africa is the most severe acute public health emergency seen in modern times. Never before in recorded history has a biosafety level four pathogen infected so many people so quickly, over such a broad geographical area, for so long.”
Data on the epidemic can be hard to find. Each affected country produces differently grained data and usually it is published in daily pdf reports (here are the Liberia reports, for example). PDF, while possibly useful for something, is generally a really bad format for public and open data. It’s hard to extract without human intervention and generally is a pain.
Recently we were approached by Ola Rosling of the Gapminder foundation, urging us to import into DataMarket metrics on the EVD epidemic. He got us in contact with Caitlin Rivers, a computational epidemiology student at Virginia Tech, who has been collecting the data from the official PDF files and making public in a Github repository. A fantastic initiative. She, along with others, has also been publishing some great insights into the data at her blog.
Today we are making public four datasets with EVD data on DataMarket. These sets are free for anyone to use, either on our site or via our API, our R package or by export into another system (vía CSV or Excel).
(Update: If you are working with this data using our API and hit the request limit for free API usage, just reach out to our support, and we’ll set you up with a generous quota to continue your work).
Caitlin provides us with these datasets:
Additionally we now also import Sub-national time series data on Ebola cases and deaths from the OCHA Regional Office for West and Central Africa:
You can access these sets and others relating to the epidemic from a single topic-page here. They will be updated with the most recent data on a daily basis.
We know that there is more work to be done getting data on EVD into usable formats and if you want to help, check out Caitlin’s Github repository. There are PDF’s that need digitizing and data that needs cleaning. Note that there might be discrepancies in the data. If you notice anything that seems off, do not hesitate to let us know.
We will continue to add to our data collection as we come across useful data.
Last week I attended O’Reilly’s Gov 2.0 Expo in Washington, DC. All in all a pretty good event (few organizers run conferences as smoothly as UBM/O’Reilly).
For obvious reasons I was there mainly to get the heads up on any Open Data efforts that I might not know about yet and to meet with like-minded people from all over the world.
Not surprisingly, the attendees were largely from the public sector, but there were also several suppliers and a few start-up companies that are working in this space – signs that the private sector is starting to realize that something big is about to happen here.
In one of the better sessions, Clay Johnson from the Sunlight Foundation talked about Truly Open Data. During Q&A there were several questions about how the private sector could be a part of the whole Open Data thing. Where the commercial opportunities were?
In the discussion that followed, the theme – and in fact this was the gist I got from most of the private sector people I met there – was that the opportunities somehow lie in assisting the public sector in opening their data. To be a supplier or consultant to government agencies, municipalities and other public entities as they open their data.
Sure, there will be projects. Sure, there is room for some new products to facilitate that. But I think this is really missing the point.
The real opportunity
The revolution here is that an enormous amount of valuable data is becoming easily and freely available all over the world. Some of this data has never been made public before and certainly not this accessible. In gathering this data lie thousands – no – probably hundreds of thousands of man-years, whose output has until now only been used to a fraction of its potential. Imagine the value in all that work, in all that data!
And this data has real, practical, monetary value to virtually every business, household and individual. But only a tiny fraction of them know that yet.
The real commercial opportunity in Open Data lies in helping people discover all this data, see its potential and realize how they can make use of it to run their businesses better, make better decisions and identify new opportunities.
The following was originally posted as a guest post on the Open Knowledge Foundation Blog.
The Wave of Open Data
Open access to public sector information, or simply “Open Data” is gathering a lot of momentum these days. High profile initiatives such as the Obama administration’s Data.gov; and UK’s Data.gov.uk; whose main advisor is none other than Sir Tim Berners-Lee are leading the way towards the more transparent and open governments of the future. Fantastic examples of data journalism by leading newspapers such as the New York Times and The Guardian show us how the power of turning data into information can be put into the hands of readers themselves. And nobody has better demonstrated the power of Open Data than statistics super hero Hans Rosling in his famous TED-talks.
The trend is obvious, and even though its still early days, there is no doubt in my mind that this trend will spread from these few prime examples to the rest of the world. I’m pretty confident that a decade from now, any self-respecting government and multi-national organization in the world will have established an Open Data policy. To me, the rules are quite straightforward: “Any government data should be open unless other, higher priorities – such as privacy or national security – indicate otherwise”. There are obviously gray areas, but governments must flip around the ruling mind-set of “keeping data closed unless there are good reasons to open it” and instead “make data open, unless there are good reasons to close it”. This will make governments more accountable, leave less room for corruption, heighten the quality of any debate and empower societies and their respective economies.
In this piece I will try to explain the status of open data in my home country of Iceland.
The first hints of a real Open Data mindset within the government probably dates back to the late 1990s, when Statistics Iceland – the local Statistics Office – concluded that their work was more valuable with open access to anybody, than in their then current model of selling access to their individual publications. This was made possible with the new delivery channel that was the internet, bringing down the cost of publication and distribution. This change was a success and today, many Icelanders, ranging from students to business leaders use data from Statistics Iceland to solve their everyday tasks and assignments.
Since, the topic has occasionally surfaced around limited topics of data, such as geospatial and mapping data, held by The National Land Survey of Iceland, fishing data held by the Marine Research Institute and weather data held by the Icelandic Met Office.
In 2006 the Icelandic “information act” was reevaluated and an updated legislation passed. In that process the reuse, redistribution and use of data by 3rd parties was openly discussed, but no major steps were taken.
In late 2007, an informal lobby group “Opin gögn” – which literally translates “Open Data” – was formed (full disclosure: I was a part of this initial group, and I may overestimate its importance in this story). Its purpose was to educate the Icelandic public sector and Icelanders in general about the Open Data movement, which was becoming quite visible in the “nerd” media at the time. In the spring of 2008, the group opened a Wiki-site – opingogn.net; (Icelandic only) with a reference to OKFN’s Open Knowledge Definition, and later a translation;. The site also listed various public and private sector entities that held important and interesting data, listed their databases / data sets and if and how that data could be accessed. An organization called Félag um stafrænt frelsi á Íslandi (meaning “The Foundation for Digital Freedom in Iceland”) or FSFÍ for short also started actively lobbying on similar notes. These ideas got some media attention and sporadic interest by different organizations and groups, but that interest can hardly be described as widespread.
The financial collapse
In October 2008 the Icelandic financial system imploded. A banking system that was more than 10 times larger than the nation’s GDP all but vanished in 3 days. The currency devalued roughly 50% to our most common trading currencies, inflation went into the double digits, strict currency restrictions were put in place, the stock market lost over 90% of its value and foreign debt increased dramatically. All in all – rather a mess!
On top of this, investigations of the crash have continued to reveal negligence by regulators, mistakes in governance, failure by government institutions to take action, incredibly unfortunate series of events, miscommunication and seemingly fraudulent and criminal actions by key players in the Icelandic finance sector.
Not surprisingly, this has led to a demand for more transparency, more access to public data and more effective communication by the government. All of a sudden Open Data is seen as a high priority among various lobby groups, branches of government and in restoration planning. Non-governmental think-tank Ministry of Ideas – founded last year by a group of local entrepreneurs – currently lists Open access to public sector data as its #1 priority. Government and opposition politicians have learned the basic reasoning for Open Data and regularly refer to it in speeches. In December a rare cross-party parliamentary proposal (the first step in passing new legislation) was made, proposing a “default open” strategy for any public sector data. The Prime Minister’s Office has formed a committee that is to propose changes and improvements in legislation and suggest how to define the boundaries between data that is to be open and data that shall remain closed.
On a related note, a new initiative – the Icelandic Modern Media Initiative (IMMI) – has emerged. Involving prominent figures from the Wikileaks-project; and members of aforementioned FSFÍ, this initiative proposes to change Iceland into a sort of “Journalistic free state”, where the legal environment would be perfect for investigative journalism, whistleblowers and protection of journalists’ sources, to name a few characteristics. Not surprisingly, this initiative has received a fair amount of media attention (BBC, NY Times, The Guardian, GigaOM, …) and these main ideas have also been passed as a parliamentary proposal. Needless to say it is likely to seriously boost any Open Data initiatives were it to pass as legislation.
But there are also Open Data sprouts out there that have nothing to do with the top-down actions of the government or parliament. More and more organizations and private sector companies are buying into the ideology of open data and have started their own efforts. A recent example includes cooperation between the leading yellow pages company – Já; – and The Árni Magnússon Institute for Icelandic Studies, Iceland University’s research center for the Icelandic language. Sponsored by Já, the institute has made openly available (not fully Open as per the Open Knowledge Definition, at least not yet) a valuable language database that is a crucial ingredient for various language technology software such as spell-checkers, automated translation software and even Icelandic language word-games. Já also sponsored a prize competition to encourage developers to make use of this data. The response has been great, and the results of the competition are to be announced this Tuesday.
Even the private sector has started opening data. Some financial institutions – not the public’s favorite at the moment – have started publishing far more detailed information about their actions in order to regain trust and show that they have nothing to hide. Examples include pension fund Frjálsi lífeyrissjóðurinn, operated by one of the fallen (and now restored) banks. The fund is now publishing a detailed account of their assets online available to anybody.
All in all, Iceland is not that far down the Open Data path – yet – but it seems like the extraordinary events of the financial collapse may push us fast towards the forefront of the Open Data tidal wave. There probably be dragons on the way, but we also have a lot to gain!
Foreigners used to know one thing about Iceland. Since last autumn, they tend to know two:
- Iceland is green and Greenland is ice
- Iceland is bankrupt
Neither is fully accurate. In fact:
- Iceland is greener than Greenland, but the dominant colors are black and gray of volcanic origin
- Iceland is not bankrupt, but experienced one of the most spectacular economic crashes any Western country has since the second World War.
A banking system that was more than 10 times larger than the nation’s GDP all but vanished in 3 days in early October. The currency has devalued roughly 50% to our most common trading currencies, inflation has been in the double digits for over a year, serious currency restrictions have been put in place, the stock market has lost over 90% of its value, housing prices are falling rapidly and foreign debt has increased dramatically.
I guess you could call it “rather a mess”.
On top of this, investigations of the crash continue to reveal negligence by regulators, mistakes in governance, failure by government institutions to take action, incredibly unfortunate series of events, miscommunication and seemingly fraudulent and criminal actions by key players in the Icelandic finance sector. And the investigations have barely begun!
As you can imagine, following such a catastrophe we have a serious trust issue.
The public hardly trusts the government (one has already fallen in public protest), let alone the semi-resurrected financial sector. In fact, most commercial activity is seen as highly suspicious and any sign of wealth is seen as a clear indication of wrong-doing.
This trust issue reaches abroad. Many Icelandic businesses are met with suspicion judged on origin alone and it is easy to get the feeling from recent developments that foreign governments don’t fully trust ours.
This level of distrust will destroy any rebuilding efforts. Restoring trust is the single most crucial thing for us to be able to move forward.
Sometimes people talk about “brutal honesty”. I believe that the way to restore trust lies in “brutal transparency”. Iceland, more than any other nation needs to reinvent the way the government operates and the way business is done.
The public sure is ready for it. Many of us feel ashamed and to an extent responsible for what has happened as we failed to keep a watchful eye out and voice our concerns.
Earlier this month our company – DataMarket – together with the leading local news portal mbl.is launched an interactive model (unfortunately Icelandic only) of a giant loan agreement the parliament is debating. Almost 25 thousand people dug in. A nation where 10% of the population is eager to explore and experiment with a financial model of an international loan agreement, is a nation that is clearly on its toes about current affairs and will not be betrayed again. People crave data and understanding.
We need to show that we have nothing to hide – neither to each other nor to the rest of the world. Luckily we are extremely well equipped to do so. I’m pretty sure this is one of the best-measured economies in the world. For another project, we’re monitoring over 7 million variables measured as time series and recorded by the various government institutions. Almost all transactions, even at the retail level are real-time (transactions between banks happen instantly) and electronic making it a virtually cash-free economy for the last decade or so. Businesses as well as the infamous banking sector are running state-of-the-art IT systems and could easily meet the demands of such an arrangement.
“Brutal transparency” is – in other words – a low hanging fruit for us. The hard bit is introducing this new way of thinking.
A few days after the crash I outlined on my personal blog a few ideas about how total transparency was the future of finance. I believe this now more than ever, but I also believe it should reach all branches of the government and business in general. For example:
- Publicly traded companies should not file quarterly reports, they should file every day.
- Company ownership should not be a secret, it should be publicly available information.
- Every bill that the government pays should be open data.
- Composition of managed funds should be available in real-time to any customer.
- Retailers should be required to publish their prices online in real-time allowing for meaningful price comparison.
- Any data gathered or recorded by the government should be open data, unless privacy or other higher-order priorities prevent it.
- Every single national budget item should be tracked in real time demanding explanations if they are significantly off-target.
- …this list could go on.
And efforts should be made to make all of this easily available and understandable to locals and foreigners alike.
The Icelandic Model
I have met with government officials and public sector management that truly get the concept of transparency and open data, believe in its power and want to make it happen. But further education is needed. The sprouts are there but the field is not blooming.
Businesses also need to understand that in the current climate transparency is a competitive edge making them more desirable to do business with, invest in and trust their services.
A serious effort in open data and transparency could not only restore the trust that has been ruined, but also make the “Icelandic method” a model for others to follow.