No joke: Open data fuels transparency, civic utility and economic activity

Open data venn diagrram

Credit: "Open government data," by Justin Grimes

Is the open data movement a ‘joke‘?

Canadian blogger Tom Slee published a post yesterday that made that claim, writing that “it’s not a movement, at least in any reasonable political or cultural sense of the word” and that “it’s doing nothing for transparency and accountability in government. Slee followed by a second post that highlighted some reactions to the first —  including my own, driven by a rather heated dialogue on Twitter with author Evgeny Morozov.

Slee makes one assertion that will be of particular interest to Govfresh readers, who may be surprised to find that civic hacking and a movement to put open government data online don’t exist:

Who, after all, is the Open Data Movement? Well it turns out there isn’t one really, at least when it comes to “open data” in the sense of “open government data”, which along with “open scientific data” is one of the two most common uses of the term.

“Open Data Movement” is a phrase dragged out by media-oriented personalities to cloak a private-sector initiative in the mantle of progressive politics. Along with other cyberculture terms (“hacktivism”, “unconferences”, “hackathons”) the word “movement” suggests a countercultural grass-roots initiative for social change, but there isn’t anything of the sort that I can see.

After a weekend at Transparency Camp in DC where an international open government (counter) culture was everywhere and a trip to Brasilia to see  representatives from civil society meet at the Open Government Partnership conference, along with years of reporting, I can honestly say that Slee simply must not be looking very hard. Sunlight Labs director Tom Lee rung in this afternoon with his assessment, defending open data’s role in accountability and activism:

But in a larger sense, there can be no question that open data is an important tool for creating accountability. Consider what FOIA means for corruption. Consider what OpenCorporates is doing. Consider our work on lobbying reform or tax expenditure transparency or the understanding of the interplay between the two that open data makes possible. Or look at Revenue Watch’s work to get better data on international royalty payments from mineral and petroleum companies. Or our push for better information about Congressional activity and political advertising.

Slee’s notion that open data is simply obfuscation for private sector initiatives was also more or less turned on its ear by TechPresident’s summary of Transparency Camp:

….an international group of transparency and open government activists got together for TransparencyCamp. Among the folks represented there: Open Data Albania, a collaborative that collects and analyzes data about the government of Albania and partners with journalists to build context to explain how that country works; Global Integrity, which partnered with others to build a 50-state corruption report card for the United States; and LittleSis, which looks to map connections and influence at the highest levels of American society.

Are there commercial interests that will use open data in their products or services? Absolutely. Look to consumer finance startups like Hello Wallet or BrightScope or or bigger players in healthcare, like Aetna, that stand to benefit from smart disclosure initiatives.

There’s little doubt that smart entrepreneurs, big and small, are going to mashup data from the rapidly expanding new sources — social data, geolocation data, mobile data, financial data, transit data, health data, etc — and build new businesses on it or improve their existing services, like Zillow or Google Maps or Consumer Reports or Bloomberg Government. In a time when job creation is critical, using public sector information to create jobs isn’t an aim to dismiss lightly, although the terms and conditions under which such activity occurs must be clear to all actors involved, to avoid the creation of new monopolies based upon artificial scarcity.

We are in such early days in this new century, and the role of civic entrepreneurs in putting data to work in the marketplace shouldn’t be discounted or dismissed out of hand, although the licensing and extraction of value of public data must be closely watched. (And yes, that includes the progress of civic startups coming out of Code for America’s civic accelerator or any funded by O’Reilly’s Alpha Tech ventures, like SeeClickFix. If you see the latter in my coverage, expect a disclosure and know I personally do not have an interest in the success of such startups and maintain a rigorous “church and state” relationship with that entity. My investments currently consist of a 401(k), and not one that’s weathered the economic downturn as well as I’d like.)

Here’s the Sunlight Foundation’s Tom Lee again, on the issue that Slee raised:

… the core of Tom’s complaint isn’t about episodic failures of activism. Rather, he seems to be bothered by open data enthusiasts’ adoption of language and an aesthetic that traditionally belong to projects with more expressly political (and progressive) aims. He seems suspicious that a self-described nonpartisan activist movement could be anything but a cynical lobbying ploy for private interests. Indeed, there’s a clear strain of hostility toward business that runs through Tom’s critique. Fair enough: more than a few such “movements” have turned out to be astroturfing operations, and it’s certainly true that on some open data issues I expect a less than enthusiastic response from the corporate world.

But I think it’s flatly wrong to consider private actors’ interest in public data to be uniformly problematic. We should be clear: we won’t tolerate those interests’ occasional attempts to lock public data into exclusive monopolies. I think our community has done a pretty good job lately of identifying such situations and stopping them, and of course people like Carl Malamud have been doing important work on this question since well before most of us ever heard of “open data.” But if commercial activity is enabled by data, that’s all to the good—the great thing about digital information is that scarcity doesn’t have to be a concern. Google Maps’ uses of Census TIGER data, for instance, is proprietary, motivated by profit, and unquestionably a huge boon to human welfare. And the source data remains free for anyone else to use! Cutting off those kinds of uses with noncommercial licensing would be nothing more than a destructive act of pique.

Open government godfather Carl Malamud offered a blunter assessment in a comment:

…In my world, the commercial sector is raping and pillaging the public treasury, getting exclusive deals on data that not only keeps out other companies, but researchers, public interest groups, and everybody else who make up “the public.” In many cases, the government data is so tightly behind a cash register that even government workers enforcing the law can’t afford to buy copies of the data they produce or the rules they promulgated.

I have no idea who Whimsley is and don’t usually bother to comment on random blogs by armchair quarterbacks, and I have no idea what is going on in Canada, but this one seems so far off the mark it seemed worth a few words. The post is backwards in the analysis, but it is also lacking a bit of reality.

I don’t give a hoot if something is a movement, but I’m not sure that making lists of who gets to use data and who doesn’t get to use public data makes any sense (many nonprofits are intensely commercial and many commercial operations seem to avoid the evilness of many of the beltway bandits). As far as Code for America’s program and their sponsors, or Tim O’Reilly and his talks, I’ve observed all of those at first hand and it is pretty clear the pseudonymous blogger doesn’t have a clue what either group does or what they think.


P.S. I’ve watched many tens of millions of people access and use government data that wasn’t available before from my servers. Maybe not a movement, but definitely a really big crowd.

Canadian open government data advocate and analyst David Eaves responded to Slee today on his own blog, where he argues (at considerable length) that open data movement is not a joke. The whole post is worth reading but I’ll quite Eaves on one point in particular:

…to be clear, I would never equate open government data as being tantamount to solving the problems of a restrictive or closed government (and have argued as much here). Just as an authoritarian regime can run on open-source software, so too might it engage in open data. Open data is not the solution for Open Government (I don’t believe there is a single solution, or that Open Government is an achievable state of being – just a goal to pursue consistently), and I don’t believe anyone has made the case that it is. I know I haven’t. But I do believe open data can help. Like many others, I believe access to government information can lead to better informed public policy debates and hopefully some improved services for citizens (such as access to transit information). I’m not deluded into thinking that open data is going to provide a steady stream of obvious “gotcha moments” where government malfeasance is discovered, but I am hopeful that government data can arm citizens with information that the government is using to inform its decisions so that they can better challenge, and ultimately help hold accountable, said government.

Expect more reactions to emerge. For my part, I’d reiterate what I’ve written about open government data before: Simply opening up data is not a replacement for a Constitution that enforces a rule of law, free and fair elections, an effective judiciary, decent schools, basic regulatory bodies or civil society, particularly if the data does not relate to meaningful aspects of society.

I’ll go a bit further, extending a comment I left on Slee’s second post. Morozov’s points about raising critical questions are legitimate and my comments on Twitter could easily be read as being dismissive of their pertinence. That was not my goal and to do so would be antithetical to my belief in free speech and robust public discourse about the serious issues that confront us as a country and humans on this small blue dot.

I did think — and still do — that the latter half of Slee’s post needed to include some basic research that would have improved its discussion of the issues involved, particularly with respect to the tension between open data and open government. The comments he received on that post particularly those of Carl Malamud, as quoted above, and the posts that I’ve linked, should serve as a bellwhether.

Readers curious about the tensions here (perhaps including Slee) would benefit from reading a recent research paper by Yu and Robinson on the ambiguity of open government and open data and from delving into the broader discussion in civil society in South America and Europe in the lead up to the Open Government Partnership, where this dynamic was the subject of much concern — and not just in a Canadian or UK context.

One reason that Slee’s post may have received attention — and continues to do so — is that it does not read as “The problem with open data versus open government in Canada.” Instead, it’s an indictment of what’s happening in the U.S. or around the world. I suspect that if he had stopped after the first half of what he’d written about Canada’s open government record, most people from civil society and NGOs would have nodded along. One reason those legitimate concerns he raised may not be receiving the attention that Slee might like is that they were coupled them with a headline and analysis that distract from them. In his followup, he walks that back a bit, but not much.

As the conversation in the public sphere over the past few years has demonstrated, there are a lot of different perspectives on what purposes “open data” should serve, often informed by what the watcher intends or the organization’s goals. That’s to be expected in a new and rapidly evolving space.

There are people who want to see legislatures open their data, to provide more insight into deliberative processes. In the U.S., for instance, has been making government legislative data open (and more useful!) by scraping it from government websites.

There are constituencies who wish to see campaign finance data open, like the Sunlight Foundation, thereby showing where influence and power lies in the political system.

There are entrepreneurs and civic activists who wish to see transit data or health data become more open, in the service of more civic utility or patient empowerment — one can look at the efforts of various cities or the Veterans Administration in the US on that count. When you consider that such data can include ratings or malpractice information about hospitals or doctors, or fees for insurance companies, transparency and accountability does crop up as a goal, which in turn does have political implications.

One could spend quite a bit of time listing organizations or individuals who are putting data online, including open government activists in Brazil, Africa or yes, Canada, and then listing consumers of that data.

Whether Slee wish to describe those activities as a “movement” is up to him — but it is indisputable that 3 years ago, a researchers would be hard-pressed to find a open government data platform nor downstream consumers.

Now there are dozens of such platforms at the national, state and city level and even more services that use that data. What matters more than their existence, however, is what goes onto them. In that respect, civil society and media must to be extremely careful about giving governments credit for just putting a “portal” online. In March, Sunlight Foundation policy director John Wonderlich wrote a post about the “missing open data policy” that every government standing up a platform could benefit from reading:

Most newly implemented open data policies, much like the Open Government Directive, are announced along alongside a package of newly released datasets, and often new data portals, like In a sense, these pieces have become the standard parts of the government data transparency structure.  There’s a policy that says data should generally be open and usefully released, a central site for accessing it, some set of new data, and perhaps a few apps that demonstrate the data’s value.

Unfortunately, this is not the anatomy of an open government.  Instead, this is the anatomy of the popular open government data initiatives that are currently in favor. Governments have learned to say that data will be open, provide a place to find it, release some selected datasets, and point to its reuse.

What gets left out of these initiatives, however, is often the most important part — the decisions as to what gets released, and how.  Many open government data discussions skip over the question of whether governments are deciding appropriately what gets released and what doesn’t.  Instead of making complex decisions about what should be released, central governments suggest that those decisions are hard, and that as long as there’s always some new information, then we’re making progress that deserves praise.

There are also, notably, many civil society and media organizations that are collecting and sharing open data, from OpenCorporates to OpenCongress to ProPublica, and startups as well, like Brightscope.

There are a lot of different voices in this space. Asking hard questions is important and useful, particularly given that motivations and context will differ from country to country and from industry to civil society. In the United States, there is a sizable group of people that believe that data created using public funds should in turn be made available to the public — and that the Internet is a highly effective place to make such data available. (See the Sunlight Foundation’s support for POIA, or “Public = Online.”) Such thinking extends to research or code now too. Support for open data also extends across the political spectrum here, as evidenced by the recent passage of the DATA Act with bipartisan support.

Whether one agrees with that perspective or not is, of course, something that free thinkers in democratic societies to decide for themselves, including my neighbors in Canada. Given the pervasive tendency towards more secrecy in governments, not less, and my experience in open government over the past few years, my tendency is towards making public information open by public by default as opposed to its inverse. If the trend towards over-classification is not reversed, the nation will be less informed.

Finally, since Slee brought the source of his frustration up in his second post, I want to be clear: the issues he cited with respect to Canada’s open government record are not founded in speculation, as his links and points demonstrate. The Harper administration has received the dubious distinction of a secrecy award from a journalism association. It did cancel Canada’s long-form census, prompting the resignation of the head of the Statscan service. And journalists have been confronted with limited access to government scientists, much in the same vein of open government issues in the United States.

Highlighting the difference between rhetoric and actions is a crucial role for civil society and independent media in any open government context. To the extent Slee has done so with these posts, I applaud his actions. To the extent he questions my motivations or those of my publisher and friend, Tim O’Reilly, I reject his conclusions.

I agreed to collaborate with Tim on focusing attention on open government and technology because he wanted to see government work better, become smarter, and be more accountable to the people whom it serves. I did so with the understanding that I’d be able to pursue storytelling with editorial independence and adherence to truth, fairness, accuracy and reason. If I ever fail in that goal, I expect the open government community to continue to hold me as accountable as they would an elected official or public servant.

Should Canada’s open government advisory panel be convened again, I will have specific issues to raise concerns about with the other members and commit to doing so.

Gov 2.0 goes mainstream with a new Associated Press article on open government data

We live in interesting times. Last week, NPR listeners learned about “local Gov 2.0.” This weekend, civic applications and open data emerged further into the national consciousness with a widely syndicated new Associated Press story by Marcus Wohlsen, who reported that a “flood of government data fuels rise of city apps. Here’s how Wohlsen describes what’s happening:

Across the country, geeks are using mountains of data that city officials are dumping on the Web to create everything from smartphone tree identifiers and street sweeper alarms to neighborhood crime notifiers and apps that sound the alarm when customers enter a restaurant that got low marks on a recent inspection. The emergence of city apps comes as a result of the rise of the open data movement in U.S. cities, or what advocates like to call Government 2.0.”

The AP covered Gov 2.0 and the open government data movement in February, when they looked at how cities were crowdsourcing ideas from citizens, or “citizensourcing.”

It’s great to see what’s happening around the country get more mainstream attention. More awareness of what’s possible and available could lead to more use of the applications and thereby interest and demand for civic data. For instance, on the @AP’s Twitter feed, an editor asked more than 634 “Hundreds of new apps use public data from cities to improve services. Have you tried any?”

Wohlson captures the paradigm behind Gov 2.0 well at the end of his article:

“New York, San Francisco and other cities are now working together to develop data standards that will make it possible for apps to interact with data from any city. The idea, advocates of open data say, is to transform government from a centralized provider of services into a platform on which citizens can build their own tools to make government work better.

Open311 and GTFS are data standards of this sort. What lies ahead for Gov 2.0 in 2012 has the potential to improve civic life in any number of interesting ways. I look forward to sharing that journey.

Carmi Levy: open government is about leveraging technology and citizens to do more with less

It looks like the dog days of August 2011 may be the month when the meme of a citizen-centric government gets some traction in the business world. Over on the finance section of Yahoo Canada, Carmi Levy writes that the future of government is citizen-focused.

Levy cites open government in British Columbia and initiatives in New Zealand, Toronto, and Lousiana as case studies for his thinking and then connects the dots with the big idea: that technology enables officials to empower citizens to work with government in new ways, driven by macrotrends towards open data, mobile connectivity, social media and austerity measures.

…governments are increasingly giving citizens free rein to do as they wish with previously inaccessible data. Costs are significantly reduced as big, conventional IT projects are replaced by more on-the-fly approaches to resource management. Timelines are also cut down to size thanks to the use of agile development methods and more collaborative models. Crowdsourcing also maximizes the use of newer technologies, thanks to home-based developers looking to market their prowess to a broader audience. This all translates into more bang for the public buck.

Proponents of open data initiatives claim they increase government efficiency and effectiveness by encouraging greater levels of citizen participation in the creation and delivery of public services. But in light of the just-completed U.S. deal to restructure its debt ceiling and begin trimming the federal budget, it’s difficult to ignore the cost side of the equation, as well.

As governments on both sides of the border find themselves increasingly pressured to deliver the same — or more — services for less, open data and so-called Government 2.0-based initiatives could hold the key to taxpayers having their cake and eating it, too. As government shrinks, citizens willingly take up the slack using rapidly evolving development and social media tools.

Open government isn’t just a philosophical concept designed to drive democracy.

It’s really about leveraging technology — and technologically enabled citizens — to do more with less. By throwing data out there and seeing what develops, governments can reduce spend and enable business in ways they simply wouldn’t be able to do if they functioned conventionally. They can leverage the motivations and skills of interested members of the public to create value that conventionally hired departmental resources have never been able to achieve; at least not at this level of efficiency.

Carmi Levy (@CarmiLevy)

It’s a bold vision, although perhaps a familiar one to those who have been following the narrative that runs through these open government stories. The notable connection is connecting this approach to the need that governments have now.

If open government is going to work better, however, citizens will have to become more civically engaged — and their governments will need to both listen to them and work with them.

Open Government MAGIC: Media Access to Government Information Conference

The right of the governed to gain access to information about their government is a core pillar of the compact between “We the People” in the United States and those they elect to office. The quality, breadth and depth of that access, however, is often troubled.

Today in Washington, the Media Access to Government Information Conference (MAGIC) will explore these issues from within the august halls of the National Archives. MAGIC is a collaborative, one-day conference sponsored by the National Archives and Records Administration (NARA) and Duke University’s DeWitt Wallace Center for Media and Democracy. The primary focus of the conference is to highlight how journalists and others writing about public affairs can gain better access to government records by journalists. A liveblog of the proceedings, agenda and associated papers are embedded below:

Program and Papers

9:00-9:20 Welcome by David S. Ferriero, Archivist of the United States, NARA; Sanford Ungar, President Goucher College, and Member, Public Interest Declassification Board

9:20-10:30 Session 1: Media Access to Federal Government Records

Journalists and NGO participants on this panel will address how FOIA and access to federal records might be re-tooled as the federal government implements its open government and transparency policies. Government panelists will describe their vision for how new policies and technologies are changing access to government records. Additional topics may include:

  • Institutionalizing the release of common records used to monitor agency activity rather than waiting for FOIA requests to come in;
  • Centralizing, updating, and documenting information systems on agency FOIA websites; and
  • Building openness into administrative (records collecting) systems that are eventually released to the public.

Moderator: Irene Wu, Director of Research, SAND-MNIA International Bureau, FCC

  • Gary Bass, Founder and Executive Director, OMB Watch;
  • Sarah Cohen, Knight Professor of the Practice of Journalism and Public Policy, Duke University;

  • William Kammer, Chief, FOIA Division, U.S. Department of Defense, and Vice President, American Society of Access Professionals;
  • Miriam Nisbet, Director, Office of Government Information Services (OGIS), NARA

10:30-10:45 Morning Break

10:45-Noon Session 2: Technical Hurdles, Research Solutions

Journalists on the panel will identify specific technical problems in dealing with government records at federal, state, local, and tribal levels. Government officials will identify specific technical solutions or research agendas to find solutions to these problems. Additional topics may include:

  • Re-tooling internal government information systems to improve the quality of records release;
  • Government agency support of research to improve the mining and analyzing of documents not born digital, handwritten responses on forms, and audio/video of government proceedings; and
  • Insights into emerging technologies and cyber infrastructure that may facilitate media access to government records.

Moderator: Robert Chadduck, Acting Director, National Archives Center for Advanced Systems and Technologies (NCAST), NARA

  • David Donald, Data Editor, Center for Public Integrity
  • Richard Marciano, Professor and Director @ Sustainable Archives and Leveraging Technologies group, UNC School of Information and Library Science
  • George Strawn, Director, National Coordination Office, Networking and Information Technology Research and Development (NITRD) Program
  • Ken Thibodeau, Former Director (Retired), National Archives Center for Advanced Systems and Technologies (NCAST)
  • Derek Willis, Web developer, New York Times

Noon-1:30 Luncheon

1:30-2:45 Session 3: Access to State, Local, and Tribal Government Records

Journalists on this panel will identify issues that arise frequently in seeking records at state, local, and tribal levels. Government panelists will discuss possible solutions to making these records more easily available, and how different levels of government may leverage IT to improve access to records. Additional topics may include:

  • Types of records sought at state, local, and tribal level;
  • Special challenges in variations in open access policies across states and localities; and
  • Federal funds expenditure rules that might trigger more transparency at state and local level.

Moderator: David McMillen, NARA External Affairs Liaison

2:45-3:15 Afternoon Break

3:15-4:30 Session 4: Private Sector Actions

NGO participants will discuss how they work to improve access to records, including participation in discussions to retool government records systems for better access by journalists. Additional topics may include:

  • What transparency advocates, journalism organizations, foundations, and academics could do to support access policies; and
  • The development of tools to aid in the analysis of government records.

Moderator: James Hamilton, Director, DeWitt Wallace Center for Media and Democracy, Duke University

  • Bill Allison, Editorial Director, Sunlight Foundation
  • Rick Blum, Coordinator, The Sunshine in Government Initiative
  • Danielle Brian, Executive Director and Project on Government Oversight
    Bryan Rahija, Blog Editor, Project on Government Oversight
  • Charles Lewis, Executive Editor, Investigative Reporting Workshop and Professor, School of Communication, American University

Dan Rather interviews Tim O’Reilly on Gov 2.0

How did Tim O’Reilly describe the potential of Gov 2.0 to Dan Rather?

“Government 2.0 is the attempt to harness the latest technology to make our government more effective, transparent and participatory,” he said.

What do the alpha geeks want to do, with respect to improving government? “They started saying, first of all, we want to open up government, we want more access to all this government data. We want to create new capabilities for citizen involvement.”

From HDNet YouTube Channel excerpt’s description:

Barack Obama’s presidential campaign was recognized for its innovative use of technology to rally voter support. But there are people out there who say that technology has the power to not only revolutionize campaigns, but the very way we view government. It’s a movement called Government 2.0 and Tim O’Reilly is one of its most vocal prophets. He has been called “the Oracle of Silicon Valley” because he saw the potential of the world wide web years before most of us had even heard of it. He’s a highly respected big thinker in the tech community. And he believes that we are at the vanguard of a radical re-think of how government works in the Internet age.

The full episode of Dan Rather Reports on “Tim O’Reilly and Government 2.0” is available as a direct download from iTunes.

Australian Senator Kate Lundy interview at the Gov 2.0 Expo

Senator Kate Lundy has represented the Australian Capital Territory in the Senate in the Australian Federal Parliament as a member of the Australian Labor Party since 1996. I wrote more about the Australian approach to open government in “Government 2.0 Down Under” for O’Reilly Radar.