McKinsey estimates open data could add more than $3 trillion in economic value

According to a new McKinsey report, “research suggests that seven sectors alone could generate more than $3 trillion a year in additional value as a result of open data, which is already giving rise to hundreds of entrepreneurial businesses and helping established companies to segment markets, define new products and services, and improve the efficiency and effectiveness of operations.”

20131028-202720.jpg

That’s a mighty big carrot to open up public sector data to the public, keeping in mind potential privacy, political and national security consequences of doing so carelessly.

A growing number of open data entrepreneurs are mining these embryonic markets, finding fertile soil using different business models that focus on adding value to a national resource: data collected for the people, released back to the people.

In depth: news and analysis about the troubles behind Healthcare.gov

npr-healthcare.gov

Over the past 27 days, as I’ve steadily shared analysis and links on what went wrong in the botched re(launch) of Healthcare.gov on TwitterFacebook and Google+, I’ve also been talking in more depth about what went wrong on various media outlets, including:

Last week, the Obama administration announced a plan to fix the issues with the software behind HealthCare.gov, including putting QSSI in charge of as the “general contractor” and prioritizing fixing errors in 834 file data first, with the goal of have the system functioning end-to-end by November 30.

The teams of Presidential Innovation Fellows and “A List” contractors in the “tech surge” to fix the software have a tough challenge ahead of them. According to reporting from The New York Times and The Washington Post,  Healthcare.gov wasn’t tested as a complete system until the last week of September, when it crashed with only a few hundred users.

Despite the issues revealed by this limited testing, government officials signed off on it launching anyway, and thus was born a historic government IT debacle whose epic proportions may still expand further.

Should the White House have delayed?

“When faced with go live pressures, I tell my staff the following:

‘If you go live months late when you’re ready, no one will ever remember. If you go live on time, when you’re not ready, no one will ever forget.”-Dr. John Halamka, CIO Beth Israel Deaconness Hospital

In retrospect, the administration might have been better served by not launching on October 1st, something that was within HHS Secretary Kathleen Sebelius’ legal purview. After all, would the federal government launch a battleship that had a broken engine, faulty wiring or non-functional weapons systems into an ongoing fight? This software wasn’t simply “buggy” at launch — It was broken. These weren’t “glitches” caused by traffic, although the surge of traffic did expose where the system didn’t work quickly. Now that a reported 90% of users are able to register, other issues on the backend that are just beginning to become clear, from subsidy calculation to enrollment data to insurers reporting issues with what they are receiving to serious concerns about system security.

Based upon what we know about troubles at Healthcare.gov, it appears that people from the industry that were brought in to test the Healthcare.gov system a month ago urged CMS not to go live. It also appears that inside the agency who saw what was going on warned leadership months in advance that the system hadn’t been tested end-to-end. Anyone building enterprise software should have the code locked down in the final month and stopped introducing new features 3-6 months prior. Instead, it appears new requirements kept coming in and development continued to the end. The result is online now. (Or offline, as the case may be.)

On September 30, President Obama could have gone before the American people and said that the software was clearly not ready, explained why and told Americans that his administration wouldn’t push it live until they knew the system would work. HHS could have published a downloadable PDF of an application that could be mailed in and a phone number on the front page, added more capacity to the call centers and paper processing. It’s notable that three weeks later, that’s pretty much what President Obama said they have done.

The failed launch isn’t just about “optics,” politics or the policy of the Affordable Care Act itself, which is a far greater shift in how people in the United States browse, buy, compare and consume health insurance and services. A working system represents the faith and trust of the American people in the ability of government. This is something Jennifer Pahlka has said that resonates: how government builds websites and software matters, given the expectations that people now have for technology. The administration has handed the opponents of the law an enormous club to bash them with now — and they’ll deserve every bit of hard criticism they get, given this failure to execute on a signature governance initiative.

Articles worth reading on Healthcare.gov and potential reforms

  • Whenever I see Fred Trotter, I’m reminded that he’s forgotten more about open source software and healthcare IT than I’m ever likely to learn. Last week, he talked with Ezra Klein about the issues with Healthcare.gov
  • Ezra Klein also talked  to Clay Johnson about the lessons of Healthcare.gov (hint: procurement, project management and insourcing)
  • Tough reporting on failures in e-government is critical to improving those services for all, but particularly for the poor.
  • A post by Development Seed founder Eric Gunderson on the open source front-end for Healthcare.gov: “It’s called Jekyll, and it works.”
  • Rusty Foster on Healthcare.gov: it could have been worse. This failure to (re)launch just happened under vastly more political scrutiny and deadlines set by Congress. The FBI’s Sentinel program, by contrast, had massive issues — but you didn’t see the Speaker of the House tweeting out bug reports or cable news pundits opining about issues. The same is true of many other huge software projects.
  • A must-read op-ed by former Obama campaign CTO Harper Reed and Blue State Digital co-founder and former Presidential Innovation Fellow Clay Johnson on what ails government IT, adding much-needed context to what ailed Healthcare.gov
  • A Mother Jones interview that asked whether Reed and other former campaign staff could fix Healthcare.gov. (Spoiler: No.)
  • If not those folks, then how should the administration fix Healthcare.gov? In the larger sense, either the federal government will reform how it buys, builds and maintains software, through a combination of reforming procurement with modular contracting, bringing more technologists into government, and adopting open source and agile development processes …or this will just keep happening. The problems go much deeper that a “website.”
  • Ezra Klein pulled all of these pieces together in a feature on the “broken promise of better government through technology” at the end of the month. (He may have been heard in the Oval Office, given that the president has said he reads him.) Speaking at an “Organizing for America” event on November 4th, President Obama acknowledged the problem. “…I, personally, have been frustrated with the problems around the website on health care,” he said, “And it’s inexcusable, and there are a whole range of things that we’re going to need to do once we get this fixed – to talk about federal procurement when it comes to IT and how that’s organized…”
  • The issues behind Healthcare.gov cannot only be ascribed to procurement or human resources, as Amy Goldstein and Juliet Eilperin reported in the Washington Post: insularity and political sensitivity were a central factor behind the launch..

    Based on interviews with more than two dozen current and former administration officials and outsiders who worked alongside them, the project was hampered by the White House’s political sensitivity to Republican hatred of the law — sensitivity so intense that the president’s aides ordered that some work be slowed down or remain secret for fear of feeding the opposition. Inside the Department of Health and Human Services’ Centers for Medicare and Medicaid, the main agency responsible for the exchanges, there was no single administrator whose full-time job was to manage the project. Republicans also made clear they would block funding, while some outside IT companies that were hired to build the Web site, HealthCare.gov, performed poorly.

  • What could be done next? Congress might look across the Atlantic Ocean for an example. After one massive IT failure too many, at the National Health Service the United Kingdom created and empowered a Government Digital Services team. UK Executive Director of Digital Mike Bracken urged U.S. to adopt a digital core.”
  • In the video below, Clay Johnson goes deep on what went wrong with Healthcare.gov and suggests ways to fix it.

  • Can the White House and Congress take on the powerful entrenched providers in Washington & do the same? I’m not optimistic, unfortunately, given the campaign contributions and lobbying prowess of those entities, but it’s not an impossible prospect. I’ll write more about it in the future.

Coalition presses White House to make secret surveillance laws open to the public

A coalition of organizations that support open government, press freedom and civil liberties have sent a letter to President Barack Obama urging him to make the laws that govern surveillance by the National Security Agency public. The letter, which I’ve published in full below, asks the constitutional law professor living at 1600 Pennsylvania Avenue to support a core principle of democratic governance that hails back (at least as far as) the 12 Tables posted in the Roman Forum: the people should be able to read the laws under which they are governed. The letter was sent to the White House on the eve of the second annual conference of the Open Government Partnership.

October 21, 2013
President Barack Obama
The White House
1600 Pennsylvania Avenue, N.W. Washington, DC 20500
Dear President Obama:
On behalf of citizens who support an open and accountable government, we are writing to urge you to pledge as part of the US’s new round of Open Government Partnership commitments to curb the secret law that enabled the National Security Agency’s surveillance programs to become much broader and more invasive than it was believed the law allowed.
Secret legal interpretations by the Justice Department’s Office of Legal Counsel (OLC) and the Foreign Intelligence Surveillance Court allowed the NSA’s surveillance programs to grow in ways that raise serious concerns about what the government is doing in our name and the extent of violations of American’s privacy and civil liberties. Documents released to the media about the NSA’s programs further raise critical questions about the scope of the US’s activities abroad, leading the President of Brazil and others to question whether the US’s programs breach international law.
This is not the first time that abuses of power have occurred when a government program operates in a bubble of secrecy with only limited oversight: similarly, Americans were outraged to learn that memos authored by the OLC during the Bush Administration approved interrogation methods that many equate to torture. Your release of these memos demonstrated a respect for the public’s right to know how the government interprets the law. Making a concrete commitment to the public’s right to legal interpretations on issues including the intelligence community’s surveillance programs and other controversial policies like targeted killing through the use of drones or other means would make this respect part of the administration’s legacy. While the government has an obligation to protect properly and appropriately classified information, democracy does not thrive when our national security programs and the intelligence community’s actions are shrouded in secrecy. The public must, at the very least, have a shared understanding of the bounds and limits of the laws of our land and be able to have an informed debate about our policies.
During the meeting of the Open Government Partnership in London, you have a unique opportunity to address this issue head-on on an international stage. By committing to give the public access to documents that significantly interpret laws, including – but not limited to—the Department of Justice’s legal interpretations and opinions by the Foreign Intelligence Surveillance Court (FISC), you can both address domestic concerns about our surveillance programs, and begin to rebuild trust with our international partners.
Thank you in advance for your attention to this issue of critical importance to transparent and accountable government. To discuss these issues in greater detail, please contact Patrice McDermott, Executive Director of OpenTheGovernment.org, at pmcdermott@openthegovernment.org or 202-332- 6736.
Sincerely,
American Booksellers Foundation for Free Expression American Civil Liberties Union
American Library Association
American Society of News Editors
Arab American Institute
ARTICLE 19
Bill of Rights Defense Committee
Brechner Center for Freedom of Information
Californians Aware
Center for Democracy and Technology
Center for Effective Government
Center for Media and Democracy
Citizens for Responsibility and Ethics in Washington – CREW The Constitution Project
Council on American-Islamic Relations – CAIR
Electronic Frontier Foundation
Electronic Privacy Information Center – EPIC
Essential Information
Federation of American Scientists
First Amendment Foundation
Government Accountability Project – GAP
Human Right Watch

iSolon.org
James Madison Project
Just Foreign Policy
Liberty Coalition
National Coalition Against Censorship
National Freedom of Information Coalition National Security Archive
No More Guantanamos
OpenTheGovernment.org
PolitiHacks
Project On Government Oversight – POGO
Public Citizen
Public Knowledge
Reporters Committee for Freedom of the Press Reporters Without Borders
Society of Professional Journalists
Sunlight Foundation
Tully Center for Free Speech at Syracuse University Understanding Government
Vermont Coalition for Open Government
Vermont Press Association
Washington Civil Rights Council
Win Without War

2013 Code for America Summit convenes civic innovators in San Francisco [LIVEBLOG]

cfa-2013-theater
Hello from San Francisco! I’ll be liveblogging from the 2013 Code for America Summit for the next 2 days. You can tune in here.

Readers should also be aware of an important disclosure: Code for America paid for the costs of my travel to and from San Francisco and has further engaged me to produce paid analysis of the themes extant at the event. Part of our agreement, however, was that the organization would have no editorial control or discretion with respect to what I write about the event, organization, partners or constituents.

Report by Committee to Protect Journalists hammers Obama administration record on transparency

Leading the day in the world of open government is a mammoth report from the Committee to Protect Journalists on the Obama administration and the press, by Leonard Downie Jr., with reporting by Sara Rafsky.

Much of this won’t be new to those who have been tracking secrecy, over-classification, prosecution of whistleblowers and selective disclosure of favorable information using new media and leaks — all core open government issues — but this pulls together those issues into a coherent whole. Abstract:

“U.S. President Barack Obama came into office pledging open government, but he has fallen short of his promise. Journalists and transparency advocates say the White House curbs routine disclosure of information and deploys its own media to evade scrutiny by the press. Aggressive prosecution of leakers of classified information and broad electronic surveillance programs deter government sources from speaking to journalists.”

While I find prosecution of whistleblowers, insider threats and the aggressive surveillance of journalists investigating national security and the surveillance state (meta!) to be particularly problematic, there are also significant issues around FOIA compliance and access to officials.

“The administration’s war on leaks and other efforts to control information are the most aggressive I’ve seen since the Nixon administration,” writes Downie

As press freedom goes, so to does open government and democracy. I’ll be making this point strongly in London in a few weeks.

Following up on the botched (re)launch of Healthcare.gov

Over the weekend, federal officials finally confirmed that coding errors and other tech issues existed with Healthcare.gov, after days of saying high traffic was at issue. As people who have attempted to use the federal site know, however, problems with account creation and subsidy calculations system prevented the vast majority of users from enrolling in the first week. The Wall Street Journal places the tech issues with initial authentication and the system built to calculate insurance subsidies, both built by CGI Global, along with assorted other bugs.

error-message-h-800x624 (1)

In a statement, a spokesperson from the U.S. Department of Health and Human Services claimed that things were getting better:

“The work done to increase access to HealthCare.gov in light of the overwhelming demand is beginning to show results. Call center wait times are seconds, not minutes, and people have been enrolling over the phone 24/7. Our work to expand the site’s capacity has led to more people successfully applying for and enrolling in affordable health coverage online, with wait times being shortened by approximately 50 percent since Friday.”

Separately, a spokesperson for the Center for Medicaid and Medicare Services (CMS) said that several fixes for IT issues went in while the site was down, adding dedicated servers for the registration and authentication components that were preventing the vast majority of users from creating accounts. Unfortunately, it’s not clear whether that fix alone will address other issues people are having.

Specifically, “engineers at Web-hosting company Media Temple Inc. found a glut of stray software code that served no purpose they could identify. They also said basic Web-efficiency techniques weren’t used, such as saving parts of the website that change infrequently so they can be loaded more quickly.”

To put it another way, fixing the account creation and authentication steps could well expose the next set of tech issues with the marketplace on Healthcare.gov, from calculating subsidies to enabling digital enrollment.

What’s behind the mess? Earlier today I made my first appearance on Washington Post TV, breaking down what went wrong with the launch of Healthcare.gov, from why it might have happened to what’s being done to fix it to the really big deadline for it to work, in mid-December.

More to come.

Federal officials are urging patience while engineers make more improvements to the system. To date, only a few thousand people are estimated to have made it through the tech issues to enroll, out of a reported 9 million visitors to date.

Digging in open data dirt, Climate Corporation finds nearly $1 billion in acquisition

“Like the weather information, the data on soils was free for the taking. The hard and expensive part is turning the data into a product.”-Quentin Hardy, in 2011, in a blog post about “big data in the dirt.”

soil-type-graphic_wide-7a4e4709ff8554fc2abafaa342589fccf0524189-s6-c30

The Climate Corporation, acquired by Monsanto for $930 million dollars on Wednesday, was founded using 30 years of government data from the National Weather Service, 60 years of crop yield data and 14 terabytes of information on soil types for every two square miles for the United States from the Department of Agriculture, per David Friedberg, chief executive of the Climate Corporation.

Howsabout that for government “data as infrastructure” and platform for businesses?

As it happens, not everyone is thrilled to hear about that angle or the acquirer. At VentureBeat, Rebecca Grant takes the opportunity to knock “the world’s most evil corporation for the effects of Monsanto’s genetically modified crops, and, writing for Salon, Andrew Leonard takes the position that the Climate Corporation’s use of government data constitutes a huge “taxpayer ripoff.”

Most observers, however, are more bullish. Hamish MacKenzie hails the acquisition as confirmation that “software is eating the world,” signifying an inflection point in data analysis transforming industries far away from Silicon Valley. Liz Gannes also highlighted the application of data-driven analysis to an offline industry. Ashlee Vance focused on the value of Climate Corporation’s approach to scoring risk for insurance in agribusiness. Stacey Higginbotham posits that the acquisition could be a boon to startups that specialize in creating data on soil and climate through sensors.

[Image Credit: Climate Corporation, via NPR]

City of Los Angeles launches open data pilot and survey

Upon election, I wondered whether Mayor-Elect Eric Garcetti would reboot Los Angeles city government for the 21st century. After 4 months, there are several promising green shoots to report.

First, Mayor Garcetti vowed to modernize city government in the City of Angels, posting agency performance statistics online and reviewing all departments. Now, the City of Los Angeles has launched its own open data pilot project using ESRI’s ArcGIS platform.

laskyline_logo_final_7_1

For veterans of such efforts, a portal to mapping data may not be particularly exciting or useful, but it’s a start. Notably, the city has put up an online survey where people can request other kind of city data and suggest changes or improvements to the pilot website.

Here’s a few suggestions:

1) Study how the cities of Chicago and New York cleaned, published and used data, including market demand.

2) Talk to the data desk at the Los Angeles Times. If you want your city’s performance data to be used by media for accountability and transparency, address their concerns.

3) Make a list of every single request for data made by journalists in Los Angeles under the California Records ActRelease the data and proactively publish that type of data going forward.

4) If your goal is economic outcomes from open data, review all requests for city data from businesses and prioritize release of those data sets. Engage startups and venture capitalists who are consuming open data and ask about quality, formats and frequency of release.

5) Check out New York City’s gorgeous new open data site and the ongoing release of more data sets. Set those free, too.

6) Check out the new NYC.gov, Utah.gov and gov.uk in the United Kingdom for ideas, principles and models. Of note: the use of open standards, citizen-centricity, adaptive Web design, powerful search, focus on modern design aesthetic.

Good luck, LA!