In depth: news and analysis about the troubles behind

Over the past 27 days, as I’ve steadily shared analysis and links on what went wrong in the botched re(launch) of on TwitterFacebook and Google+, I’ve also been talking in more depth about what went wrong on various media outlets, including:

Last week, the Obama administration announced a plan to fix the issues with the software behind, including putting QSSI in charge of as the “general contractor” and prioritizing fixing errors in 834 file data first, with the goal of have the system functioning end-to-end by November 30.

The teams of Presidential Innovation Fellows and “A List” contractors in the “tech surge” to fix the software have a tough challenge ahead of them. According to reporting from The New York Times and The Washington Post, wasn’t tested as a complete system until the last week of September, when it crashed with only a few hundred users.

Despite the issues revealed by this limited testing, government officials signed off on it launching anyway, and thus was born a historic government IT debacle whose epic proportions may still expand further.

Should the White House have delayed?

“When faced with go live pressures, I tell my staff the following:

‘If you go live months late when you’re ready, no one will ever remember. If you go live on time, when you’re not ready, no one will ever forget.”-Dr. John Halamka, CIO Beth Israel Deaconness Hospital

In retrospect, the administration might have been better served by not launching on October 1st, something that was within HHS Secretary Kathleen Sebelius’ legal purview. After all, would the federal government launch a battleship that had a broken engine, faulty wiring or non-functional weapons systems into an ongoing fight? This software wasn’t simply “buggy” at launch — It was broken. These weren’t “glitches” caused by traffic, although the surge of traffic did expose where the system didn’t work quickly. Now that a reported 90% of users are able to register, other issues on the backend that are just beginning to become clear, from subsidy calculation to enrollment data to insurers reporting issues with what they are receiving to serious concerns about system security.

Based upon what we know about troubles at, it appears that people from the industry that were brought in to test the system a month ago urged CMS not to go live. It also appears that inside the agency who saw what was going on warned leadership months in advance that the system hadn’t been tested end-to-end. Anyone building enterprise software should have the code locked down in the final month and stopped introducing new features 3-6 months prior. Instead, it appears new requirements kept coming in and development continued to the end. The result is online now. (Or offline, as the case may be.)

On September 30, President Obama could have gone before the American people and said that the software was clearly not ready, explained why and told Americans that his administration wouldn’t push it live until they knew the system would work. HHS could have published a downloadable PDF of an application that could be mailed in and a phone number on the front page, added more capacity to the call centers and paper processing. It’s notable that three weeks later, that’s pretty much what President Obama said they have done.

The failed launch isn’t just about “optics,” politics or the policy of the Affordable Care Act itself, which is a far greater shift in how people in the United States browse, buy, compare and consume health insurance and services. A working system represents the faith and trust of the American people in the ability of government. This is something Jennifer Pahlka has said that resonates: how government builds websites and software matters, given the expectations that people now have for technology. The administration has handed the opponents of the law an enormous club to bash them with now — and they’ll deserve every bit of hard criticism they get, given this failure to execute on a signature governance initiative.

Articles worth reading on and potential reforms

  • Whenever I see Fred Trotter, I’m reminded that he’s forgotten more about open source software and healthcare IT than I’m ever likely to learn. Last week, he talked with Ezra Klein about the issues with
  • Ezra Klein also talked  to Clay Johnson about the lessons of (hint: procurement, project management and insourcing)
  • Tough reporting on failures in e-government is critical to improving those services for all, but particularly for the poor.
  • A post by Development Seed founder Eric Gunderson on the open source front-end for “It’s called Jekyll, and it works.”
  • Rusty Foster on it could have been worse. This failure to (re)launch just happened under vastly more political scrutiny and deadlines set by Congress. The FBI’s Sentinel program, by contrast, had massive issues — but you didn’t see the Speaker of the House tweeting out bug reports or cable news pundits opining about issues. The same is true of many other huge software projects.
  • A must-read op-ed by former Obama campaign CTO Harper Reed and Blue State Digital co-founder and former Presidential Innovation Fellow Clay Johnson on what ails government IT, adding much-needed context to what ailed
  • A Mother Jones interview that asked whether Reed and other former campaign staff could fix (Spoiler: No.)
  • If not those folks, then how should the administration fix In the larger sense, either the federal government will reform how it buys, builds and maintains software, through a combination of reforming procurement with modular contracting, bringing more technologists into government, and adopting open source and agile development processes …or this will just keep happening. The problems go much deeper that a “website.”
  • Ezra Klein pulled all of these pieces together in a feature on the “broken promise of better government through technology” at the end of the month. (He may have been heard in the Oval Office, given that the president has said he reads him.) Speaking at an “Organizing for America” event on November 4th, President Obama acknowledged the problem. “…I, personally, have been frustrated with the problems around the website on health care,” he said, “And it’s inexcusable, and there are a whole range of things that we’re going to need to do once we get this fixed – to talk about federal procurement when it comes to IT and how that’s organized…”
  • The issues behind cannot only be ascribed to procurement or human resources, as Amy Goldstein and Juliet Eilperin reported in the Washington Post: insularity and political sensitivity were a central factor behind the launch..

    Based on interviews with more than two dozen current and former administration officials and outsiders who worked alongside them, the project was hampered by the White House’s political sensitivity to Republican hatred of the law — sensitivity so intense that the president’s aides ordered that some work be slowed down or remain secret for fear of feeding the opposition. Inside the Department of Health and Human Services’ Centers for Medicare and Medicaid, the main agency responsible for the exchanges, there was no single administrator whose full-time job was to manage the project. Republicans also made clear they would block funding, while some outside IT companies that were hired to build the Web site,, performed poorly.

  • What could be done next? Congress might look across the Atlantic Ocean for an example. After one massive IT failure too many, at the National Health Service the United Kingdom created and empowered a Government Digital Services team. UK Executive Director of Digital Mike Bracken urged U.S. to adopt a digital core.”
  • In the video below, Clay Johnson goes deep on what went wrong with and suggests ways to fix it.

  • Can the White House and Congress take on the powerful entrenched providers in Washington & do the same? I’m not optimistic, unfortunately, given the campaign contributions and lobbying prowess of those entities, but it’s not an impossible prospect. I’ll write more about it in the future.

White House turns to Twitter to discuss President Obama’s Middle East speech

The role of mobile phones, Facebook, YouTube and Twitter in the revolutions that have swept the Middle East in 2011’s historic “Arab Spring” has been the subject of much debate for months. While connection technologies helped to accelerate and amplify the news coming out of Egypt, Tunisia, Syria, Yemen, Libya, Bahrain and the rest of the region, it’s the incredible bravery of the young people who have stood up for their freedom in the streets that made change happen.

Tomorrow, President Obama will deliver a speech that many experts expect to reshape the Middle East policy debate. The speech will be live-streamed from the State Department and available to anyone online  at In conjunction with the speech, the foremost curator of online media about the Arab Spring, Andy Carvin (@acarvin), will join with Foreign Policy’s Mark Lynch (@abuaardvark) in hosting a Twitter chat on the Middle East that will include an interview White House deputy national security advisor Ben Rhodes after the president concludes his remarks.

Earlier today, White House new media director Macon Phillips (@macon44) explained how the White House will continue the conversation on Twitter over at

Immediately afterwards, the live-stream will switch to a follow-up Twitter chat with Deputy National Security Advisor Ben Rhodes, where anyone will be able to pose questions and reactions via Twitter.

NPR’s Andy Carvin (@acarvin) and Foreign Policy’s Marc Lynch (@abuaardvark), two experts who bring both a deep understanding of foreign policy and extensive online networks, will facilitate a world-wide conversation that will include participants from the Middle East and North Africa.  As Andy explains:

Rather than come up with all the questions ourselves, we’d like to invite you to help us craft the questions. If you’re on Twitter and want to submit a question, please post a tweet with your question and include the hashtag #MEspeech in the tweet. You can pose your question before or during the speech. We won’t be able to get to every question, of course, so we encourage everyone to follow the #MEspeech hashtag and join the broader conversation about the speech on Twitter.

Folks at the White House (@whitehouse) will be keeping an eye on the #MESpeech hashtag as well, so be sure to use that to share thoughts before, during and after the speech.

Keep an eye on that hashtag tomorrow: this will get interesting.


UPDATE: I’ve embedded the archive from today’s Twitter chat at the end of the post along with video from the White House. Quick thoughts on the outcome of today’s “Twitter chat.”

1) I thought Carvin and Lynch did a phenomenal job, and that this was well worth their considerable effort. Carvin described the experience of a Twitter interview as akin to “juggling and riding a unicycle simultaneously, all the while trying to interview a senior administration official.” The two men asked tough questions from a global audience, starting with fundamental issues of trust and policy towards Bahrain, shared the questions and answers as they went on Twitter, often including the @ handle of the person who asked them. The result was a fascinating window into one future of new media, geopolitics and journalism. The considerable trust that they were given to find and ask the best questions was, in my eyes, paid back with interest. Well done.

2) It was also hard not to see a marked contrast between the recent “Facebook townhall” where founder Mark Zuckerberg asked President Obama all of the questions and Andy Carvin bringing questions culled from a real time global conversation to Ben Rhodes.

3) On a week when the executive editor of the New York Times offered considerable skepticism about Twitter’s value, this event serves as a quiet, powerful reminder of the platform’s global reach in the hands of those willing to fully engage the public there.

4) Poynter has an excellent background post regarding why NPR’s Andy Carvin is moderating the White House Middle East speech Twitter chat. NPR’s managing editor for digital news explained:

“Andy has cultivated a unique audience and following around the world and is the right person to carry on a conversation and channel a wide range of questions from around the world. He’s been doing it for months.”

If the interview were being conducted another way, Stencel said, perhaps a program host or a beat reporter would’ve done it.

Carvin “has the talent in this format that our hosts have in commanding the air. Andy is in essence the Neal Conan [host of NPR’s “Talk of the Nation”] of social media. He understands the format and he can conduct the interview in a way that takes a combination of smarts and skills that’s rare in media.”

That’s about right. In May 2011, there is no one using Twitter in this way more effectively than Andy Carvin. He has been more actively and deeply engaged in finding, validating and distributing the digital documentary evidence emerging from the Middle East on social networks and video sharing sites over the past six months than anyone else on the planet. He’s been storytelling using what Zenyep Tufecki aptly called “Twitter’s oral history” in a way that brings the voices and vision of millions of people crying out for freedom to the rest of the world. Today, he brought their questions into the State Department and White House. Pairing him with a foreign policy expert in Lynch brought significant weight and contextual experience to interviewing Rhodes. Well done.

For more, Nancy Scola wrote up Carvin and Lynch’s ‘Twitter interview’ with Rhodes over at techPresident. As always, she’s worth reading on it.