United States Releases Draft National Open Source Software Policy

IMG_1256On September 23, 2014, the White House announced that the United States would create an official policy for open source software. Today, the nation took a big step towards making more software built for the people available to the people.

“We believe the policies released for public comment today will fuel innovation, lower costs, and better serve the public,” wrote U.S. chief information officer Tony Scott in a blog post at WhiteHouse.gov, announcing that the Obama administration had published a draft open source policy and would now take public comments on it online.

This policy will require new software developed specifically for or by the Federal Government to be made available for sharing and re-use across Federal agencies. It also includes a pilot program that will result in a portion of that new federally-funded custom code being released to the public.

Through this policy and pilot program, we can save taxpayer dollars by avoiding duplicative custom software purchases and promote innovation and collaboration across Federal agencies. We will also enable the brightest minds inside and outside of government to review and improve our code, and work together to ensure that the code is secure, reliable, and effective in furthering our national objectives. This policy is consistent with the Federal Government’s long-standing policy of technology neutrality through which we seek to ensure that Federal investments in IT are merit-based, improve the performance of our Government, and create value for the American people.

Scott highlighted several open source software projects that the federal government has deployed in recent years, including a tool to find nearby housing counselors, NotAlone.gov, the College Scorecard, data.gov, and an online traffic dashboard. platform, and the work of 18F, which publishes all of its work as free and open software by default.

The draft policy is more limited than it might be: as noted by Greg Otto at Fedscoop, federal agencies will be required to release 20 percent of newly developed code as open source.

As Jack Moore reports at NextGov, the policy won’t apply to software developed for national security systems, a development that might prove disappointing to members of the military open source community that has pioneered policy and deployment in this area.

The draft policy sensibly instructs federal agencies to prioritize releasing of code that could have broader use outside of government.

The federal government is now soliciting feedback to the following considerations regarding its use of open source software.

Considerations Regarding Releasing Custom Code as Open Source Software

  • To what extent is the proposed pilot an effective means to fuel innovation, lower costs, benefit the public, and meet the operational and mission needs of covered agencies?
    • Would a different minimum percentage be more or less effective in achieving the goals above?
    • Would an “open source by default” approach that required all new Federal custom code to be released as OSS, subject to exceptions for things like national security, be more or less effective in achieving the goals above?
    • Is there an alternative approach that OMB should consider?
  • What are the advantages and disadvantages associated with implementing this type of pilot program? To what extent could this policy have an effect on the software development market? For example, could such a policy increase or decrease competition among vendors, dollar amounts bid on Federal contracts, or total life-cycle cost to the Federal Government? How could it impact new products developed or transparency in quality of vendor-produced code?
  • What metrics should be used to determine the impact and effectiveness of the pilot proposed in this draft policy, and of an open source policy more generally?
  • What opportunities and challenges exist in Government-wide adoption of an open source policy?
  • How broadly should an open source policy apply across the Government? Would a focus on particular agencies be more or less effective?
  • This policy addresses custom code that is created by Federal Government employees as well as custom code that is Federally-procured. To what extent would it be appropriate and desirable for aspects of this draft policy to be applied in the context of Federal grants and cooperative agreements?
  • How can the policy achieve its objectives for code that is developed with Government funds while at the same time enabling Federal agencies to select suitable software solutions on a case-by-case basis to meet the particular operational and mission needs of the agency? How should agencies consider factors such as performance, total life-cycle cost of ownership, security and privacy protections, interoperability, ability to share or reuse, resources required to later switch vendors, and availability of support?

If you have thoughts on any of these questions, you can email sourcecode@omb.eop.gov,
participate in discussions on existing issues on Github, start a new one, or make a pull request to the draft policy on Github. You can see existing pull requests here and view all comments received here.

With this policy, the White House has fulfilled one of the commitments added to the second National Action Plan for open government in the fall of 2014. While there has been limited progress (or worse) on of the dozens of other new and old commitments made in the three action plans published to date, this draft open source policy is a historic recognition of the principle that the source code for software developed by government agencies or contractors working for them can and should be released to other agencies and the general public for use or re-use.

Obama Administration Secretly Lobbied Against FOIA Reform In Congress

IMG_1992

A Freedom of Information Act lawsuit showed that the Obama administration vigorously lobbied against Freedom of Information Act reform in Congress.  The documents and correspondence, which were obtained through the Freedom of the Press Foundation’s lawsuit against the Justice Department and reported out by Jason Leopold at Vice Media, showed that the administration was literally lobbying against its own policy becoming law.

The Department of Justice’s six page memorandum shows that the agency opposed Congress making the exact language in Attorney General Eric Holder and President Obama’s 2009 memorandums on FOIA law.

The Justice Department opposing FOIA reform direct conflicts commitments made in the U.S. National Action Plan on Open Government required as part of its participation in  the Open Government Partnership.

I asked Ambassador Power how the United States can be a credible leader on open government if the White House and DoJ does this. In an alternate universe, she and the administration would respond publicly.

Unfortunately, it’s easy to predict the outcome of this news: publicly committing to open government reforms and then undermining them privately will erode abysmal levels of trust in government even more.

In the face of hypocrisy from the Justice Department on this count, the public should  call on their Senators to make the Freedom of Information Act reform legislation the House of Representatives passed in January into law.

U.S. Civil Society Groups release model National Open Government Action Plan

This is the week for seeking feedback on open government in the United States. 4 days ago, the White House published a collaborative online document that digitized the notes from an open government workshop held during Sunshine Week in March. Today, Abby Paulson from OpenTheGovernment.org uploaded a final draft of a Model National Action Plan to the Internet, as a .doc. I’ve uploaded it to Scribd and embedded it below for easy browsing.

Nelson shared the document over email with people who contributed to the online draft.

Thank you so much for contributing to the civil society model National Action Plan. The Plan has made its way from Google Site to Word doc (attached)! We will share these recommendations with the White House, and I encourage you to share your commitments with any government contacts you have. If you notice any errors made in the transition from web to document, please let me know. If there are any other organizations that should be named as contributors, we will certainly add them as well. The White House’s consultation for their plan will continue throughout the summer, so there are still opportunities to weigh in. Additional recommendations on surveillance transparency and beneficial ownership are in development. We will work to secure meetings with the relevant agencies and officials to discuss these recommendations and make a push for their inclusion in the official government plan. So, expect to hear from us in the coming weeks!

From Stickies to Wikis: White House seeks online feedback on open government

In a followup post, the White House shared a link to a collaborative online document where the notes from an open government workshop held during Sunlight Week were posted online for comment. In doing so, they moves from sticky notes to a wiki.

What will come of asking the broader public for feedback on the ideas that a workshop of advocates and policy wonks in DC suggested? Stay tuned.

On its 3rd anniversary, opportunities and challenges for the Open Government Partnership

opgFrance1

In 2010, President Barack Obama spoke to the United Nations General Assembly about open government. “The common thread of progress is the principle that government is accountable to its citizens,” he said, “and the diversity in this room makes clear — no one country has all the answers, but all of us must answer to our own people.”

In all parts of the world, we see the promise of innovation to make government more open and accountable.  And now, we must build on that progress.  And when we gather back here next year, we should bring specific commitments to promote transparency; to fight corruption; to energize civic engagement; to leverage new technologies so that we strengthen the foundations of freedom in our own countries, while living up to the ideals that can light the world.

Open government, said Samantha Power, now the U.S. ambassador to the United Nations, could have a global impact.

In 2011, a historic Open Government Partnership launched in New York City, hailed as a fresh approach to parting the red tape by the Economist. “The partnership is really the first time that there is a multilateral platform to address these issues,” said Maria Otero, former under secretary of state for democracy and global affairs at the United States State Department. “The partnership could have focused on countries come in and present best practices and exchange ideas and then just go home.”

“The partnership is really focused on first having countries participate that have already demonstrated interest in this area and have already put in place a number of specific things and the material laid out, if you will, the minimum standards that are being requested. What the partnership really looks for is to provide a mechanism by which the countries can each develop their own national plans on ways to expand what they’re doing on transparency, accountability, and civic engagement, or to start new initiatives for them. That is really what is very different and important about this partnership, is that it is very action- and results-oriented.”

In 2012, the Open Government Partnership became a player on the world stage as it hosted a global gathering of national leaders and civil society an annual meeting in Brazil, with the responsibilities and challenges that accompany that role, including pushing participants to submit missing action plans and progress reports, not just letters of commitment.

In January 2013, Power hailed the Open Government Partnership (OGP) as President Obama’s signature governance initiative:

It’s not about the abstraction about ‘fighting corruption’ or ‘promoting transparency’ or ‘harnessing innovation’ — it’s about ‘are the kids getting the textbooks they’re supposed to get’ or does transparency provide a window into whether resources are going where they’re supposed to go and, to the degree to which that window exists, are citizens aware and benefiting from the data and that information such that they can hold their governments accountable. And then, does the government care that citizens care that those discrepancies exist?

In May 2013, a seminal event in the evolution of OGP occurred when Russia withdrew from the Open Government Partnership:

If the dominant binary of the 21st century is between open and closed, Russia looks more interested in opting towards more controllable, technocratic options that involve discretionary data releases instead of an independent judiciary or freedom of assembly or the press. One of the challenges of the Open Government Partnership has always been the criteria that a country had to pass to join and then continue to be a member. Russia’s inclusion in OGP instantly raised eyebrows, doubts and fears last April, given rampant corruption in the public sector and Russia’s terrible record on press freedom. “Russia’s withdrawal from the OGP is an important reminder that open government isn’t easy or politically simple,” said Nathaniel Heller, executive director of Global Integrity. “While we don’t yet fully understand why Russia is leaving OGP, it’s safe to assume that the powers that be in the Kremlin decided that it was untenable to give reformers elsewhere in the Russian government the freedom to advance the open government agenda within the bureaucracy.”

In November 2013, the world may have hit ‘peak open‘ at the OGP annual summit in London, despite the partnerships’ members facing default states of closed.

Swirling underneath the professional glitz of an international summit were strong undercurrents of concern about its impact upon governments reluctant to cede power, reveal corruption or risk embarrassment upon disclosure of simple incompetence. The OGP summit took place at a moment where 21st century technology-fueled optimism has splashed up against the foundations of institutions created in the previous century. While the use of the Internet as a platform for collective action has grown, so too have attendent concerns about privacy and surveillance, in the wake of disclosures by NSA contractor Edward Snowden, where the same technologies that accelerated revolutions across the Middle East and North Africa are being used to capture and track the people advocating for change.

In 2014 the Open Government Partnership has matured and expanded, with France joining earlier in the year and Bosnia and Herzegovina bringing the total number of participating countries to 65 out of about 88 eligible countries worldwide. As OGP turns three, the partnership is celebrating the success of its expansion and looking ahead to its future, with a clearer mission and goals and ambitious four year strategy (PDF). The partnership is finally writing letters to countries that are not living up to their commitments, although the consequences for their continued participation if they do not comply remain to be seen.

The challenges and opportunities ahead for a partnership that provides a platform for civil society to hold government accountable are considerable, given the threats to civil society worldwide and the breathtaking changes brought about through technological innovation. Today, 10 national leaders will speak in New York City to mark OGP’s third anniversary. (I’ll be there to listen and share what I can.)

After the speeches end and the presidents and prime ministers return home, serious questions will remain regarding their willingness to put political capitol behind reforms and take tough stands to ensure that their governments actually open up. Digital government is not open government, just as not all open data supports democratic reforms.  As Mexico prepares to become lead co-chair of OGP, one element that didn’t make it into the challenges listed for the country is the state of press freedom in Mexico. As the Committee to Protect Journalists highlighted, open government is not sustainable without a free press. As long as the murders of journalists go unpunished in Mexico, the commitments and efforts of the Mexican national government will have to be taken in context.

Given this blog’s past stance that as press freedom goes, so too does open government, I’ve signed a petition urging the White House to explicitly support a right to report. Every other country that has committed to open government should do the same. Given OGP’s own challenges around the media and open government (PDF), I would also urge the partnership to make sure that press freedom and freedom of expression occupies a prominent place in its advocacy efforts in the years ahead.

The White House (quietly) asks for feedback on the open government section of its website

Obama at computer. Official White House Photo by Pete Souza

Official White House Photo by Pete Souza

Over at Govfresh, Luke Fretwell took note of the White House asking for feedback on the open government section of WhiteHouse.gov. Yesterday, Corinna Zarek, senior advisor for open government in the White House Office of Science and Technology Policy (OSTP), where the administration’s Open Government Initiative was originally spawned under former deputy chief technology officer Beth Noveck, published a email to the US Open Government Google Group:

We are working on a refresh of the Open Gov website, found at whitehouse.gov/open, and we’d like your help!

If you’re familiar with the history of the page, you can see we have begun updating it by shifting some of the existing content and adding new tabs and material.

What suggestions do you have for the site? What other efforts might we feature?

Please let us know – reply back to this thread, email us at opengov@ostp.gov, or tweet us at @OpenGov!

Here’s some background on the group and its purpose: The White House’s Open Government Working Group needs to solicit feedback from civil society in the United States on the various initiatives and commitments the administration has made. Such engagement is essential to the providing feedback from governance experts, advocates and the public on the development of new agency open government plans and discuss progress on the national open government action plan.

As a result of a discussion at the working group this spring, OSTP created the US Open Government discussion group to connect White House staff and agency officials who work on open government to people outside of the federal government. According to the group’s description, the goal of this group is to “provide a safe and welcoming arena for government-focused collaboration and news-sharing around Open Government efforts of the United States government.” That “safe and welcoming” language is notable: the group is moderated by OpenTheGovernment.org with an eye on constructive, on-topic feedback, as opposed to, say, the much more open-ended freewheeling posts and threads on the (long-since closed) Open Government Dialog of 2009 or Change.gov.

After almost six months, the open government group, which can be accessed through a Web browser or using an email listserv, has 177 members and 37 posts. By almost any measure, these are extremely low levels of participation and engagement, although the quality of feedback from those members remains extremely high. By way of contrast, a open government and civic tech group on Facebook now has over 1900 members and an open government community on Google+ has over 1400 members, with both enjoying almost daily contributions. Low participation rates on this US Open Government Google Group are likely due in part to lack of promotion by other White House staff to the media or using the various social media platforms has joined, which cumulatively have millions of followers, and, more broadly, the historic lows of public trust in government which have created icy headwinds for open government initiatives in recent years.

So far, Zarek’s solicitation has received two responses. One comes from Daniel Schuman, policy director for Citizens for Responsibility and Ethics (CREW) in Washington, who made great suggestions, like adding a link to ethics.data.gov, a list of staff working on openness in the White House and their areas of responsibility, a link to 18f and the USDS.

“Finally, there are many great ideas about how to make government more open and transparent,” wrote Schuman. “Consider including a way for people to submit ideas where those submissions are also visible to the public (assuming they do not violate TOS). Consider how agencies or the government could respond to these suggestions. Perhaps a miniature version of “We the People,” but without the voting requiring a response.”

The other idea comes from open government consultant Lucas Cioffi, who suggested adding a link to a “community-powered open government phone hotline” like the experiment he recently created.

To those ideas, I’ll add eight quick suggestions in the spirit of open government:

1) Reinstate the open government dashboard that was removed and update it to the current state of affairs and compliance, with links to each. The Sunlight Foundation and CREW have already audited agency compliance with the Open Government Directive. By keeping an updated scorecard in a prominent place, the Obama administration could both increase transparency to members of the public wondering about what has been done and by whom, and put more pressure on agencies to be accountable for the commitments they have made.

2) Re-integrate individual case studies from the “Innovator’s Toolkit,” which was also removed, under participation and collaboration

3) Create a Transparency tab and link to the “IC on the Record” tumblr and other public repositories for formerly secret laws, policies or documents that have been released.

4) Blog and tweet more about what’s happening in the open government world outside of the White House. Multiple open government advocates do daily digests and there’s a steady stream of news and ideas on the #opengov and #opendata hashtags on Twitter. Link to what’s happening and show the public that you’re reading and responding to feedback.

5) Link to the White House account and open government projects on Github under both the new participation and collaboration tabs, like Project Open Data.

6) Highlight 18F’s effort to reboot the Freedom of Information Act.

7) Publish the second national action plan on open government as HTML on the site, and post and link to a version on Github where people can comment on it.

8)  Create a FAQ under “participation” that lists replies to questions sent to @OpenGov

If you have ideas for what should be wh.gov/open, well, now you know who to tell, and where.

18F launches alpha foia.gov in a bid to reboot Freedom of Information Act requests for the 21st century

alpha foia gov

18F, the federal government’s new IT development shop, has launched a new look at the Freedom of Information Act (FOIA) in the form of a open source application hosted on Github. Today’s announcement is the most substantive evidence yet that the Obama administration will indeed modernize the Freedom of Information Act, as the United States committed to doing in its second National Action Plan on Open Government. Given how poor some of the “FOIA portals” and underlying software that supports them exists is at all level of government, this is tremendous news for anyone that cares about the use of technology to support open government.

Notably, 18F already has a prototype (pictured above) online that shows what a consolidated request submission hub could look like and plans to iterate upon it.  This is a perfect example of “lean government,” or the application of lean startup principles and agile development to the creation of citizen-centric services in the public sector.  Demonstrating its commitment to developing free and open source software in the open, 18F asked the public to follow the process online at their FOIA software repository on Github, send them feedback or even contribute to the project.

18F has now committed to creating software that improvse how requests made under the Freedom of Information Act can be improved through technology. Specifically that it will develop tools that “improve the FOIA request submission experience,” “create a scalable infrastructure for making requests to federal agencies” and “make it easier for requesters to find records and other information that have already been made available online.”

According to 18F’s blog post, this work is supported and overseen by a “FOIA Task Force,” consisting of representatives from the Department of Justice, Environmental Protection Agency, the Office of Management and Budget, the Office of Science and Technology Policy. The task force will need to focus upon more than technology: while poor software has hindered requests and publishing, that’s not the primary issue that’s hindering the speed or quality of responses.

Despite the U.S. attorney general’s laudable commitment to a new era of open government in 2009, the Obama administration received a .91 GPA in FOIA compliance earlier this year from the Center for Effective Government.

While White House press secretary Josh Earnest may be well correct in stating that the federal government is processing more FOIA requests than ever, As the National Security Archive noted in March, the use of a FOIA exemption (protecting “deliberative processes”) to deny or heavily redact requests has skyrocketed in the past two years.

use of B5 exemptions

[NATIONAL SECURITY ARCHIVE: Chart created by Lauren Harper.]

As with the reduced access to government staff and scientists that a group of 38 journalism and open government advocates decried earlier this year, improving FOIA compliance cannot solely be addressed through technological means. To address endemic government secrecy and outright abuse of exemptions to protect against politically inconvenient disclosures, Obama administration — in particular, the U.S. Justice Department — will need to expend political capital and push agencies to actually shift the cultural default towards openness and release uncomfortable or embarrassing data and documents and not redact them beyond understanding.

That’s admittedly a huge challenge, particularly for an administration facing multiple foreign and domestic conundrums, including a scandal over missing IRS emails and obfuscated records in an election year and the most politically polarized Congress and electorate in the nation’s history, but if President Barack Obama is truly committed to “creating an unprecedented level of openness in government,” it’s one that he and his administration will need to take on.

[REPORT] On data journalism, democracy, open government and press freedom

On May 30, I gave a keynote talk on my research on the art and science of data journalism at the first Tow Center research conference at Columbia Journalism School in New York City. I’ve embedded the video below:

My presentation is embedded below, if you want to follow along or visit the sites and services I described.

Here’s an observation drawn from an extensive section on open government that should be of interest to readers of this blog:

“Proactive, selective open data initiatives by government focused on services that are not balanced by support for press freedoms and improved access can fairly be criticized as “openwashing” or “fauxpen government.”

Data journalists who are frequently faced with heavily redacted document releases or reams of blurry PDFs are particularly well placed to make those critiques.”

My contribution was only one part of the proceedings for “Quantifying Journalism: Metrics, Data and Computation,” which you can catch up through the Tow Center’s live blog or TechPresident’s coverage of measuring the impact of journalism.