Today in Washington, DC, the White House Office of Management and Budget and the Data Coalition are hosting a public forum for feedback on the draft action plan for the new federal data strategy. Remote access is available for those … Continue reading
For a decade, I’ve tracked the contours of open data, digital journalism, open source software and open government, publishing research on the art and science of data journalism that explored and tied together those threads.
This week, I’m proud to announce that a new book chapter on open data, journalists, and the media that I co-authored with data journalism advisor Eva Constantaras has been published online as part of “The State of Open Data: Histories and Horizons!”
We joined 43 other authors in a 18-month project that reflected on “10 years of community action and review the capacity of open data to address social and economic challenges across a variety of sectors, regions, and communities.”
The publication and the printed review copy I now possess is the end of a long road.
Back in January 2018, I asked a global community of data journalists, open government advocates, watchdogs and the public for help documenting the “State of Open Data” and journalism for a new project for Open Data for Development. While I was at the Sunlight Foundation, I seeded the initial network scan on journalists, media and open data.
Over the past year, people and organizations from around the world weighed in – & Eva Constantaras joined me as co-editor & lead author, creating a “network scan” of the space. The writing project traveled with me, after I left Sunlight, and over this winter we edited and synthesized the scan into a polished chapter.
The book was originally going to be introduced at the 5th International Open Government Data Conference in Buenos Aires, in the fall of 2018, but will instead be officially launched at the Open Government Partnership’s global summit in Ottawa, Canada at the end of May 2019.
The book, which was funded by the International Research Centre and supported by the Open Data for Development (OD4D) Network. As a result of that support, “The State of Open Data” was also published in print by African Minds, from whom it may also be purchased. (While I was paid to edit and contribute, I don’t receive any residuals.)
Last night, I joined Tim Davies and other authors at the OpenGov Hub in DC to talk about the book & our chapters. Video of Davies, who managed to cover a tremendous amount of ground in ten minutes, is embedded below.
Thank you to everyone who contributed, commented and collaborated in this project, which will inform the public, press and governments around the world.
On May 10, the Select Committee on the Modernization of Congress in the United States House of Representatives held a hearing on “opening up the process,” at which 4 different experts talked with Congress about making legislative information more transparent,” from ongoing efforts to proposed reforms to the effect of sunshine laws passed decades ago.
If you’re not up to speed on this committee, it was established on January 4, 2019 the House voted in favor of establishing the Select Committee by an overwhelming margin (418-12) by adopting of Title II of H.Res.6, the Rules of the House of Representatives for the One Hundred Sixteenth Congress, with a sole authority of investigating, studying, holding public hearings, making finding, and developing recommendations to modernize Congress – but no legislative jurisdiction nor ability to take legislative action. The committee has has been fairly described by IssueOne as “the best opportunity in decades” for Congress to improve itself, by looking inward.
The two hours of discussion on May 10 mostly added up to good news for good government, with useful summaries of progress opening up Congress to date from the deputy clerk of the House, proposed recommendations and reforms from GovTrack founder Josh Tauberer and DemandProgress policy director Daniel Schuman that would build on that progress, and some skepticism of sunshine in the legislative process from University of Maryland professor Frances Lee.
I attended the hearing and tweeted from it, as has been my practice for nearly a decade in DC:
The thread of tweets above, however is not meant to be comprehensive, nor could it be fully contextualized in the moment. For that, watch the hearing on YouTube, in the video embedded below:
…and read a summary of the hearing from the Congressional Institute, reporting from Federal Computer Week on “transparency through technology” and Roll Call on ongoing development of an “artificial intelligence engine” (applying machine-learning to structured legislative data), and the latest edition of Demand Progress’ First Branch Forecast, in which Schuman summarizes his testimony and aspects of the hearing.
He and Tauberer recommended from appointing a legislative branch Chief Data Officer, releasing structured data that would feed into the clerk’s tool to show how proposed amendments would change bills, and enact a mandate and open standards for a unique identifier for lobbyists across the U.S. government. Whether those ideas make it into the committee’s recommendations remains to be seen, but they’re worth weighing – along with further studying of the value or risk of increasing or decreasing public access to various aspects of the deliberative processes that constitute legislative and oversight activities.
On a meta note, the process on display at this forum was notable for comity between witnesses and members, openness to the public and press, and engagement online.
The medium is still the message, when it comes showing (not telling) how open a given institution is.
While paying attention to the digital component has become downright mundane in 2019, the Committee demonstrated admirable competence, streaming the hearing online at YouTube, publicized it on social media prior to the event, engaged with the public during the hearing, and published testimony on its website afterwards. (Unfortunately, there’s no text transcription of the hearing on the hearing page. Given the committee’s acknowledgement of the importance of accessibility, it should make sure to get transcripts online.)
As at the most recent “Congressional hackathons,” Members of Congress were able to show good government, ethics and transparency, far away from partisan rancor over white hot political issues or Congressional attempts to conduct constitutional oversight of a corrupt administration.
If you’re interested in following the activities of the committee or providing feedback, visit Modernizecongress.House.gov and scrub in: the People’s House will only be as open, accessible, accountable, and effective as we make it – or demand it to be. There’s no one else coming to help.
For those unfamiliar with this annual celebration of open government around the United States, Sunshine Week was founded over a decade ago by the American Society of Newspaper Editors, who now support it with the Reporters Committee for Freedom of the Press.
Sunshine Week always falls around President James Madison’s birthday on March 16. Madison is generally regarded as the forefather of open government in the United States, as evidenced by this memorable letter in 1822:
“Knowledge will forever govern ignorance: And a people who mean to be their own Governors, must arm themselves with the power which knowledge gives,” he wrote.
In 2018, the “information darkness” of the Trump administration led to ignominy, from the Electronic Frontier Foundation’ “Foilies” to the Society for Professional Journalists’ Black Hole Award to the National Security Archives’ Rosemary Awards.
In 2019, transparency and accountability have taken on additional context during the Trump administration, which has continued to be allergic to transparency, rife with conflicts of interest, and hostile to the essential role journalism plays in a democracy. When the President of the United States repeatedly calls journalists “the enemy of the people,” a disinformation virus is weakening our body politic.
Despite the enactment of a historic open government data bill, the state of open government (data) remains divided, at risk, and underfunded in the United States.
After years of delays and democratic regression, the US government released a weak open government plan for the Open Government Partnership that was not responsive to the demands of this moment. The Open Government Partnership’s researchers found backsliding in the USA, confirming what watchdogs have been highlighting daily about this administration since January 2017.
Every Sunshine Week is an opportunity to take stock of how federal, state and local governments are complying with public records, public meetings laws, and ethics statutes with disclosure requirements, from city halls to legislatures. As the National Security Archive highlighted, the state of the Freedom of Information Act is cloudy:
Representative Elijah Cummings, D-Md, and Senators Chuck Grassley, R-Iowa, Patrick Leahy, D-Vt., and John Cornyn, R-Texas, recently sent a stern letter to the Interior Department reprimanding the agency for its efforts to weaken its FOIA regulations and urging it to reconsider the rule change. The proposed rule changes which garnered more than 65,000 comments – include allowing the DOI to preemptively reject what it defines as “unreasonably burdensome” requests, the possibility of imposing a monthly limit to the number of either pages or requests from a single requester the agency will process, and a host of other changes that may make it more difficult to obtain fee waivers and expedited processing.
The bipartisan group of lawmakers, all seasoned FOIA champions, told the Interior Department, “We write to express significant concern with the rule recently proposed by the Department of the Interior (DOI) concerning its Freedom of Information Act (FOIA) procedures. The proposed rule appears to restrict public access to DOI’s records and delay the processing of FOIA requests in violation of the letter and spirit of FOIA. The American people have the right to access information from DOI, and the proposed rule needlessly encroaches on that right.”
The context for oversight of open government at the national level for this year, however, is different. 2018 midterm elections delivered a 116th Congress that brought with it a commitment to oversight that was sorely lacking in the last session. A core element of that oversight has been public hearings that hold public officials and corporate executives accountable for their service or services.
That will continue next week, when, as has been the tradition in past years, the U.S. House Oversight Committee will be holding a hearing during Sunshine Week in 2019, considering the Freedom of Information Act and transparency under the Trump administration.
There will be many other Sunshine Week events around DC as well. The Department of Justice’s awards for FOIA officers. The National Archives celebrating the 10th anniversary of the Office of Government Information Services (aka the federal FOIA ombudman) during the day on March 11. That night, there will be a panel discussion on how to obtain and improve coverage of climate data at the National Press Club hosted by its Journalism Institute and Freedom of the Press Committee.
There will be a DC Open Government Summit and a forum on science in the Trump era put on by the Government Accountability Project, the Union of Concerned Scientists, and the Sunlight Foundation’s Web Integrity Project on the evening of March 12.
Despite the threats to American democracy at the federal level and ongoing challenges to open government in the states and cities, there’s still much to celebrate in 2019.
The free press, independent judiciary, and watchdog organizations have continued to provide transparency when elected officials and civil servants have tried to make decisions in secrecy, shedding light on corruption, fraud, waste, abuse and incompetence.
Their combined efforts to bring in sunshine in government across American civil society have been a bulwark against tyranny and corruption in the United States and around the world. Thank you to everyone who continues to support, defend and extend the public’s right to know in the 21st century.
If the American public wants to see meaningful progress on transparency, accountability or ethics in U.S. government, it should call on Congress to act, not the Trump White House. With little fanfare or notice, the United States of America has … Continue reading
What’s next for open data in the United States? That was the open question posed at the Center for Data Innovation (CDI) last week, where a panel of industry analysts and experts gathered to discuss the historic open government data … Continue reading
In March 2018, three public policy scholars posted a provocative question: could the open government movement shut the door on freedom of information? At the time, I let it flow without refuting it from the Sunlight Foundation’s platform, but it’s … Continue reading
On December 21, 2018, the United States House of Representatives voted to enact H.R. 4174, the Foundations for Evidence-Based Policymaking Act of 2017, in a historic win for open government in the United States of America.
The Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN Government Data Act) is about to become law as a result. This codifies two canonical principles for democracy in the 21st century:
- public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security
- federal agencies should use evidence when they make public policy
For the full backstory on what’s in the bill and how it came to pass, read yesterday’s feature.
It’s worth noting that last minute objection did result in two amendments that the Senate had to act upon. Thankfully, on Saturday, December 22nd, the Senate acted, passing the resolution required to send the bill onwards to the president’s desk.
Here’s what changed: First, the text of Title I was amended so that it only applied to CFO Act agencies, not the Federal Reserve or smaller agencies. Title II (the Open Government Act) still applies to all federal agencies.
Second, there was a carve out in Title I “for data that does not concern monetary policy,” which relates to the Federal Reserve, among others.
While the shift weakened the first title of the bill a bit, this was still a historic moment: Congress has passed a law to make open data part of of the US Code.
While the United States is not the first or even the second democracy to pass an open data law – France and Germany have that distinction – this is a welcome advance, codifying the open government data policies, practices, roles and websites (looking at you, Data.gov) that the federal government had adopted over the past decade.
Open government activists, advocates and champions continue to celebrate, online and off.
Victory! Last night the @DataCoalition got the #OPENGovData Act through the Senate, as part of H.R. 4174. Expected to sail through the House TODAY. Sets a presumption, in law, that govt info ought to be published as #opendata, using data standards. https://t.co/jTdM1rIVTW
— Hudson Hollister (@hudsonhollister) December 20, 2018
#Opendata will soon be the way our government publishes information by default!! What a great day for #opendata #opengov #transparency! This would not have been possible w/o @SpeakerRyan @PattyMurray @SenBrianSchatz @SenSasse @RepDerekKilmer. Thank you! #OPENGovDataAct (HR 4174)
— Data Coalition (@DataCoalition) December 21, 2018
Merry Christmas to me! The U.S. Congress has passed an open data law. pic.twitter.com/tDwo8j8UCB
— Rebecca Williams (@internetrebecca) December 21, 2018
Excited to see the Open Government Data Act pass the House today! H.R. 4174 will enable libraries to provide businesses, researchers and students with valuable data that fuels innovation and economic growth. #OPENGovData #opendata
— Gavin Baker (@OpenGavin) December 21, 2018
“The bipartisan passage of the Foundations for Evidence-Based Policymaking Act is a significant step toward a more efficient, more effective government that uses evidence and data to improve results for the American people,” said Michele Jolin, CEO and co-founder of Results for America, in a statement. “We commend Speaker Ryan, Senator Murray and their bipartisan colleagues in both chambers for advancing legislation that will help build evidence about the federally-funded practices, policies and programs that deliver the best outcomes. By ensuring that each federal agency has an evaluation officer, an evaluation policy and evidence-building plans, we can maximize the impact of public investments.”
“The OPEN Government Data Act will ensure that the federal government releases valuable data sets, follows best practices in data management, and commits to making data available to the public in a non-proprietary and electronic format,” said Daniel Castro, in a statement. “Today’s vote marks a major bipartisan victory for open data. This legislation will generate substantial returns for the public and private sectors alike in the years to come.”
“The passage of the OPEN Government Data Act is a win for the open data community”, said Sarah Joy Hays, Acting Executive Director of the Data Coalition, in a statement. “The Data Coalition has proudly supported this legislation for over three years, along with dozens of other organizations. The bill sets a presumption that all government information should be open data by default: machine-readable and freely-reusable. Ultimately, it will improve the way our government runs and serves its citizens. This would not have been possible without the support of Speaker Paul Ryan (WI-1-R), Senators Patty Murray (WA-D), Brian Schatz (HI-D), Ben Sasse (NE-R), and Rep. Derek Kilmer (WA-6-D). Our Coalition urges the President to promptly sign this open data bill into law.”
Congratulations to everyone who has pushed for this outcome for years.
[Image Credit: Sunlight Foundation]
This post has been updated, and corrected: France was ahead of Germany in enacting an open data law.
The arc of open government in United States is long, but perhaps it will bend towards transparency and accountability as 2018 comes to a close, in an unlikely moment in our history. After years of dedicated effort by advocates and bipartisan leadership in both houses of Congress, the Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN Government Data Act) is about to become law after the United States Senate passed the bill as part of H.R. 4174 on December 19.
Senator Brian Schatz (D-HI) shared the news in a tweet last night:
We just passed a bill that requires all data the govt collects (that isn’t secret or private) to be machine readable and interoperable. It’s data that taxpayers paid for and they deserve access – weather, traffic, census, budget numbers – it’s your info and you should have it.
— Brian Schatz (@brianschatz) December 19, 2018
…followed by Representative Derek Killmer (D-
Last night the Senate passed a bill I introduced in the House called the Open Government Data Act! There are so many possibilities for our economy when folks can access data they paid to develop and fund through the nation’s largest angel investor, Unce Sam. https://t.co/BvMX54mcCY
— Rep. Derek Kilmer (@RepDerekKilmer) December 20, 2018
Here’s Schatz speaking about the bill at a Data Coalition event last winter:
Two steps remain: passage of the bill in the House and President Donald J. Trump signing it into law. Barring a scheduling issue or unexpected change (keep an eye out for shenanigans on the House floor today), the nation is close to a historic codification of two powerful principles:
- public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security
- federal agencies should use evidence to make public policy
Along with making open government data the default in U.S. government and requiring the White House Office of Management and Budget to oversee enterprise data inventories for every agency, the bill would require federal agencies to do the following, as listed in the summary from the Law Library of Congress:
This bill requires departments and agencies identified in the Chief Financial Officers Act to submit annually to the Office of Management and Budget (OMB) and Congress a plan for identifying and addressing policy questions relevant to the programs, policies, and regulations of such departments and agencies.
The plan must include: (1) a list of policy-relevant questions for developing evidence to support policymaking, and (2) a list of data for facilitating the use of evidence in policymaking.
The OMB shall consolidate such plans into a unified evidence building plan.
The bill establishes an Interagency Council on Evaluation Policy to assist the OMB in supporting government-wide evaluation activities and policies. The bill defines “evaluation” to mean an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency.
Each department or agency shall designate a Chief Evaluation Officer to coordinate evidence-building activities and an official with statistical expertise to advise on statistical policy, techniques, and procedures.
The OMB shall establish an Advisory Committee on Data for Evidence Building to advise on expanding access to and use of federal data for evidence building.
Open, Public, Electronic, and Necessary Government Data Act or the OPEN Government Data Act
This bill requires open government data assets to be published as machine-readable data.
Each agency shall: (1) develop and maintain a comprehensive data inventory for all data assets created by or collected by the agency, and (2) designate a Chief Data Officer who shall be responsible for lifecycle data management and other specified functions.
The bill establishes in the OMB a Chief Data Officer Council for establishing government-wide best practices for the use, protection, dissemination, and generation of data and for promoting data sharing agreements among agencies.
While the United States would not be not the first democracy to pass such a law, it would be a welcome advance, codifying many aspects of the open government data policies that have been developed and promulgated in the federal government over the past decade.
How did open government data get into the US Code?
This was no accident of fate or circumstance: This bill, which was previously passed by the House last month, was sponsored by the Speaker of the House, Paul Ryan. It’s an important element of his legislative legacy, and one that can and should earn praise – unlike other aspects of his time with the gavel.
It’s taken years of advocacy and activism by a broad coalition to get here, including the Sunlight Foundation, the EFF, Business Software Alliance, Center for Data Innovation, the Scholarly Publishing and Academic Resources Coalition, American Library Association, the R Street Institute, among many others, and bipartisan efforts on both sides of the aisle. Senator Ben Sasse (R-NE) co-sponsored the OGDA in the Senate, with 5 others, and former Representative Blake Farenthold (R-TX) cosponsored it in House, with 12 others.
That original bill almost made it into law in 2016, when the Senate passed OGDA by unanimous consent, but the House didn’t move it before the end of the 115th Congress. In September 2017, when it was poised to pass Congress as part of the National Defense Authorization Act., before it was stripped from the final version.
In October 2017, the text of the Open Government Data Act, however, was incorporated into H.R. 4174, the Foundations for Evidence-Based Policymaking Act of 2017,
The new bipartisan, bicameral companion legislation was introduced on October 31, 2017 by Speaker of the House Paul Ryan (R-WI) and Senator Patty Murray (D-WA) to enact the recommendations contained in the final report from the Commission of Evidenced-Based Policy.
While it has been watered down a bit, what I argued then is still true today: the bill “offers a genuine opportunity to not only improve how the nation makes decisions but embed more openness into how the federal government conducts the public’s business.”
The addition of OGDA into that bill was “an important endorsement of open government data by one of the most powerful politicians in the world” and “a milestone for the open movement, an important validation of this way of making public policy, and the fundamental principles of data-driven 21st century governance.”
The OGDA was one of the primary legislative priorities for me during my years as a senior analyst and then deputy director at the Sunlight Foundation, along with Freedom of Information Act Reform and Honest Ads Act.
I picked up the transparency baton on OGDA from former Sunlight analyst Matt Rumsey, Sunlight federal policy manager Sean Vitka, OpenGov Foundation founder Seamus Kraft, and Data Coalition founder Hudson Hollister, who drafted the original open data bill, working to make the principle that “public data created with taxpayer dollars should be available to the public in open, machine-readable forms when doing so does not damage privacy or national security” the law.
This is a huge win for public access to public information that every American can and should celebrate today. Special thanks for this victory are due to Christian Hoehner, policy director for the Data Coalition, who did extraordinary yeoman’s work getting this through Congress, Sasha Moss of the R Street Institute, Hollister, Daniel Schuman of Demand Progress, Daniel Castro and Joshua New at the Center for Data Innovation, and Gavin Baker from the American Library Association, some with whom I went to Congress with me to meet with staff over the years and advocated for the bill on and offline.
The passage of this bill won’t mean that the scanned images of spreadsheets that agencies still send in response to FOIA requests will magically go away tomorrow, but journalists, watchdogs and the public can now tell civil servants that they’re now behind the times: open government data is now the default in the USA! Please publish our data on the agency website in a structured format and let the public know.
[Image Credit: Sunlight Foundation]
When I asked whether when or if it is acceptable for the United States government to charge companies, journalists and the public for government data, citing the example of paywalled immigration data, the chief information officer of the United States told me that “it’s part of the commercial equation” and that it was “actually a discussion point for the strategy” in her office in the White House Office of Management and Budget.
“I don’t have a specific answer,” Suzette Kent went on. “That is something that we’re looking at because there’s many tenets of it. There’s some data the government collects & document on behalf of the american public that may have the mode. There’s other types of data, that people are asking for. It’s a broad spectrum and one we are going to continue to explore.” Kent was speaking at the Data Coalition’s Data Demo Day on June 6 in Washington, DC. Video of the keynote speech she gave on data is embedded below:
When asked about the continued disclosure of data in PDFs and non-machine readable forms by federal agencies, despite President Barack Obama’s 2013 executive order, Kent said simply that she advocates compliance with every executive order and law and cited a Cross-Agency Priority goal to remove paper from agency systems.
Charging for public data is not a new topic or debate, but it has continued to be relevant during the Trump administration, when new concerns have grown about government data access, collection, and quality.
As I wrote back in 2014, local, state and national governments across the United States and around the world can and do charge for access to government data.
While some developers in Europe advocate charging for public sector information (PSI) as a way to ensure higher service levels and quality, adding fees does effectively charge the public for public access and has frequently been used as a barrier to press requests:
A city hall, state house or government agency charging the press or general public to access or download data that they have already paid for with their tax revenues, however, remains problematic.
It may make more sense for policy makers to pursue a course where they always make bulk government data available for free to the general public and look to third parties to stand up and maintain high quality APIs, based upon those datasets, with service level agreements for uptime for high-volume commercial customers.
Instead of exploring a well-trodden path, the United States government should follow the money and determine which data is agencies are currently charging for under public records requests or other means, using FOIA demand to drive open data disclosure.