New “State of Open Data” book captures a global zeitgeist around public access to information and use

For a decade, I’ve tracked the contours of open data, digital journalism, open source software and open government, publishing research on the art and science of data journalism that explored and tied together those threads.

This week, I’m proud to announce that a new book chapter on open data, journalists, and the media that I co-authored with data journalism advisor Eva Constantaras has been published online as part of “The State of Open Data: Histories and Horizons!

We joined 43 other authors in a 18-month project that reflected on “10 years of community action and review the capacity of open data to address social and economic challenges across a variety of sectors, regions, and communities.”

The publication and the printed review copy I now possess is the end of a long road.

Back in January 2018, I asked a global community of data journalists, open government advocates, watchdogs and the public for help documenting the “State of Open Data” and journalism for a new project for Open Data for Development. While I was at the Sunlight Foundation, I seeded the initial network scan on journalists, media and open data.

Over the past year, people and organizations from around the world weighed in – & Eva Constantaras joined me as co-editor & lead author, creating a “network scan” of the space. The writing project traveled with me, after I left Sunlight, and over this winter we edited and synthesized the scan into a polished chapter.

The book was originally going to be introduced at the 5th International Open Government Data Conference in Buenos Aires, in the fall of 2018, but will instead be officially launched at the Open Government Partnership’s global summit in Ottawa, Canada at the end of May 2019.

The book, which was funded by the International Research Centre and supported by the Open Data for Development (OD4D) Network. As a result of that support, “The State of Open Data” was also published in print by African Minds, from whom it may also be purchased. (While I was paid to edit and contribute, I don’t receive any residuals.)

Last night, I joined Tim Davies and other authors at the OpenGov Hub in DC to talk about the book & our chapters. Video of Davies, who managed to cover a tremendous amount of ground in ten minutes, is embedded below.

Thank you to everyone who contributed, commented and collaborated in this project, which will inform the public, press and governments around the world.

 

Sunshine and comity featured at US House Modernization Committee hearing

On May 10, the Select Committee on the Modernization of Congress in the United States House of Representatives held a hearing on “opening up the process,” at which 4 different experts talked with Congress about making legislative information more transparent,” from ongoing efforts to proposed reforms to the effect of sunshine laws passed decades ago.

If you’re not up to speed on this committee, it was established on January 4, 2019 the House voted in favor of establishing the Select Committee by an overwhelming margin (418-12) by adopting of Title II of H.Res.6, the Rules of the House of Representatives for the One Hundred Sixteenth Congress, with a sole authority of investigating, studying, holding public hearings, making finding, and developing recommendations to modernize Congress – but no legislative jurisdiction nor ability to take legislative action. The committee has has been fairly described by IssueOne as “the best opportunity in decades” for Congress to improve itself, by looking inward.

The two hours of discussion on May 10 mostly added up to good news for good government, with useful summaries of progress opening up Congress to date from the deputy clerk of the House, proposed recommendations and reforms from GovTrack founder Josh Tauberer and DemandProgress policy director Daniel Schuman that would build on that progress, and some skepticism of sunshine in the legislative process from University of Maryland professor Frances Lee.

I attended the hearing and tweeted from it, as has been my practice for nearly a decade in DC:

The thread of tweets above, however is not meant to be comprehensive, nor could it be fully contextualized in the moment. For that, watch the hearing on YouTube, in the video embedded below:

…and read a summary of the hearing from the Congressional Institute, reporting from Federal Computer Week on “transparency through technology” and Roll Call on ongoing development of an “artificial intelligence engine” (applying machine-learning to structured legislative data), and the latest edition of Demand Progress’ First Branch Forecast, in which Schuman summarizes his testimony and aspects of the hearing.

He and Tauberer recommended from appointing a legislative branch Chief Data Officer, releasing structured data that would feed into the clerk’s tool to show how proposed amendments would change bills, and enact a mandate and open standards for a unique identifier for lobbyists across the U.S. government. Whether those ideas make it into the committee’s recommendations remains to be seen, but they’re worth weighing – along with further studying of the value or risk of increasing or decreasing public access to various aspects of the deliberative processes that constitute legislative and oversight activities.

On a meta note, the process on display at this forum was notable for comity between witnesses and members, openness to the public and press, and engagement online.

The medium is still the message, when it comes showing (not telling) how open a given institution is.

While paying attention to the digital component has become downright mundane in 2019, the Committee demonstrated admirable competence, streaming the hearing online at YouTube, publicized it on social media prior to the event, engaged with the public during the hearing, and published testimony on its website afterwards. (Unfortunately, there’s no text transcription of the hearing on the hearing page. Given the committee’s acknowledgement of the importance of accessibility, it should make sure to get transcripts online.)

As at the most recent “Congressional hackathons,” Members of Congress were able to show good government, ethics and transparency, far away from partisan rancor over white hot political issues or Congressional attempts to conduct constitutional oversight of a corrupt administration.

If you’re interested in following the activities of the committee or providing feedback, visit Modernizecongress.House.gov and scrub in: the People’s House will only be as open, accessible, accountable, and effective as we make it – or demand it to be. There’s no one else coming to help.

US House hearing on transparency misses the open government forest for the FOIA trees

When sunshine is applied to government, what’s revealed can be determined by the eye of the beholder. After the 115th Congress neglected oversight of open government, the U.S. House Oversight Committee held a hearing on the Freedom of Information Act … Continue reading

US government officials and Congressmen praise open government during Sunshine Week in DC

Celebrating Sunshine Week is off to a good start in the nation’s capitol, but not without some shadows along the way.

So far, the public has heard robust defenses of the role of access to information and journalists in our democracy on a national Freedom of Information Day conference at the Newseum, and a forum on open government at the National Archives that included reflections from all three branches. I attended both events in person. I didn’t go over to the Justice Department, where Principal Deputy Associate Attorney General Jesse Panuccio gave a speech at the agency’s Sunshine Week kick-off, but I did hear that he criticized unnamed groups whose requests are “straining the system.”

“Some groups have turned FOIA into a means of generating attorneys’ fees or of attempting to shut down policymaking,” he said. ”Immediate litigation has become a feature of FOIA administration rather than a last resort, and the result is often that large and complex requests by institutional actors are moved, by court order, ahead of requests by average citizens.”

Panuccio said that he sent a a memorandum to all agency General Counsels and Chief FOIA Officers stressing that “improving FOIA performance requires the active participation of agency Chief FOIA Officers.” The Department of Justice has not disclosed the memorandum yet.

Here’s else what I saw, heard, and learned so far on Sunshine Week in 2019.

Freedom of Information Day 2019

Embedded above is a Twitter thread on Freedom of Information Day thread on Freedom of Information Day, which included an impassioned speech celebrating the press by House Oversight Committee Chairman Elijah Cummings and encouragement to journalists.

In an age where far too many politicians are attacking the press, it was a powerful benediction from a preacher’s son. The following text is a rough paraphrase of Cummings’ remarks, not an official transcript. You can watch video of the entire event, including his speech, on CSPAN, which is in of itself a great example of government transparency.

200 years from now, people will be reading about this period in history.

They will ask: when you saw the press being suppressed, a White House that blocked info from getting to people’s reps, a President who sent memos to career employees telling them they could not be whistleblowers, they’re going to ask what did you do? What did you do? Did you stand on the sidelines? Did you say it was someone else’s business? Did you write the story? Did you take a moment to write the editorial? Did you become fearful?

I view an attack on our press as an attack on all of us, and I will fight with everything I’ve got. And I want you to understand that I’ve got your back. I am the son of 2 Pentecostal ministers. In my house, a lie was a lie and the truth was the truth. We cannot allow fake information to become the norm. And you are the guardians of our freedom of the press. You are the ones who are in guerrilla warfare trying to get info out.

That’s why it’s so critical for Congress to protect your rights and people throughout the nation to get information through FOIA. People want to hold Trump accountable, but how can you hold him accountable if you don’t have information? If you have a secret meeting with Putin & no one knows about it?

Come on now: We’re better than that. If you block information, it’s impossible to have accountability. FOIA is critical to help the American people understand the decisions being made by their government. It is also crucial to understanding who is making those decisions and how they will affect their daily lives.

One of our top priorities this Congress is to investigate agency compliance with FOIA & evaluate how we can improve the law. “Never mistake a comma for a period.”

Things can always be improved upon. I hope Republicans will work with us in a bipartisan manner. It took us 3 years of hard work, with the help of many, & I thank you, & negotiation, & then with the help of Senator John Cornyn and Senator Leahy, we got it over the finish line. The FOIA Improvement Act is a prime example of how Congress can work together.

This week’s Freedom of Information hearing will have the EPA, Department of the Interior and The Justice Department. I invited The Justice Department because they’re in charge of ensuring compliance with the FOIA. In my opinion, they must do a much, much, much better job. At EPA, FOIA requests sent through political appointees and “Deep 6’ed.” Certain FOIA requests were deliberately delayed. That’s why Congress needs to conduct oversight, to shed a light. Secretary Ross will be coming before House Oversight to testify about how citizenship question was added to the 2020 Census.

As Oversight Chairman, my immediate responsibility is to do this kind of oversight. That work helps us to develop reforms for the future. You are so important. You are more important than you know.

I beg you to tear down any walls that might block you from getting info to the American people that they need to know. Stand up for strong FOIA law. Work with us. Don’t be silent.

I want our grandchildren to know that we stood up for this democracy. That we, all of us, had great respect for the people who created the Constitution. That we decided to be about freedom of the press & getting the information out. We are at a critical moment in our country’s history. I am so glad that you have been called to this moment to be the guardians of our information and the flow of our information.

Sunshine Week at the National Archives

Embedded below is a tweeted thread from the National Archives Sunshine Week event, including Judge Howell, discussion with federal FOIA ombudsman staff, and a livetweet of Senator Patrick Leahy’s comments. Video of the entire event is embedded above.

The National Archives subsequently wrote about celebrating Sunshine Week on its website.

Leahy, who subsequently tweeted out a transcript of his remarks, observed that ”the list of threats to transparency under the Trump administration goes on, and expands far beyond just FOIA itself.”

“President Trump’s ongoing, opaque ties to his business organization make it impossible to know whether foreign governments and corporations are able to curry favor with him by spending money on his business,” the senator said. “The Trump administration has also issued an unprecedented number of lobbyist waivers to its appointees in secret – preventing the public from knowing whether Trump agency officials are simply continuing their advocacy on behalf of special interests in their official capacities.”

Senator Cornyn, a former judge and Texas attorney general, also spoke to the value of open government but not posted a transcript yet. He did tweet about his visit, stating that “Government transparency is the cornerstone of democracy. I’m proud to have passed the #FOIA Improvement Act and will continue to push for policies that ensure our government remains accountable to we the people.

As I tweeted from the event, I learned the FOIA was amended in 1974 to include sanctions against individual federal agency employees, as noted in this 2013 paper on enforcing the public’s right to access information, after Ralph Nader suggested it:

I also confirmed that the Justice Department hasn’t ever sanctioned any U.S. government officials or staff for violating FOIA under the amended statute. The fact that sanctions for violating government access laws are almost never applied at any level of government might shed some sunlight on why the state of compliance with public records laws not just in the US government but around the USA.

If FOIA officers and appointees faced fines or even jail time for obstructing disclosures under the law, we might see more of a normative shift towards the “openness by default” described in the FOIA statute. Unfortunately, the Department of Justice’s Office of Information Policy does not show any inclination to do more than encourage agencies to comply and praise those that do.

Sunshine Week in the People’s House

Earlier today, the Oversight Committee released a fact sheet that documents various aspects of the Trump administration record of secrecy.

Tomorrow morning, the public will see what Sunshine Week looks like in the House of Representatives, when the Oversight Committee holds a hearing on FOIA and transparency, followed by a  hearing with Commerce Secretary Wilbur Ross on Thursday “to determine why he and other Trump Administration officials gave misleading testimony about how the citizenship question was added to the Census, which has been deemed ‘unconstitutional and a violation of federal statute‘ by two federal judges.”

Whether more sunshine disinfects secrecy and corruption remains to be seen, but more public information about how public power is being wielded will inform both the public and their representatives in Congress. On Sunshine Week, that’s a good start.atch

Shadows and celebrations for Sunshine Week 2019

swlogo2-1024x601.jpg

Sunshine Week 2019 may officially begin on March 10, but it’s unofficially kicking off tomorrow with the National Freedom of Information Day Conference at the Newseum in DC.

For those unfamiliar with this annual celebration of open government around the United States, Sunshine Week was founded over a decade ago by the American Society of Newspaper Editors, who now support it with the Reporters Committee for Freedom of the Press.

Sunshine Week always falls around President James Madison’s birthday on March 16. Madison is generally regarded as the forefather of open government in the United States, as evidenced by this memorable letter in 1822:

madison letter

“Knowledge will forever govern ignorance: And a people who mean to be their own Governors, must arm themselves with the power which knowledge gives,” he wrote.

In 2018, the “information darkness” of the Trump administration led to ignominy, from the Electronic Frontier Foundation’ “Foilies” to the Society for Professional Journalists’  Black Hole Award to the National Security Archives’ Rosemary Awards.

In 2019, transparency and accountability have taken on additional context during the Trump administration, which has continued to be allergic to transparency, rife with conflicts of interest, and hostile to the essential role journalism plays in a democracy. When the President of the United States repeatedly calls journalists “the enemy of the people,” a disinformation virus is weakening our body politic.

Despite the enactment of a historic open government data bill, the state of open government (data) remains divided, at risk, and underfunded in the United States.

After years of delays and democratic regression, the US government released a weak open government plan for the Open Government Partnership that was not responsive to the demands of this moment. The Open Government Partnership’s researchers found backsliding in the USA

Representative Elijah Cummings, D-Md, and Senators Chuck Grassley, R-Iowa, Patrick Leahy, D-Vt., and John Cornyn, R-Texas, recently sent a stern letter to the Interior Department reprimanding the agency for its efforts to weaken its FOIA regulations and urging it to reconsider the rule change. The proposed rule changes which garnered more than 65,000 comments – include allowing the DOI to preemptively reject what it defines as “unreasonably burdensome” requests, the possibility of imposing a monthly limit to the number of either pages or requests from a single requester the agency will process, and a host of other changes that may make it more difficult to obtain fee waivers and expedited processing.

The bipartisan group of lawmakers, all seasoned FOIA champions, told the Interior Department, “We write to express significant concern with the rule recently proposed by the Department of the Interior (DOI) concerning its Freedom of Information Act (FOIA) procedures. The proposed rule appears to restrict public access to DOI’s records and delay the processing of FOIA requests in violation of the letter and spirit of FOIA. The American people have the right to access information from DOI, and the proposed rule needlessly encroaches on that right.”

The context for oversight of open government at the national level for this year, however, is different. 2018 midterm elections delivered a 116th Congress that brought with it a commitment to oversight that was sorely lacking in the last session. A core element of that oversight has been public hearings that hold public officials and corporate executives accountable for their service or services.

That will continue next week, when, as has been the tradition in past years, the U.S. House Oversight Committee will be holding a hearing during Sunshine Week in 2019, considering the Freedom of Information Act and transparency under the Trump administration.

There will be many other Sunshine Week events around DC as well. The Department of Justice’s awards for FOIA officers. The National Archives celebrating the 10th anniversary of the Office of Government Information Services (aka the federal FOIA ombudman) during the day on March 11. That night, there will be a panel discussion on how to obtain and improve coverage of climate data at the National Press Club hosted by its Journalism Institute and Freedom of the Press Committee.

There will be a DC Open Government Summit and a forum on science in the Trump era put on by the Government Accountability Project, the Union of Concerned Scientists, and the Sunlight Foundation’s Web Integrity Project on the evening of March 12.

Despite the threats to American democracy at the federal level and ongoing challenges to open government in the states and cities, there’s still much to celebrate in 2019.

The free press, independent judiciary, and watchdog organizations have continued to provide transparency when elected officials and civil servants have tried to make decisions in secrecy, shedding light on corruption, fraud, waste, abuse and incompetence.

Their combined efforts to bring in sunshine in government across American civil society have been a bulwark against tyranny and corruption in the United States and around the world. Thank you to everyone who continues to support, defend and extend the public’s right to know in the 21st century.

Open Government Partnership IRM finds backsliding in the USA

This White House’s decision to continue U.S. government participation in the Open Government Partnership was far from certain, given the demonstrated distaste of the Trump administration for international agreements and institutions. In that context, The Trump administration’s commitment to participating … Continue reading

After years of delays and democratic regression, USA releases weak open government plan

If the American public wants to see meaningful progress on transparency, accountability or ethics in U.S. government, it should call on Congress to act, not the Trump White House. With little fanfare or notice, the United States of America has … Continue reading

The state of open government (data) remains divided, at risk, and underfunded

What’s next for open data in the United States? That was the open question posed at the Center for Data Innovation (CDI) last week, where a panel of industry analysts and experts gathered to discuss the historic open government data … Continue reading

Open government data complements FOI laws, but it cannot replace them

In March 2018, three public policy scholars posted a provocative question: could the open government movement shut the door on freedom of information? At the time, I let it flow without refuting it from the Sunlight Foundation’s platform, but it’s … Continue reading

Congress votes to make open government data the default in the United States

On December 21, 2018, the United States House of Representatives voted to enact H.R. 4174, the Foundations for Evidence-Based Policymaking Act of 2017, in a historic win for open government in the United States of America.

The Open, Public, Electronic, and Necessary Government Data Act (AKA the OPEN Government Data Act) is about to become law as a result. This codifies two canonical principles for democracy in the 21st century:

  1. public information should be open by default to the public in a machine-readable format, where such publication doesn’t harm privacy or security
  2. federal agencies should use evidence when they make public policy

For the full backstory on what’s in the bill and how it came to pass, read yesterday’s feature.

It’s worth noting that last minute objection did result in two amendments that the Senate had to act upon. Thankfully, on Saturday, December 22nd, the Senate acted, passing the resolution required to send the bill onwards to the president’s desk.

Here’s what changed: First, the text of Title I was amended so that it only applied to CFO Act agencies, not the Federal Reserve or smaller agencies. Title II (the Open Government Act) still applies to all federal agencies.

Second, there was a carve out in Title I “for data that does not concern monetary policy,” which relates to the Federal Reserve, among others.

 

While the shift weakened the first title of the bill a bit, this was still a historic moment: Congress has passed a law to make open data part of of the US Code.

While the United States is not the first or even the second democracy to pass an open data law –  France and  Germany have that distinction – this is a welcome advance, codifying the open government data policies, practices, roles and websites (looking at you, Data.gov) that the federal government had adopted over the past decade.

Open government activists, advocates and champions continue to celebrate, online and off.

https://platform.twitter.com/widgets.js

https://platform.twitter.com/widgets.js

https://platform.twitter.com/widgets.js

https://platform.twitter.com/widgets.js

“The bipartisan passage of the Foundations for Evidence-Based Policymaking Act is a significant step toward a more efficient, more effective government that uses evidence and data to improve results for the American people,” said Michele Jolin, CEO and co-founder of Results for America, in a statement. “We commend Speaker Ryan, Senator Murray and their bipartisan colleagues in both chambers for advancing legislation that will help build evidence about the federally-funded practices, policies and programs that deliver the best outcomes. By ensuring that each federal agency has an evaluation officer, an evaluation policy and evidence-building plans, we can maximize the impact of public investments.”

“The OPEN Government Data Act will ensure that the federal government releases valuable data sets, follows best practices in data management, and commits to making data available to the public in a non-proprietary and electronic format,” said Daniel Castro, in a statement. “Today’s vote marks a major bipartisan victory for open data. This legislation will generate substantial returns for the public and private sectors alike in the years to come.”

“The passage of the OPEN Government Data Act is a win for the open data community”, said Sarah Joy Hays, Acting Executive Director of the Data Coalition, in a statement. “The Data Coalition has proudly supported this legislation for over three years, along with dozens of other organizations. The bill sets a presumption that all government information should be open data by default: machine-readable and freely-reusable. Ultimately, it will improve the way our government runs and serves its citizens. This would not have been possible without the support of Speaker Paul Ryan (WI-1-R), Senators Patty Murray (WA-D), Brian Schatz (HI-D), Ben Sasse (NE-R), and Rep. Derek Kilmer (WA-6-D). Our Coalition urges the President to promptly sign this open data bill into law.”

Congratulations to everyone who has pushed for this outcome for years.

[Image Credit: Sunlight Foundation]

This post has been updated, and corrected: France was ahead of Germany in enacting an open data law.