Why I’m joining the Sunlight Foundation

I won’t bury the lede on this story: today is my first day at the Sunlight Foundation as a senior analyst. I’m enormously excited to be joining an organization that’s been at the heart of a global movement towards opening governments to the people they serve with technology, from open source to open data.

If you’ve followed my writing and interests over the past decade, you know that I’m passionate about open government in all of its forms. I’ve been humbled to meet thousands of people around the world who are deeply committed to public service and improving how government functions.

This is a natural fit. From improving public access to information to civic engagement to collaboration around code to participation in democratic governance processes, from regulations to legislation, the Sunlight Foundation has been at the cutting edge of making government more open, effective and accountable.

There’s also a personal reason I made this decision: Jake Brewer, a former Sunlighter and White House staffer who we lost far too early last year, frequently urged me to to make the most of my short time on Earth. This is the right place for me to be.

Long-time readers should expect me to continue writing and participating in this role, creating acts of advocacy journalism in the public interest.

I believe that people have a right to know what is being done in their name by their elected governments. Implicit in that view is the notion that representative democracy is the worst form of government, save for all the rest. It’s up to us to protect and improve the states that we have founded and fought to preserve.

As people who have been paying close attention to Sunlight know, it’s an organization in transition. I’m proud to join up with this open government “restartup”, pitching in where ever my talents are helpful. I believe 2016 is going to be a dynamic year at Sunlight, which is why I’ve thrown in my lot with the extraordinary folks on staff.

I hope that you will continue to send your thoughts, feedback, suggestions, tips and ideas my way in the days and months to come.

Obama Administration Secretly Lobbied Against FOIA Reform In Congress

IMG_1992

A Freedom of Information Act lawsuit showed that the Obama administration vigorously lobbied against Freedom of Information Act reform in Congress.  The documents and correspondence, which were obtained through the Freedom of the Press Foundation’s lawsuit against the Justice Department and reported out by Jason Leopold at Vice Media, showed that the administration was literally lobbying against its own policy becoming law.

The Department of Justice’s six page memorandum shows that the agency opposed Congress making the exact language in Attorney General Eric Holder and President Obama’s 2009 memorandums on FOIA law.

The Justice Department opposing FOIA reform direct conflicts commitments made in the U.S. National Action Plan on Open Government required as part of its participation in  the Open Government Partnership.

I asked Ambassador Power how the United States can be a credible leader on open government if the White House and DoJ does this. In an alternate universe, she and the administration would respond publicly.

Unfortunately, it’s easy to predict the outcome of this news: publicly committing to open government reforms and then undermining them privately will erode abysmal levels of trust in government even more.

In the face of hypocrisy from the Justice Department on this count, the public should  call on their Senators to make the Freedom of Information Act reform legislation the House of Representatives passed in January into law.

With hours of sunshine left, passage of FOIA reform in the U.S. House hangs in the balance

IMG_1992

Update: The House Majority Leader didn’t put S.2520 on Thursday’s legislative calendar (PDF). Per Congress.gov, it was “held at the desk.” We can’t pronounce it dead until 3:30 PM, as the Speaker of the House could bring the bill up by unanimous consent, but FOIA reform in this Congress likely just expired at midnight.

Imagine if an important reform to public access to government information hung in the balance in the United States Congress and the editorial boards of the country’s major newspapers ignored it. Unfortunately, that’s exactly what has happened. Only a few weeks ago, it looked this ‘do nothing Congress’ was actually set to do something: pass much-needed reforms to the Freedom of Information Act. Over the weekend, an unexpected hold in the Senate by Senator Jay Rockefeller put months of bipartisan collaboration in jeopardy. If the U.S. House of Representatives doesn’t schedule a vote tomorrow on the Freedom of Information Improvement Act that passed the Senate on Monday, however, FOIA reform will quietly expire.

Senator Patrick Leahy (D-VT), the chairman of the Senate Judiciary Committee, is urging the House to pass the bill before the 113th Congress ends.

“This legislation is all about government transparency. If House Republicans want this administration to be more accountable, then they must put it on the suspension calendar without delay. Let’s get it done,” Leahy said. “With the sun about to set on this congressional session, the House should not leave this sunshine bill undone, on the table.  Time is quickly running out, and the House must act without further delay.

Representative Darrell Issa (R-CA), the chairman of the House Oversight and Government Reform Committee, and Ranking Member Representative Elijah Cummings (D-MD) have also called on the House to pass the Freedom of Information Act (FOIA) Improvement Act and send it to President Barack Obama after its unanimous passage in the Senate.

“The FOIA Improvement Act will strengthen FOIA, the cornerstone open government law,” Issa and Cummings said, in a joint statement.  “The House unanimously passed companion legislation, H.R. 1211, earlier this year.  The FOIA Improvement Act is a bipartisan bill that, after last night’s passage by the Senate, deserves to be taken up by the House and sent to the President.”

Given that FOIA reform passed the U.S. House unanimously 410-0 in February, why aren’t Speaker of the House John Boehner and Minority Leader Representative Nancy Pelosi bringing S.2520 to a vote? One source tells me that banks have sent lobbyists to the offices of House Financial Services members to oppose the FOIA reform and have told staff there that proprietary regulatory information could be released under the bill. FreedomInfo.org is also reporting that banking lobbyists are opposing the FOIA reform bill. This rumored pressure is in addition to the pressure of the same federal agencies that lobbied against the bill in the Senate.

A similar argument was made in the Senate, and it’s by all accounts a bogus one: Exemption 8 of the FOIA provides protection against such disclosure and the “foreseeable harm” standard embraced by this reform would not result in the release of such regulatory records,  given that there would be a clear foreseeable harm in their release. As FreedomOfInfo.org notes, “the Senate committee report includes lengthy language underscoring the importance of protecting financial information”:

 The paragraph secured support for the bill by the chairman of the Senate Banking Committee, Sen. Tom Johnson (D-SD), sources said. The relevant section of the report begins with a caution: “Extreme care should be taken with respect to disclosure under Exemption 8 which protects matters that are “contained in or related to examination, operating, or condition reports prepared by, on behalf of, or for the use of an agency responsible for the regulation or supervision of financial institutions.” The quote is from the FOIA.

The report language (minus footnotes) continues:

Currently, financial regulators rely on Exemption 8, and other relevant exemptions in Section 552(b), to protect sensitive information received from regulated entities, or prepared in connection with the regulation of such entities, in fulfilling their goals of ensuring safety and soundness of the financial system, compliance with federal consumer financial law, and promoting fair, orderly, and efficient financial markets. Exemption 8 was intended by Congress, and has been interpreted by the courts, to be very broadly construed to ensure the security of financial institutions and to safeguard the relationship between the banks and their supervising agencies. The D.C. Circuit has gone so far as to state that in Exemption 8 Congress has provided “absolute protection regardless of the circumstances underlying the regulatory agency’s receipt or preparation of examination, operating or condition reports.” Nothing in this legislation shall be interpreted to compromise the stability of any financial institution or the financial system, disrupt the operation of financial markets or undermine consumer protection efforts due to the release of confidential information about individuals or information that a financial institution may have, or encourage the release of confidential information about individuals. This legislation is not intended to lessen the protection under Exemption 8 created by Congress and traditionally afforded by the courts.

There’s a lot at stake here, and almost no time on the legislative clock. It is, as Sean Vitka wrote for the Sunlight Foundation wrote today, literally do-or-die time for FOIA reform. It’s crunch time. It’s now or (almost) never: if Speaker of the House John Boehner doesn’t bring the bill up for a vote by unanimous consent today, the process begins again in the next Congress, but without key sponsors of the FOIA reforms in the House and Senate occupying the chairs of committees.

“For all of his talk about the desire of House Republicans to hold the Obama Administration accountable, we are shocked and angered that Speaker Boehner would decide to allow a bill that strengthens and reforms the Freedom of Information Act (FOIA) die without a vote,” said Danielle Brian, chair of OpenTheGovernment.org and executive director of the Project On Government Oversight, in a statement. “S.2520 is a critical bill that strengthens the FOIA watchdog, the Office of Government Information Services, and would force agencies to finally deliver the levels of transparency that the Administration promised on their first day in office in 2009. We call on Speaker Boehner to do the right thing for the American public and call for a vote on S.2520 before the House leaves for the year.”

Update: On Thursday morning, when he was asked about FOIA reform at a press conference, Speaker Boehner said that “I have no knowledge of what the plan is for that bill.”

“We are particularly concerned that Speaker Boehner has now said that he has ‘no knowledge of the plan’ to pass the bipartisan, bicameral FOIA reform bill,” said Brian, in response. “If accountability and making the federal government answer to the public is really a priority for the Republican Caucus, passing this bill should be a priority. The House passed the House companion bill 410 – 0. The Senate passed the bill by unanimous consent after the open government community waged an all-out war against a last second attempt by the Federal Trade Commission (FTC) and other independent agencies that are supposed to be on the public’s side to stop the bill. It’s up to Speaker Boehner to put this bill to a vote and create the levels of open government the public needs.”

If the people’s right to know what their government does in their name matters to you, please let your Member of Congress know that FOIA reform matters to you, and let the Speaker of the House know. You can email the Speaker directly through OpenCongress, call him up at (202) 225-6205 and tweet him @SpeakerBoehner. Even if the press won’t represent itself and the people by asking Congress to support the free flow of government information, you can.

Update: FOIA reform failed to pass in the 113th Congress. As Newsweek reported, in an opaque move, Speaker Boehner tabled the government transparency bill. It was never brought up for a vote in the House. Unless the Speaker reconvenes the House, the FOIA bill is likely dead.

“…the fact that the bill was very close and was tabled because of the influence of lobbyists that found a problem in the legislation that didn’t even exist is frustrating not only for those who wanted the bill to pass but for those who want the American democratic process to be a shining light for the world – not an embarrassment,” wrote Scott A. Hodes, at the FOIA Blog.

“For all of his talk about his desire to hold the Obama Administration accountable, we find it unfathomable that that Speaker John Boehner would allow a bill that strengthens and reforms the Freedom of Information Act (FOIA) to die without a vote,” said Danielle Brian.

In reaction, Senator Leahy made the following statement:

“I am deeply disappointed that last night the House failed to pass the FOIA Improvement Act. This bipartisan bill was reported unanimously by the Senate Judiciary Committee last month, and it was the product of months of hard work by Senator Cornyn and me. Our bill is supported by more than 70 public interest groups that advocate for government transparency and it passed out of the Senate unanimously. I would think that members of the House Republican leadership, who have spent so much time on oversight of the Obama administration, would support the goal of making government more accountable and transparent. But instead of supporting this bill, they have chosen secrecy over sunlight.

“The FOIA Improvement Act would codify what the President laid out in his historic executive order in 2009 by requiring Federal agencies to adopt a ‘Presumption of Openness’ when considering the release of government information under FOIA. This bill would require agencies to find a foreseeable harm if they want to withhold information from the public. Prioritizing the people’s interest in what their government is doing, our bill will reduce the overuse of exemptions to withhold information. Federal agencies have been required to apply this standard since 2009. They also used this same standard during President Clinton’s terms in office. It was only during President George W. Bush’s term of secrecy that this standard was rolled back. It appears the House leadership wants to return to that era. It should not matter who is in the White House, information about what their government is doing belongs to the people.

“In a political climate as divided as this, I had hoped that we could come together in favor of something as fundamental to our democracy as the public’s right to know. That government transparency and openness would not just be the standard applied to the Obama Administration but what is applied to every future administration. The FOIA Improvement Act would have done just that.”

Postscript: Writing for the Sunlight Foundation, Matt Rumsey published a sunny post about the death of FOIA reform.

Sunlight has been strongly supportive of the FOIA Improvement Act because it addresses real world problems faced by requesters every day, specifically targeting overly broad exemptions and limiting unnecessary fees. Just like Sen. Patrick Leahy, D-Vt., one of its strongest champions, we aredisappointed that it did not become law.

And yet, we are hopeful for the future.

Most laws never make it out of committee even after repeated attempts spread over multiple years. The FOIA Improvement Act came tantalizingly close to becoming law its first time around.

Rest assured that the FOIA Improvement Act will be reintroduced in the 114th Congress and that the Sunlight Foundation and its allies will be fighting harder than ever for its passage. We want to say a hearty thank you to Leahy, Sen. John Cornyn, R-Texas, and everyone else that worked so hard during the 113th Congress to make these needed reforms possible. We’ll see you next year!

The Washington Post, to its credit, did a post-mortem on how this popular government transparency bill died in Congress. The reason the FOIA reform stalled in the House may not simply have been lobbying by the financial industry, however, as had been previously reported.

According to House aides, some lawmakers balked at the legislation because several agencies, including the Justice Department, warned that those making information requests would use the “forseeable harm” requirement as the basis for frequent lawsuits.

This detail led Trevor Timm, executive director of the Freedom of the Press Foundation, to argue that it was the Justice Department that secretly tried to stop FOIA reform, despite the text of the legislation being almost word-for-word the poilcy that the agency itself embraced in 2009.

The “foreseeable harm” section referred to by the Post would force federal agencies to justify withholding information if they wished to do so. Essentially, they would have to show the information would cause “foreseeable harm” if released. Not exactly a tall order. But what makes the Justice Department’s objection so shocking is that this “foreseeable harm” provision would not deviate at all from the Justice Department’s own policy. In fact, it was based on it.

In a March 19, 2009 memo to all federal agencies, Attorney General Eric Holder himself wrote that the Justice Department would carry out Obama’s aforementioned transparency order by rescinding the Bush DOJ’s more restrictive FOIA rules and designating new ones. From that moment on, Holder declared:

[T]he Department of Justice will defend a denial of a FOIA request only if (1) the agency reasonably foresees that disclosure would harm an interest protected by one of the statutory exemptions, or (2) disclosure is prohibited by law.

Now read the full text of the provision in the just-killed FOIA reform bill that the Justice Department allegedly objected to:

An agency shall withhold information under this section only if a) the agency reasonably foresees that disclosure would harm an interest protected by an exemption described in subsection or other provision of law; or b) disclosure is prohibited by law.

As you can see, the two passages are virtually identical. How does the Justice Department think this provision will lead to more lawsuits it would have to defend if they’re not supposed to be defending those lawsuits in the first place?

The Justice Department is objecting to making its own supposed policy the law, and confirms what many have long believed: the agency does not want to—or have to—comply with its own FOIA rules.

The DOJ has repeatedly been criticized for failing to enforce, and downright ignoring its own FOIA guidance for years, and their stance on transparency in general has been incredibly hypocritical. For example, Holder has claimed hewanted the torture report to be public as soon as possible, meanwhile fighting in court to prevent the release of any documents on its own torture investigation. Likewise, he’s claimed the Justice Department supports a federal shield law so reporters can protect their sources, while at the same time destroying the already-existing reporter’s privilege in the Fourth Circuit.

Writing for the National Security Archive, Nate Jones looked for lessons from the death of the unanimously supported FOIA bill and decried “Janus-faced support for open government.”  Here was his key takeaway:

Many people –in Congress, in the agencies, in the White House, in the media– proclaim they believe in open government, but don’t really.  To me, that’s the only plausible reason a FOIA bill could garner unanimous approval (thrice in the Senate over the past seven years!) and still die; that’s the only plausible reason agencies whisper that instructions about FOIA currently on the books will ruin the federal government as we know it; that’s the reason for White House silence on the benefits the FOIA Ombuds office not being forced to run its reports though the Department of Justice so they can be “rosified;” that’s the reason the New York Times wins Pulitzers for its FOIA-based reporting, but doesn’t assign a Congressional beat reporter to cover the bill’s death.

How do we overcome these FOIA Januses?  First, we must avoid being stalled out.  We should force Speaker Boehner to act on his pledge that he “look[s] forward to working to resolve this issue [FOIA reform] early in the new Congress.”  FOIA champions Senators Leahy, Cornyn, and Grassley remain in the Senate Judiciary Committee; these senators have an impressive history of defending and working to reform FOIA, no matter which party is in the majorly.  Replacing Representative Issa on the House Oversight Committee is Jason Chaffetz (R-Ut); Democratic FOIA champion Elijah Cummings remains.  Encouragingly, Chaffetz has said he “wants to address the Freedom of Information Act and the difficulties many have in getting the executive branch to comply with FOIA requests.”  Both houses should immediately reintroduce the FOIA bill.  More than 440 members who voted for FOIA reform remain in Congress.

On Saturday, December 20, the New York Times editorial board called for the 114th Congress to revisit the freedom to see government records. In doing so, it made no mention of the reporting on the cause of death by this blog, Vice News, the Washington Post, the Hill, Roll Call or Politico, nor lobbying by banks or federal agencies, nor silence by the White House while most of the press looked the other way.

On its 3rd anniversary, opportunities and challenges for the Open Government Partnership

opgFrance1

In 2010, President Barack Obama spoke to the United Nations General Assembly about open government. “The common thread of progress is the principle that government is accountable to its citizens,” he said, “and the diversity in this room makes clear — no one country has all the answers, but all of us must answer to our own people.”

In all parts of the world, we see the promise of innovation to make government more open and accountable.  And now, we must build on that progress.  And when we gather back here next year, we should bring specific commitments to promote transparency; to fight corruption; to energize civic engagement; to leverage new technologies so that we strengthen the foundations of freedom in our own countries, while living up to the ideals that can light the world.

Open government, said Samantha Power, now the U.S. ambassador to the United Nations, could have a global impact.

In 2011, a historic Open Government Partnership launched in New York City, hailed as a fresh approach to parting the red tape by the Economist. “The partnership is really the first time that there is a multilateral platform to address these issues,” said Maria Otero, former under secretary of state for democracy and global affairs at the United States State Department. “The partnership could have focused on countries come in and present best practices and exchange ideas and then just go home.”

“The partnership is really focused on first having countries participate that have already demonstrated interest in this area and have already put in place a number of specific things and the material laid out, if you will, the minimum standards that are being requested. What the partnership really looks for is to provide a mechanism by which the countries can each develop their own national plans on ways to expand what they’re doing on transparency, accountability, and civic engagement, or to start new initiatives for them. That is really what is very different and important about this partnership, is that it is very action- and results-oriented.”

In 2012, the Open Government Partnership became a player on the world stage as it hosted a global gathering of national leaders and civil society an annual meeting in Brazil, with the responsibilities and challenges that accompany that role, including pushing participants to submit missing action plans and progress reports, not just letters of commitment.

In January 2013, Power hailed the Open Government Partnership (OGP) as President Obama’s signature governance initiative:

It’s not about the abstraction about ‘fighting corruption’ or ‘promoting transparency’ or ‘harnessing innovation’ — it’s about ‘are the kids getting the textbooks they’re supposed to get’ or does transparency provide a window into whether resources are going where they’re supposed to go and, to the degree to which that window exists, are citizens aware and benefiting from the data and that information such that they can hold their governments accountable. And then, does the government care that citizens care that those discrepancies exist?

In May 2013, a seminal event in the evolution of OGP occurred when Russia withdrew from the Open Government Partnership:

If the dominant binary of the 21st century is between open and closed, Russia looks more interested in opting towards more controllable, technocratic options that involve discretionary data releases instead of an independent judiciary or freedom of assembly or the press. One of the challenges of the Open Government Partnership has always been the criteria that a country had to pass to join and then continue to be a member. Russia’s inclusion in OGP instantly raised eyebrows, doubts and fears last April, given rampant corruption in the public sector and Russia’s terrible record on press freedom. “Russia’s withdrawal from the OGP is an important reminder that open government isn’t easy or politically simple,” said Nathaniel Heller, executive director of Global Integrity. “While we don’t yet fully understand why Russia is leaving OGP, it’s safe to assume that the powers that be in the Kremlin decided that it was untenable to give reformers elsewhere in the Russian government the freedom to advance the open government agenda within the bureaucracy.”

In November 2013, the world may have hit ‘peak open‘ at the OGP annual summit in London, despite the partnerships’ members facing default states of closed.

Swirling underneath the professional glitz of an international summit were strong undercurrents of concern about its impact upon governments reluctant to cede power, reveal corruption or risk embarrassment upon disclosure of simple incompetence. The OGP summit took place at a moment where 21st century technology-fueled optimism has splashed up against the foundations of institutions created in the previous century. While the use of the Internet as a platform for collective action has grown, so too have attendent concerns about privacy and surveillance, in the wake of disclosures by NSA contractor Edward Snowden, where the same technologies that accelerated revolutions across the Middle East and North Africa are being used to capture and track the people advocating for change.

In 2014 the Open Government Partnership has matured and expanded, with France joining earlier in the year and Bosnia and Herzegovina bringing the total number of participating countries to 65 out of about 88 eligible countries worldwide. As OGP turns three, the partnership is celebrating the success of its expansion and looking ahead to its future, with a clearer mission and goals and ambitious four year strategy (PDF). The partnership is finally writing letters to countries that are not living up to their commitments, although the consequences for their continued participation if they do not comply remain to be seen.

The challenges and opportunities ahead for a partnership that provides a platform for civil society to hold government accountable are considerable, given the threats to civil society worldwide and the breathtaking changes brought about through technological innovation. Today, 10 national leaders will speak in New York City to mark OGP’s third anniversary. (I’ll be there to listen and share what I can.)

After the speeches end and the presidents and prime ministers return home, serious questions will remain regarding their willingness to put political capitol behind reforms and take tough stands to ensure that their governments actually open up. Digital government is not open government, just as not all open data supports democratic reforms.  As Mexico prepares to become lead co-chair of OGP, one element that didn’t make it into the challenges listed for the country is the state of press freedom in Mexico. As the Committee to Protect Journalists highlighted, open government is not sustainable without a free press. As long as the murders of journalists go unpunished in Mexico, the commitments and efforts of the Mexican national government will have to be taken in context.

Given this blog’s past stance that as press freedom goes, so too does open government, I’ve signed a petition urging the White House to explicitly support a right to report. Every other country that has committed to open government should do the same. Given OGP’s own challenges around the media and open government (PDF), I would also urge the partnership to make sure that press freedom and freedom of expression occupies a prominent place in its advocacy efforts in the years ahead.

Data journalism and the changing landscape for policy making in the age of networked transparency

This morning, I gave a short talk on data journalism and the changing landscape for policy making in the age of networked transparency at the Woodrow Wilson Center in DC, hosted by the Commons Lab.

Video from the event is online at the Wilson Center website. Unfortunately, I found that I didn’t edit my presentation down enough for my allotted time. I made it to slide 84 of 98 in 20 minutes and had to skip the 14 predictions and recommendations section. While many of the themes I describe in those 14 slides came out during the roundtable question and answer period, they’re worth resharing here, in the presentation I’ve embedded below:

[REPORT] On data journalism, democracy, open government and press freedom

On May 30, I gave a keynote talk on my research on the art and science of data journalism at the first Tow Center research conference at Columbia Journalism School in New York City. I’ve embedded the video below:

My presentation is embedded below, if you want to follow along or visit the sites and services I described.

Here’s an observation drawn from an extensive section on open government that should be of interest to readers of this blog:

“Proactive, selective open data initiatives by government focused on services that are not balanced by support for press freedoms and improved access can fairly be criticized as “openwashing” or “fauxpen government.”

Data journalists who are frequently faced with heavily redacted document releases or reams of blurry PDFs are particularly well placed to make those critiques.”

My contribution was only one part of the proceedings for “Quantifying Journalism: Metrics, Data and Computation,” which you can catch up through the Tow Center’s live blog or TechPresident’s coverage of measuring the impact of journalism.

After a false positive, Twitter suspends open government blog for being too social

Twitter’s best practices for tweeting don’t appear to mix well with its rules for tweeting, as I found out last month when the social networking company briefly suspended the Twitter account for this blog. While I was able to quickly get the account back online, the episode raises somr issues regarding how Twitter’s algorithm flags media accounts and some contradictions in the company’s guidance for new users.

When I found that I couldn’t file a help request to Twitter Support to appeal the suspension of @e_pluribusunum_ through that account, I used my main account (@digiphile).

Initially, I thought the suspension was due to spam, similar to the situation David Seaman encountered in 2011.

After I directly contacted Twitter for help, the account went back online later that day:

As I found out days later, however, the suspension was for “sending multiple unsolicited @replies or mentions,” per the statement I have from Twitter Support on @e_pluribusunum_:
“This account was suspended for sending multiple unsolicited @replies or mentions. Twitter monitors the use of these features to make sure they’re not abused. Using either feature to post messages to other users in an unsolicited or egregious manner is considered an abuse of its use, which results in account suspension. You can find more information about @replies and mentions here:https://support.twitter.com/articles/14023-what-are-replies-and-mentions

I have now unsuspended your account. Please note that it may take an hour or so for your follower and following numbers to return to normal. Be sure to review the Twitter Rules, as repeat violations may result in permanent suspension: http://twitter.com/rules”

suspension-notice

The tweets in question, however, are extremely similar to the way I’ve been using Twitter for years, advise others to use Twitter, and that Twitter itself recommends to new users.

Here are the tweets sent the day before the suspension and the three that morning, which I have to assume triggered the suspension.

The seventh tweet, embedded above, had six different @names in it, but it was appropriate: I was attributing the source of the information, referring to an NPR program (The Kojo Nnamdi Show) and naming the 4 guests who were on it. The eighth tweet had three @mentions in it, as I had retweeted a media account that referred to a reporter and added the subject of the story for context.

So: Were there a lot of @mentions? Yep. Were they “unsolicited?” Yep.  That accurately describes tens of thousands of tweets that I’ve sent over the past seven years. In this case, they were far from “abuse.”

That led me to wonder how many people, journalists, government or media companies or nonprofit organizations a Twitter account is allowed to @mention before it’s suspended. Should any of the categories of users I listed now have to actively ask followers for feedback or allow others to talk about them? That doesn’t seem practical nor scalable. Are there different rules for different users, Verified or not? (I’ve asked Twitter for comment on these general questions but have received no answers after two weeks. I will update the post if I do.)

In the meantime, I’ve tried to think them through myself. The “newness” of this account likely tripped Twitter’s automated filter, leading to the suspension. That means that other new users have to think about whether they’re sending “unsolicited replies or mentions” to keep clear.

I found that deeply jarring. I used the @E_Pluribus Unum_ account exactly as I have @digiphile, for over 7 years now, resharing tweets with attributed context and quotes, tweeting about public figures and government officials, tagging mastheads, retweeting select tweets.

That’s more or less how I define being “social” and engaging on the platform. That’s how I thought Twitter defined it, too. Twitter’s own best practices for engaging followers recommends it:

Mention high-profile users
@HillaryClinton included Secretary of State Madeleine Albright username@madeleine in a Tweet welcoming the former Secretary to Twitter. In turn, Albright replied to @HillaryClinton and also mentioned the Kennedy Center (@kencen), where she had recently performed. Including so many mentions of other users makes it more likely that people will find the conversation and join in. ”

If Twitter is suspending new accounts that @mention too many high profile users or reply to them in an “unsolicited” fashion, I can’t help but have serious concerns about Twitter’s future and commitment to being a platform for free expression, government accountability, or hosting civic dialogue.

I do see potential issues with “egregious” @mentions — “@reply spam” has been an issue on Twitter for years — but isn’t that exactly what the block button has been used for, or the new abuse reporting button should be used for? People have been tweeting “#FollowFriday” recommendations for years with many unsolicited @mentions. Are they risking suspension?

Honestly, knocking new accounts offline for being “too social” suggests a tone-deaf algorithm. Ignoring my questions regarding general standards suggests something else. (The company generally refuses to comment on individual accounts.)

Given reports of retention issues and low activity by most users, an overly aggressive approach to filtering new users that are engaging in activity that Twitter itself recommends, particularly media accounts, strikes me as actively self-defeating.

Twitter and its investors should care about the people who never tweet. This experience reminded me that those same parties should care about the people who do tweet and are caught up on algorithmic censorship, followed by vague missives not to talk about other accounts too much.

As I’ve written before, Twitter is not a public utility. It’s a private company with a Terms of Service and Rules it itself sets.  If Twitter’s users don’t like them or lose trust, their option is to stop using the service or complain loudly on other platforms.

In general, Twitter’s record on censorship, Internet freedom and privacy is the best of the big tech companies, as an analysis by the Electronic Frontier Foundation last year highlighted. They’ve gone to bat for their users, from Turkey to Washington. Today, however, I just wish they’d clarify how social those users are allowed to be.

Editor’s Note: The headline of this post has been amended, with “After a false positive” added.

On data journalism, accountability and society in the Second Machine Age

On Monday, I delivered a short talk on data journalism, networked transparency, algorithmic transparency and the public interest at the Data & Society Research Institute’s workshop on the social, cultural & ethical dimensions of “big data”. The forum was convened by the Data & Society Research Institute and hosted at New York University’s Information Law Institute at the White House Office of Science and Technology Policy, as part of an ongoing review on big data and privacy ordered by President Barack Obama.

Video of the talk is below, along with the slides I used. You can view all of the videos from the workshop, along with the public plenary on Monday evening, on YouTube or at the workshop page.

Here’s the presentation, with embedded hyperlinks to the organizations, projects and examples discussed:

For more on the “Second Machine Age” referenced in the title, read the new book by Erik Brynjolfsson and Andrew McAfee.

Federal government agencies receive .91 GPA in FOIA compliance from Center for Effective Government

Today, the Center for Effective Government released a scorecard for access to information from the 15 United States federal government agencies that received the most Freedom of Information Act (FOIA) requests, focusing upon an analysis of their performance in 2013.

The results of the report (PDF) for the agencies weren’t pretty: if you computed a grade point average from this open government report card (and I did) the federal government would receive a D for its performance. 7 agencies outright failed, with the State Department receiving the worst grade (37%).

The grades were based upon:

  1. How well agencies processed FOIA requests, including the rate of disclosure, fullness of information provided, and timeliness of the response
  2. How well the agencies established rules of information access, including the effectiveness of agency polices on withholding information and communications with requestors
  3. Creating user-friendly websites, including features that facilitate the flow of information to citizens, associated online services, and up-to-date reading rooms

The report is released at an interesting historic moment for the United States, with Sunshine Week just around the corner. The United States House of Representatives just unanimously passed a FOIA Reform Act that is substantially modeled upon the Obama administration’s proposals for FOIA reforms, advanced as part of the second National Open Government Action Plan. If the Senate takes up that bill and passes it, it would be one of the most important, substantive achievements in institutionalizing open government beyond this administration.

The Citizens for Responsibility and Ethics in Washington have disputed the accuracy of this scorecard, based upon the high rating for the Department of Justice. CREW counsel Anne Weismann:

It is appropriate and fair to recognize agencies that are fulfilling their obligations under the FOIA. But CEG’s latest report does a huge disservice to all requesters by falsely inflating DOJ’s performance, and ignoring the myriad ways in which that agency — a supposed leader on the FOIA front — ignores, if not flouts, its obligations under the statute.

Last Friday, I spoke with Sean Moulton, the director of open government policy at the Center for Effective Government, about the contents of the report and the state of FOIA in the federal government, from the status quo to what needs to be done. Our interview, lightly edited for content and clarity, follows.

What was the methodology behind the report?

Moulton: Our goal was to keep this very quantifiable, very exact, and to try and lay out some specifics. We thought about what the components were necessary for a successful FOIA program. The processing numbers that come out each year are a very rich area for data. They’re extremely important: if you’re not processing quickly and releasing information, you can’t be successful, regardless of other components.

We did think that there are two other areas that are important. First, online services. Let’s face it, the majority of us live online in a big way. It’s a requirement now for agencies to be living there as well. Then, the rules. They’re explained to the agencies and the public, in how they’re going to do things when they get a request. A lot of the agencies have outdated rules. Their current practices may be different, and they may be doing things that the rules don’t say they have to, but without them, they may stop. Consistent rules are essential for consistent long term performance.

A few months back, we released a report that laid out what we felt were best practices for FOIA regulations. We went through a review of dozens of agencies, in terms of their FOIA regulations, and identified key issues, such as communicating with the requester, how you manage confidential business information, how you handle appeals, and how you handle timelines. Then we found inside existing regulations the best ways this was being handled. It really helped us here, when we got to the rules. We used that as our roadmap. We knew agencies were already doing these things, and making that commitment. The main thing we measured under the rules were the items from that best practices report that were common already. If things were universal, we didn’t want to call a best practice, but a normal practice.

Is FOIA compliance better under the Obama administration, more than 4 years after the Open Government Directive?

Moulton: In general, I think FOIA is improving in this administration. Certainly, the administration itself is investing a great deal of energy and resources in trying to make greater improvements in FOIA, but it’s challenging. None of this has penetrated into national security issues.

I think it’s more of a challenge than the administration thought it would be. It’s different from other things, like open data or better websites. The FOIA process has become entrenched. The biggest open government wins were in areas where they were breaking new ground. There wasn’t a culture or way of doing this or problems that were inherited. They were building from the beginning. With FOIA, there was a long history. Some agencies may see FOIA as some sort of burden, and not part of their mission. They may think of it as a distraction from their mission, in fact. When the Department of Transportation puts out information, it usually gets used in the service of their mission. Many agencies haven’t internalized that.

There’s also the issue of backlogs, bureaucracy, lack of technology or technology that doesn’t work that well — but they’re locked into it.

What about redaction issues? Can you be FOIA compliant without actually honoring the intent of the request?

Moulton: We’re very aware of this as well. The data is just not there to evaluate that. We wish it was. The most you get right now is “fully granted” or “partly granted.” That’s incredibly vague. You can redact 99% or 1% and claim it’s partially redacted, either way. We have no indicator and no data on how much is being released. It’s frustrating, because something like that would help us get a better sense on whether agencies would benefit would new policies

We do know that the percentage of full grants has dropped every year, for 12 years, from the Clinton administration all the way through the Bush administration to today. It’s such a gray area. It’s hard to say whether it’s a terrible thing or a modest change.

Has the Obama administration’s focus on open government made any difference?

Moulton: I think it has. There were a couple of agencies that got together on FOIA reform. The EPA led the team, with the U.S. National Archives and the Commerce Department, to build a new FOIA tool. The outward-facing part of the tool enables a user to go to a single spot, request and track it. Other people could come and search FOIA’ed documents. Behind the scenes, federal workers could use the tool to forward requests back and forth. This fits into what the administration has been trying to do, using technology better in government

Another example, again at the EPA, is where they’ve put together a proactive disclosure website. They got a lot of requests, like if there are inquiries about properties, environmental history, like leaks and spills, and set up a site where you could look up real estate. They did this because they went to FOIA requests and see what people wanted. That has cut down their requests to a certain percentage.

Has there been increasing FOIA demand in recent years, affecting compliance?

Moulton: I do think FOIA requests have been increasing. We’ll see what this next year of data shows. We have seen a pretty significant increase, after a significant decrease in the Bush administration. That may be because this administration keeps speaking about open government, which leads to more hopeful requestors. We fully expect that in 2013, there will be more requests than the prior year.

DHS gets the biggest number of all, but that’s not surprising when we look at the size of it. It’s second biggest agency, after Defense, and the biggest domestic facing agency. when you start talking about things like immigration and FEMA, which go deep into communities and people’s lives, in ways that have a lot impact, that makes sense.

What about the Department of Justice’s record?

Moulton: Well, DoJ got the second highest rating, but we know they have a mixed record. There are things you can’t measure and quantify, in terms of culture and attitude. I do know there were concerns about the online portal, in terms of the turf war between agencies. There were concerns about whether the tech was flexible, in terms of meeting all agency needs. If you want to build a government-wide tool, it needs to have real flexibility. The portal changed the dialogue entirely

Is FOIA performance a sufficient metric to analyze any administration’s performance on open government?

Moulton: We should step back further and look at the broader picture, if we’re going to talk about open government. This administration has done things, outside of FOIA, to try to open up records and data. They’ve built better online tools for people to get information. You have to consider all of those things.

Does that include efforts like the Intelligence Community Tumblr?

Moulton: That’s a good example. One thing this administration did early on is to identify social media outlets. We should be going there. We can’t make citizens come to us. We should go to where people are. The administration pushed early on that agencies should be able to use Tumblr and Twitter and Facebook and Flickr and so on.

Is this social media use “propaganda,” as some members of the media have suggested?

Moulton: That’s really hard to decide. I think it can result in that. It has the potential to be misused to sidestep the media, and not have good interaction with the media, which is another important outlet. People get a lot of their information from the media. Government needs to have good relationship.

I don’t think that’s the intention, though, just as under Clinton, when they started setting up websites for the first time. That’s what the Internet is for: sharing information. That’s what social media can be used for, so let’s use what’s there.