The VA is being neither open nor transparent about its missing open government plans or policies. On March 29, 2018, I made a Freedom of Information Act request to the United States Department of Veterans Affairs (VA) in which I … Continue reading
researchers from the Open Government Partnership’s Independent Reporting Mechanism (IRM) shared a new end of term report that detailed both progress and regression in meeting the commitments in the third United States National Action Plan for Open Government between October 2015 and May 2017. To be charitable, the researchers found a mixed record on open government during that time period, with poor public engagement, limited government feedback, and lack of civil society setting the agenda or participating in an iterative dialog with government. Continue reading
Imagine searching Facebook, Google or Twitter for the status of a bill before Congress and getting an instant result. That future is now here, but it’s not evenly implemented yet.
When the Library of Congress launched Congress.gov in 2012, they failed to release the data behind it. Yesterday, that changed when the United States Congress started releasing data online about the status of bills.
For the open government advocates, activists and civic hackers that have been working for over a decade for this moment, seeing Congress turn on the data tap was a historic shift.
It took 14 years 9 months 6 days after I asked: Congress is now publishing actual data on the status of legislation. https://t.co/ITtDev12Xs
— Joshua Tauberer (@JoshData) February 24, 2016
//platform.twitter.com/widgets.js
Congressional leaders from both sides of the aisle applauded the release of House and Senate bill status information by the U.S. Government Printing Office and Library of Congress.
“Today’s release of bill status information via bulk download is a watershed moment for Congressional transparency,” said House Majority Leader Kevin McCarthy (R-CA), in a statement. “By modernizing our approach to government and increasing public access to information, we can begin to repair the relationship between the people and their democratic institutions. The entire Congressional community applauds the dedication of the Legislative Branch Bulk Data Task Force, the Office of the Clerk, the House Appropriations Committee, GPO, and the Library of Congress, which worked together to make this progress possible.”
“Building off previous releases of bills and summaries, today’s release of bill status information largely completes the overarching goal of providing bulk access to all the legislative data that traditionally has been housed on Thomas.gov and now also resides on Congress.gov,” said Democratic Whip Steny Hoyer (D-MD). “This is a major accomplishment that has been many years in the making. It goes a long way toward making Congress more transparent and accessible to innovation through third party apps and systems. I applaud the dedicated civil servants who made this possible at the Legislative Branch service agencies, and I want to thank the Bulk Data Task Force for their leadership in this effort. While this largely completes a major goal of the Task Force, I look forward to continuing to workwith them to further modernize the U.S. Congress.”
The impact of open government data releases depend upon publicy and political agency. Releasing the states of bills before Congress in a way that can be baked in by third party apps and services is a critical, laudable step in that direction, but much more remains to be done in making the data more open and putting it to use and re-use. If the Library of Congress opens up an application programming interface for the data that supplies both Congress.gov and the public, it would help to reduce the asynchrony of legislative information between the public and elites who can afford to pay for Politico’s Legislative Compass or Quorum Analytics that is the status quo today.
In an era when Congress job approval ratings and trust in government are at historic lows, the shift didn’t make news beyond the Beltway. Govtrack.us, which is based upon data scraped from the Library of Congress, has been online for years. Until this XML data is used by media and technology companies in ways that provide the public with more understanding of what Congress is doing on their behalf and give them more influence in that legislative process, that’s unlikely to change quickly.
This is the week for seeking feedback on open government in the United States. 4 days ago, the White House published a collaborative online document that digitized the notes from an open government workshop held during Sunshine Week in March. Today, Abby Paulson from OpenTheGovernment.org uploaded a final draft of a Model National Action Plan to the Internet, as a .doc. I’ve uploaded it to Scribd and embedded it below for easy browsing. Nelson shared the document over email with people who contributed to the online draft.
Thank you so much for contributing to the civil society model National Action Plan. The Plan has made its way from Google Site to Word doc (attached)! We will share these recommendations with the White House, and I encourage you to share your commitments with any government contacts you have. If you notice any errors made in the transition from web to document, please let me know. If there are any other organizations that should be named as contributors, we will certainly add them as well. The White House’s consultation for their plan will continue throughout the summer, so there are still opportunities to weigh in. Additional recommendations on surveillance transparency and beneficial ownership are in development. We will work to secure meetings with the relevant agencies and officials to discuss these recommendations and make a push for their inclusion in the official government plan. So, expect to hear from us in the coming weeks!
In a followup post, the White House shared a link to a collaborative online document where the notes from an open government workshop held during Sunlight Week were posted online for comment. In doing so, they moves from sticky notes to a wiki.
This is how @opengovpart National Action Plans get made pic.twitter.com/3d9gvmFUOs
— Nathaniel Heller (@Integrilicious) March 17, 2015
What will come of asking the broader public for feedback on the ideas that a workshop of advocates and policy wonks in DC suggested? Stay tuned.
Yesterday, I wrote up 15 key insights from the Pew Internet and Life Project’s new research on the American public’s attitude towards open data and open government. If you missed it, what people think about government data and the potential impact of releasing it is heavily influenced by the prevailing low trust in government and their politics.
Media coverage of the survey reflected the skepticism of the reporters (“Most Americans don’t think government transparency matters a damn“) or of the public (“Who cares about open data” and “Americans not impressed by open government initiatives“). This photo by Pete Souza below might be an apt image for this feeling:
Other stories pulled out individual elements of the research (“Open data on criminals and teachers is a-okay, say most US citizens” or mixed results (“People Like U.S. Open Data Initiatives, But Think Government Could Do More” and “Sorry, open data: Americans just aren’t that into you“) or general doubts about an unfamiliar topic (“Many Americans Doubt Government Open Data Efforts“). At least one editor’s headline suggested that the results were an indictment of everything government does online: (“Americans view government’s online services and public data sharing as a resounding ‘meh’.) Meh, indeed.
As usual, keep a salt shaker handy as you browse the headlines and read the original source. The research itself is more nuanced than those headlines suggest, as my interview with the lead researcher on the survey, John Horrigan, hopefully made clear.
Over at TechPresident, editor-in-chief Micah Sifry saw a glass half full:
- Digging deeper into the Pew report, it’s interesting to find that beyond the “ardent optimists” (17% of adults) who embrace the benefit of open government data and use it often, and the “committed cynics” (20%) who use online government resources but think they aren’t improving government performance much, there’s a big group of “buoyant bystanders” (27%) who like the idea that open data can improve government’s performance but themselves aren’t using the internet much to engage with government. (Heads up Kate Krontiris, who’s been studying the “interested bystander.”)
- It’s not clear how much of the bystander problem is also an access problem. According to a different new analysis done by the Pew Research Center, about five million American households with school-age children–nearly one in five–do not have high-speed internet access at home. This “broadband gap” is worst among households with incomes under $50,000 a year.
Reaction from foundations that have advocated, funded or otherwise supported open government data efforts went deeper. Writing for the Sunlight Foundation, communications director Gabriela Schneider saw the results from the survey in a rosy (sun)light, seeing public optimism about open government and open data.
People are optimistic that open data initiatives can make government more accountable. But, many surveyed by Pew are less sure open data will improve government performance. Relatedly, Americans have not quite engaged very deeply with government data to monitor performance, so it remains to be seen if changes in engagement will affect public attitudes.
That’s something we at Sunlight hope to positively affect, particularly as we make new inroads in setting new standards for how the federal government discloses its work online. And as Americans shift their attention away from Congress and more toward their own backyards, we know our newly expanded work as part of the What Works Cities initiative will better engage the public, make government more effective and improve people’s lives.
Jonathan Sotsky, director of strategy and assessment for the Knight Foundation, saw a trust conundrum for government in the results:
Undoubtedly, a greater focus is needed on explaining to the public how increasing the accessibility and utility of government data can drive accountability, improve government service delivery and even provide the grist for new startup businesses. The short-term conundrum government data initiatives face is that while they ultimately seek to increase government trustworthiness, they may struggle to gain structure because the present lack of trust in government undermines their perceived impact.
Steven Clift, the founder of e-democracy.org, views this survey as a wakeup call for open data advocates.
One reason I love services like CityGram, GovDelivery, etc. is that they deliver government information (often in a timely way) to the public based on their preferences/subscriptions. As someone who worked in “e-government” for the State of Minnesota, I think most people just want the “information” that matters to them and the public has no particular attachment to the idea of “open data” allowing third parties to innovate or make this data available. I view this survey as a huge wake up call to #opengov advocates on the #opendata side that the field needs to provide far more useful stuff to the general public and care a lot more about outreach and marketing to reach people with the good stuff already available.
Mark Headd, former chief data officer for the City of Philadelphia and current developer evangelist for Accela software, saw the results as a huge opportunity to win hearts and minds:
The modern open data and civic hacking movements were largely born out of the experience of cities. Washington DC, New York City and Chicago were among the first governments to actively recruit outside software developers to build solutions on top of their open data. And the first governments to partner with Code for America – and the majority over the life of the organization’s history – have been cities.
How do school closings impact individual neighborhoods? How do construction permit approvals change the character of communities? How is green space distributed across neighborhoods in a city? Where are vacant properties in a neighborhood – who owns them and are there opportunities for reuse?
These are all the kinds of questions we need people living and working in neighborhoods to help us answer. And we need more open data from local governments to do this.
If you see other blog posts or media coverage that’s not linked above, please let me know. I storified some reactions on Twitter but I’m certain that I missed conversations or opinions.
There are two additional insights from Pew that I didn’t write about yesterday that are worth keeping in mind with respect to how how Americans are thinking about the release of public data back to the public. First, it’s unclear whether the public realizes they’re using apps and services built upon government data, despite sizable majorities doing so.
Majorities of Americans use apps & services that use #opendata; Just 5% think govt shares well http://t.co/e11502mCUZ pic.twitter.com/zdwYzpnaJM
— Alex Howard (@digiphile) April 21, 2015
Second, John Horrigan told me that survey respondents universally are not simply asking for governments to make the data easier to understand so that they can figure out what I want to figure out: what people really want is intermediaries to help them make sense of the data.
“We saw a fair number of people pleading in comments for better apps to make the data make sense,” said Horrigan. “When they went online, they couldn’t get budget data to work. When the found traffic data, couldn’t make it work. There were comments on both sides of the ledger. Those that think government did an ok job wish they did this. Those that thin government is doing a horrible job also wish they did this.”
This is the opportunity that Headd referred to, and the reason that data journalism is the critical capacity that democratic governments which genuinely want to see returns on accountability and transparency must ensure can flourish in civil society.
If a Republican is elected as the next President of the United States, we’ll see if public views shift on other fronts.
Today, a new survey released by the Pew Research Internet and Life Project provided one of the most comprehensive snapshots into the attitudes of the American public towards open data and open government to date. In general, more people surveyed are guardedly optimistic about the outcomes and release of open data, although that belief does vary with their political views, trust in government, and specific areas. (Full disclosure: I was consulted by Pew researchers regarding useful survey questions to pose.)
“Trust in government is the reference that people bring to their answers on open government and open data,” said John Horrigan, the principal researcher on the survey, in an interview. “That’s the frame of reference people bring. A lot of people still aren’t familiar with the notion, and because they don’t have a framework about open data, trust dominates, and you get the response that we got.”
While majorities of the American public use applications and services that use government data, from GPS to weather to transit to health apps, relatively few are aware that data produced and released by government drives them.
“The challenge for activists or advocates in this space will be to try to make the link between government data and service delivery outcomes,” said Horrigan. “If the goals are to make government perform better and maybe reverse the historic tide of lowered trust, then the goal is to make improvements real in delivery. If this is framed just as argument over data quality, it would go into an irresolvable back and forth into the quality of government data collection. If you can cast it beyond whether unemployment statistics are correct or not but instead of how government services improve or saved money, you have a chance of speaking to wether government data makes things better.”
The public knowledge gap regarding this connection is one of the most important points that proponents, advocates, journalists and publishers who wish to see funding for open data initiatives be maintained or Freedom of Information Act reforms pass.
“I think a key implication of the findings is that – if advocates of government data initiatives hope that data will improve people’s views about government’s efficacy – efforts by intermediaries or governments to tie the open data/open government to the government’s collection of data may be worthwhile,” said Horrigan. “Such public awareness efforts might introduce a new “mental model” for the public about what these initiatives are all about. Right now, at least as the data for this report suggests, people do not have a clear sense of government data initiatives. And that means the context for how they think about them has a lot to do with their baseline level of trust in the government – particularly the federal government.”
Horrigan suggested thinking about this using a metaphor familiar to anyone who’s attended a middle school dance.
“Because people do engage with the government online, just through services, it’s like getting them on a big dance floor,” he suggested. “They’re on the floor, where you want them, but they’re on the other part of it. They don’t know that there’s another part of the dance that they’d like to see or be drawn to that they’d want to be in. There’s an opportunity to draw them. The good news that they’re on the dance floor, the bad news is they don’t know about all of it. Someone might want to go over and talk to them an explain that if you go over here you might have a better experience.”
Following are 13 more key insights about the public’s views regarding the Internet, open data and government. For more, make sure to read the full report on open government data, which is full of useful discussion of its findings.
One additional worth noting before you dive in: this survey is representative of American adults, not just the attitudes of people who are online. “The Americans Trends Panel was recruited to be nationally representative, and is weighted in such a way (as nearly all surveys are) to ensure responses reflect the general population,” said Horrigan. “The overall rate of internet use is a bit higher than we typically record, but within the margin of error. So we are comfortable that the sample is representative of the general population.”
While Pew cautions that the questions posed in this survey are different from another conducted in 2010, the trend is clear: the way citizens communicate with government now includes the Internet, and the way government communicates with citizens increasingly includes digital channels. That use now includes getting information or data about federal, state and local government.
About a third of college grads, young people and wealthy Americans have checked out performance data or government contracting data, or about 50% more than other age groups, lower income or non-college grads.
Pew grouped the 3,212 respondents into four quadrants, seen below, with a vertical axis ranging from optimism to skepticism and a horizontal axis that described use. Notably, more use of data doesn’t correlate to more belief in positive outcomes.
“In my mind, you have to get to the part of the story where you show government ran better as a result,” said Horrigan. “You have to get to a position where these stories are being told. Then, at least, while you’re opening up new possibilities for cynicism or skepticism, you’re at least focused on the data as opposed to trust in government.”
Instead…
Put simply, Democrats trust the federal government more, and that relates to how people feel about open data released by that government.
One challenge is that if President Barack Obama says “open data” again, he may further associate the release of government data with Democratic policies, despite bipartisan support for open government data in Congress. If a Republican is elected President in November 2016, however, this particular attitude may well shift.
“That’s definitely the historic pattern, tracked over time, dating to 1958,” said Horrigan, citing a Pew study. “If if holds and a Republican wins the White House, you’d expect it to flip. Let’s say that we get a Republican president and he continues some of these initiatives to make government perform better, which I expect to be the case. The Bush administration invested in e-government, and used the tools available to them at the time. The Obama administration picked it up, used the new tools available, and got better. President [X] could say this stuff works.”
The unresolved question that we won’t know the answer to until well into 2017, if then, is whether today’s era of hyper-partisanship will change this historic pattern.
There’s bipartisan agreement on the need to use government data better in government. Democratss want to improve efficiency and effectiveness, Republicans want to do the same, but often in the context of demonstrating that programs or policies are ineffective and thereby shrink government. If the country can rise about partisan politics to innovate government, awareness of the utility of releases will grow, along with support for open data will grow.
“Many Americans are not much attuned to government data initiatives, which is why they think about them (in the attitudinal questions) through the lens of whether they trust government,” said Horrigan. “Even the positive part of the attitudinal questions (i.e., the data initiatives can improve accountability) has a dollop of concern, in that even the positive findings can be seen as people saying: ‘These government data initiatives might be good because they will shine more light on government – which really needs it because government doesn’t perform well enough.’ That is an opportunity of course – especially for intermediaries that might, through use of data, help the public understand how/whether government is being accountable to citizens.”
That opportunity is cause for hope.
“Whether it is ‘traditional’ online access for doing transactions/info searches with respect to government, or using mobile apps that rely on government data, people engage with government online, “said Horrigan. “That creates the opportunity for advocates of government data initiatives to draw citizens further down the path of understanding (and perhaps better appreciating) the possible impacts of such initiatives.”
On April 1st, some reporters, open government advocates and people in industry may have hoped that a new redesign of USASpending.gov, the flagship financial transparency website of the United States government, was just a poorly conceived April Fool’s joke. Unfortunately, an official statement about the USASpending.gov redesign at the U.S. Treasury’s blog confirmed that the redesign was real. Analysts, media and businesses that rely on the contracting data on the site were loudly decried the decreased functionality of USASpending.gov.
A week later, there’s a still no evidence of deliberate intent on the part of Treasury not to publish accurate spending data or break the tool, despite headlines about rolling back transparency. Rather, it looks more likely that there were been a number of mistakes or even unavoidable errors made in the transitioning the site and data from a bankrupt federal contractor. There was certainly poor communication with the business community and advocates who use the site, a reality that Luke Fretwell helpfully suggested at Govfresh that other government agencies work to avoid next time.
Today, as Fretwell first reported, the federal government launched a new repository for tracking issues on USASpending.gov on Github, the social coding site that’s become an increasingly important platform for 18F, which committed to developing free and open source software by default last year.
In an email to the White House’s open government Google Group, Corinna Zarek, the senior advisor for open government in the Obama administration, followed up on earlier concerns about the redesign:
The USAspending team has been working to improve the usability of the site and has made some great strides to make it easier for average citizens to navigate information. But at the same time, we all understand that some of our expert users (like a lot of you) seek more technical information and the team is striving to meet your needs as well.
This is definitely a work in progress so please keep working with the team as it iterates on the best ways to improve function of the site while maintaining the content you seek. Your initial comments have been really helpful and the USAspending team is already working to address some of them.
Zarek also said that several of the problems with data that people have reported been addressed, including the capacity to download larger data sets and define specific dates in search, and asked for more feedback.
Specifically, this week the team addressed data export issues to allow the ability to specify date ranges to download data, added the bulk file format API, and modified the download capability so larger datasets can be downloaded. Additionally, data archives are being added continually. This week, they loaded the 2014 and 2015 delta files that show the new transactions in the last month. You can keep track of the ongoing improvements on the “What’s new” page.
Please keep sharing your feedback and continue working with the USAspending team as it makes improvements to the site. You can do this through the site’s contact page or on the new Github page where you can report issues and track them in the open.
If you find bugs, let the feds know about them on Github so that everyone can see the issues and how they’re addressed. As Mollie Walker reported for FierceGovernmentIT, there’s still missing functionality yet to be restored.
[Image Credit: Govfresh, via USASpending.gov]
In hiring Stephanie Hannon to be her campaign’s chief technology officer, former Secretary of State Hillary Clinton is not just bringing on a former Cisco software engineer with product management experience at Google and Facebook and time in the trenches at tech startups: she’s added a woman who’s deeply immersed in the civic technology movement and knowledgeable about open data. Just watch her talk at the 2014 Code for America Summit:
So, here’s the bad news: Hannon was a product manager for Gmail and Google Wave, so steel yourself for a lot of bad jokes in the days and months ahead in the media, given her new boss’ questionable choices about email as Secretary of State and Google Wave’s demise. While a B.A. and an M.S. in electrical engineering from Stanford and an MBA from Harvard will insulate her from the some of the lamer slings and arrows, get ready for unsubtle, tortured metaphors and misogyny in the comment sections.
Here’s the mixed news: Hannon’s focus on open data on Google appears to have been on standards, services and civic impact, not accountability and transparency. Take a look at her presentation, embedded below:
Given the record of the Obama administration, it remains to be seen whether Clinton will proactively adopt an ambitious agenda on open government if elected, from implementing and resource FOIA reforms, whistleblower support or nominating inspector generals for all federal agencies.
And here’s the good news: while political observers will (and should!) no doubt focus upon her ability to duplicate the success of Harper Reed, the CTO for Barack Obama’s 2012 presidential campaign, Clinton just brought someone who’s tapped into what’s happening in the civic tech space into orbit. If that experience elevates those issues into the campaign and national conversation, it will stand to benefit everyone working towards improving civic life in America. If Clinton starts talking about “building better governance with the people, not for them” on the campaign trail, you’ll know something important has occurred.
There are roughly 1,361 .gov domains* operated by the executive branch of the United States federal government, 700-800 of which are live and in active use. Today, for the first time, the public can see how many people are visiting 300 executive branch government domains in real-time, including every cabinet department, by visiting analytics.usa.gov.
According to a post on the White House blog, the United States Digital Service “will use the data from the Digital Analytics Program to focus our digital service teams on the services that matter most to the American people, and analyze how much progress we are making. The Dashboard will help government agencies understand how people find, access, and use government services online to better serve the public – all while protecting privacy. The program does not track individuals. It anonymizes the IP addresses of all visitors and then uses the resulting information in the aggregate.”
On Thursday morning, March 19th, tax-related services, weather, and immigration status are all popular. Notably, there’s an e-petition on the White House WeThePeople platform listed as well, adding data-driven transparency to what’s popular there right now.
Former United States deputy chief technology officer Nick Sinai is excited about seeing the Web analytics data opened up online. Writing for the Harvard Shorenstein Center, where he is currently a fellow, Sinai adds some context for the new feature:
“Making government web performance open follows the digital services playbook from the new U.S. Digital Services,” he wrote. “Using data to drive decisions and defaulting to open are important strategies for building simple and useful citizen-facing digital services. Teal-time and historical government web performance is another example of how open government data holds the promise of improving government accountability and rebuilding trust in government.”
Here’s what the U.S. digital services team says they’ve already learned from analyzing this data:
Here’s what we’ve already learned from the data:
- Our services must work well on all devices. Over the past 90 days, 33% all traffic to our sites came from people using phones and tablets. Over the same period last year, the number was 24%. Most of this growth came from an increase in mobile traffic. Every year, building digital services that work well on small screens becomes more important.
- Seasonal services and unexpected events can cause surges in traffic. As you might expect, tax season is a busy time for the IRS. This is reflected in visits to pages on IRS.gov, which have more than tripled in the past 90 days compared with the previous quarter. Other jumps in traffic are less easy to predict. For example, a recently-announced settlement between AT&T and the Federal Trade Commissiongenerated a large increase in visits to the FTC’s website. Shortly after the settlement was announced, FTC.gov had four times more visitors than the same period in the previous year. These fluctuations underscore the importance of flexibility in the way we deploy our services so that we can scale our web hosting to support surges in traffic as well as save money when our sites are less busy.
- Most people access our sites using newer web browsers. How do we improve digital services for everyone when not all web browsers work the same way? The data tells us that the percentage of people accessing our sites using outdated browsers is declining steadily. As users adopt newer web browsers, we can build services that use modern features and spend less time and money building services that work on outdated browsers. This change will also allow us to take advantage of features found in modern browsers that make it easier to build services that work well for Americans with disabilities, who access digital services using specialized devices such as screen readers.
If you have ideas, feedback or questions, the team behind the dashboard is working in the open on Github.
Over the coming months, we will encourage more sites to join the Digital Analytics Program, and we’ll include more information and insights about traffic to government sites with the same open source development process we used to create the Dashboard. If you have ideas for the project, or want to help improve it, let us know by contributing to the project on GitHub or emailing digitalgov@gsa.gov.
That last bit is notable; as its true all of the projects that 18F works on, this analytics dashboard is open source software.
There are some interesting additional details in 18F’s blog post on how the analytics dashbard was built, including the estimate that it took place “over the course of 2-3 weeks” with usability testing at a “local civic hacking meetup.”
First, that big number is made from HTML and D3, a Javascript library, that downloads and render the data. Using open standards means it renders well across browsers and mobile devices.
Second, 18F made an open source tool to manage the data reporting process called “analytics-reporter” that downloads Google Analytics reports and transforms that data into JSON.
Hopefully, in the years ahead, the American people will see more than the traffic to .gov websites: they’ll see concrete performance metrics like those displayed for the digital services the United Kingdom’s Government Digital Services team publishes at gov.uk/performance, including uptime, completion rate and satisfaction rate.
In the future, if the public can see the performance of Heathcare.gov, including glitches, or other government digital services, perhaps the people building and operating them will have more accountability for uptime and quality of service.
