Category Archives: government information technology

Esri’s new ArcGIS feature is live. Will terabytes of new open data follow?

Posted on by

0

esri-open-data

Back in February, I reported that Esri would enable governments to open their data to the public.Today, the geographic information systems (GIS) software giant pushed ArcGIS Open Data live, instantly enabling thousands of its local, state and federal government users to open up the public data in their systems to the public, in just a few minutes.

open-data-esri

“Starting today any ArcGIS Online organization can enable open data, specify open data groups and create and publicize their open data through a simple, hosted and best practices web application,” wrote Andrew Turner, chief technology officer of Esri’s Research and Development Center in D.C., in a blog post about the public beta of Open Data ArcGIS. “Originally previewed at FedGIS ArcGIS Open Data is now public beta where we will be working with the community on feedback, ideas, improvements and integrations to ensure that it exemplifies the opportunity of true open sharing of data.”

Turner highlighted what this would mean for both sides of the open data equation: supply and demand.

Data providers can create open data groups within their organizations, designating data to be open for download and re-use, hosting the data on the ArcGIS site. They can also create public microsites for the public to explore. (Example below.) Turner also highlighted the code for Esri’s open-source GeoPortal Server on Github as a means to add metadata to data sets.

Data users, from media to developers to nonprofits to schools to businesses to other government entities, will be able to download data in common open formats, including KML, Spreadsheet (CSV), Shapefile, GeoJSON and GeoServices.

“As the US Open Data Institute recently noted, [imagine] the impact to opening government data if software had ‘Export as JSON’ by default,” wrote Turner.

“That’s what you now have. Users can also subscribe to the RSS feed of updates and comments about any dataset in order to keep up with new releases or relevant supporting information. As many of you are likely aware, the reality of these two perspectives are not far apart. It is often easiest for organizations to collaborate with one another by sharing data to the public. In government, making data openly available means departments within the organization can also easily find and access this data just as much as public users can.”

EmploymentLaborForces_B230251

Turner highlighted what an open data site would look like in the wild:

“Data Driven Detroit a great example of organizations sharing data. They were able to leverage their existing data to quickly publish open data such as census, education or housing. As someone who lived near Detroit, I can attest to the particular local love and passion the people have for their city and state – and how open data empowers citizens and businesses to be part of the solution to local issues.

In sum, this feature could, as I noted in February, could mean a lot more data is suddenly available for re-use. When considered in concert with Esri’s involvement in the White House’s Climate Data initiative, 2014 looks set to be a historic year for the mapping giant.

It also could be a banner year for open data in general, if governments follow through on their promises to release more of it in reusable forms. By making it easy to upload data, hosting it for free and publishing it in the open formats developers commonly use in 2014, Esri is removing three major roadblocks governments face after a mandate to “open up” come from a legislature, city council, or executive order from the governor or mayor’s office.

“The processes in use to publish open data are unreasonably complicated,” said Waldo Jacquith, director of the U.S. Open Data Institute, in an email. 

“As technologist Dave Guarino recently wrote, basically inherent to the process of opening data is ETL: “extract-transform-load” operations. This means creating a lot of fragile, custom code, and the prospect of doing that for every dataset housed by every federal agency, 50 states, and 90,000 local governments is wildly impractical.

Esri is blazing the trail to the sustainable way to open data, which is to open it up where it’s already housed as closed data. When opening data is as simple as toggling an “open/closed” selector, there’s going to be a lot more of it. (To be fair, there are many types of data that contain personally identifiable information, sensitive information, etc. The mere flipping of a switch doesn’t address those problems.)

Esri is a gold mine of geodata, and the prospect of even a small percentage of that being released as open data is very exciting.”

New study details technology deficit in government and civil society

Posted on by

0

stem-talent-federal-agenciesThe botched re-launch of Healthcare.gov led many observers unfamiliar with the endemic issues in government information technology to wonder how the first Internet president produced the government’s highest Internet failure. The Obama administration endured a winter full of well-deserved criticism, some informed, some less, regarding what went wrong at Healthcare.gov, from bad management to poor technology choices and implementation, agency insularity and political sensitivity at the White House.

While “Obama’s trauma team” successfully repaired the site, enabling millions to enroll in the health insurance plans offered in the online marketplace, the problems the debacle laid bare in human resources and IT procurement are now receiving well-deserved attention. While the apparent success of “the big fix” has taken some urgency away from Congress or the administration to address how the federal government can avoid another Healthcare.gov, the underlying problems remain. Although lawmakers have introduced legislation to create a “Government Digital Office” and the U.S. House of Representatives passed a bill to reform aspects of federal IT, neither has gotten much traction in the Senate. In the meantime, hoping to tap into the success of the United Kingdom’s Government Digital Services team, the U.S. General Services Administration has stood up a new IT services unit, 18F, which officials hope will help government technology projects fail fast instead of failing big.

Into this mix comes  a new report from Friedman Consulting, commissioned by the Ford and MacArthur Foundations. Notably, the report also addresses the deficit of technology talent in the nonprofit sector and other parts of civil society, where such expertise and capacity could make demonstrable improvements to operations and performance. The full 51 page report is well worth reading, for those interested in the topic, but for those limited by time, here are the key findings:

1) The Current Pipeline Is Insufficient: the vast majority of interviewees indicated that there is a severe paucity of individuals with technical skills in computer science, data science, and the Internet or other information technology expertise in civil society and government. In particular, many of those interviewed noted that existing talent levels fail to meet current needs to develop, leverage, or understand technology.
2) Barriers to Recruitment and Retention Are Acute: many of those interviewed said that substantial barriers thwart the effective recruitment and retention of individuals with the requisite skills in government and civil society. Among the most common barriers mentioned were those of compensation, an inability to pursue groundbreaking work, and a culture that is averse to hiring and utilizing potentially disruptive innovators.
3) A Major Gap Between The Public Interest and For-Profit Sectors Persists: as a related matter, interviewees discussed superior for-profit recruitment and retention models. Specifically the for-profit sector was perceived as providing both more attractive compensation (especially to young talent) and fostering a culture of innovation, openness, and creativity that was seen as more appealing to technologists and innovators.
4) A Need to Examine Models from Other Fields: interviewees noted significant space to develop new models to improve the robustness of the talent pipeline; in part, many existing models were regarded as unsustainable or incomplete. Interviewees did, however, highlight approaches from other fields that could provide relevant lessons to help guide investments in improving this pipeline.
5) Significant Opportunity for Connection and Training: despite consonance among those interviewed that the pipeline was incomplete, many individuals indicated the possibility for improved and more systematic efforts to expose young technologists to public interest issues and connect them to government and civil society careers through internships, fellowships, and other training and recruitment tools.
6) Culture Change Necessary: the culture of government and civil society – and its effects on recruitment and other bureaucratic processes – was seen as a
vital challenge that would need to be addressed to improve the pipeline. This view manifested through comments that government and civil society organizations needed to become more open to utilizing technology and adopting a mindset of experimentation and disruption.

And here’s the conclusion:

Based on this research, the findings of the report are clear: technology talent is a key need in government and civil society, but the current state of the pipeline is inadequate to meet that need. The bad news is that existing institutions and approaches are insufficient to build and sustain this pipeline, particularly in the face of
sharp for-profit competition. The good news is that stakeholders interviewed identified a range of organizations and practices that, at scale, have the potential to make an enormous difference. While the problem is daunting, the stakes are high. It will be critical for civil society and government to develop sustainable and
effective pathways for the panoply of technologists and experts who have the skills to create truly 21st century institutions.

For those interested, the New America Foundation will be hosting a forum on the technology deficit in Washington, DC, on April 29th. The event will be livestreamed and archived.

Boston Mayor Marty Walsh issues open data executive order; city council ordinance to come?

Posted on by

0

5943910065_b422feecec_o

The City of Boston has joined the growing list of cities around the world that have adopted open data. The executive order issued yesterday by Mayor Marty Walsh has been hailed by open government advocates around the country. The move to open up Boston’s data has been followed by action, with 411 data sets listed on data.cityofboston.gov as of this morning. The EO authorizes and requires Boston’s chief information officer to issue a City of Boston Open Data Policy and “include standards for the format and publishing of such data and guidance on accessibility, re-use and minimum documentation for such data.”

The element on re-use is critical: the success of such initiatives should be judged based upon the network effects of open data releases, not the raw amount of data published online, and improvements to productivity, efficiency, city services, accountability and transparency.

Notably, Boston City Councilor-at-Large Michelle Wu also filed a proposal yesterday morning to create an open data ordinance that would require city agencies and departments to make open data available, codifying the executive order into statue as San Francisco, New York City and Philadelphia have done.

“Government today should center on making data-driven decisions and inviting in the public to collaborate around new ideas and solutions,” said Wu, in a statement.  “The goal of this ordinance is greater transparency, access, and innovation.  We need a proactive, not a reactive, approach to information accessibility and open government.”

 

Notably, she posted the text of her proposed open data ordinance online on Monday, unlike the city government, and tweeted a link to it. (It took until today for the city of Boston to post the order; city officials have yet to share it on social media. )

“Boston is a world-class city full of energy and talent,” said Wu. “In addition to promoting open government, making information available to the fullest extent possible will help leverage Boston’s energy and talent for civic innovation. From public hackathons to breaking down silos between city departments, putting more data online can help us govern smarter for residents in every neighborhood.”

As long-time readers know, I lived in Boston for a decade. It’s good to see the city government move forward to making the people’s data available to them for use and reuse. I look forward to seeing what the dynamic tech, financial, health care, educational and research communities in the greater Boston area do with it.

EXECUTIVE ORDER OF MAYOR MARTIN J. WALSH

An Order Relative to Open Data and Protected Data Sharing

Whereas, it is the policy of the City of Boston to practice Open Government, favoring participation, transparency, collaboration and engagement with the people of the City and its stakeholders; and
Whereas, information technologies, including web-based and other Internet applications and services, are an essential means for Open Government, and good government generally; and
Whereas, the City of Boston should continue, expand and deepen the City’s innovative use of information technology toward the end of Open Government, including development and use of mobile computing and applications, provision of online data, services and transactions; and
Whereas, the City of Boston also has an obligation to protect some data based upon privacy, confidentiality and other requirements and must ensure that protected data not be released in violation of applicable constraints; and
Whereas, clarification and definition of open data, privacy, security requirements, interoperability and interaction flows is necessary for the City’s Open Government agenda;
NOW THEREFORE, pursuant to the authority vested in me as Chief Executive Officer of the City of Boston by St. 1948, c. 452 Section 11, as appearing in St. 1951, c. 376, Section 1, and every other power hereto enabling, I hereby order and direct as follows:

1. The City of Boston recognizes Open Government as a key means for enabling public participation, transparency, collaboration and effective government, including by ensuring the availability and use of Open Data, appropriate security and sharing of Protected Data, effective use of Identity and Access Management and engagement of stakeholders and experts toward the achievement of Open Government.
2. The City of Boston Chief Information Officer (“CIO”), in consultation with City departments, is authorized and directed to issue a City of Boston Open Data Policy.
a) The Open Data Policy shall include standards for the format and publishing of such data and guidance on accessibility, re-use and minimum documentation for such data;

b) The Open Data Policy shall include guidance for departments on the classification of their data sets as public or protected and a method to report such classification to the CIO. All departments shall publish their public record data sets on the City of Boston open data portal to the extent such data sets are determined to be appropriate for public disclosure, and/or if appropriate, may publish their public record data set through other methods, in accordance with API, format, accessibility and other guidance of the Open Data Policy.
3. The City of Boston CIO, in consultation with City departments, is authorized and directed to issue a City of Boston Protected Data Policy applicable to non-public data, such as health data, educational records and other protected data;

a) The policy shall provide guidance on the management of Protected Data, including guidance on security and other controls to safeguard Protected Data, including appropriate Identity and Access Management and good practice guidelines for compliance with legal or other rules requiring the sharing of Protected Data with authorized parties upon the grant of consent, by operation of law or when otherwise so required;
b) The policy shall provide a method to ensure approval by the Corporation Counsel of the City of Boston to confirm Protected Data is only disclosed in accordance with the Policy.
4. This Executive Order is not intended to diminish or alter the rights or obligations afforded under the Massachusetts Public Records Law, Chapter 66, Section 10 of the Massachusetts General Laws and the exemptions under Chapter 4, Section 7(26). Additionally, this Executive Order is intended to be interpreted consistent with Federal, Commonwealth, and local laws and regulations regarding the privacy, confidentiality, and security of data. Nothing herein shall authorize the disclosure of data that is confidential, private, exempt or otherwise legally protected unless such disclosure is authorized by law and approved by the Corporation Counsel of the City of Boston.
5. This Executive Order is not intended to, and does not, create any right or benefit, substantive or procedural, enforceable at law or in equity by any party against the City of Boston, its departments, agencies, or entities, its officers, employees, or agents, or any other person.
6. The City of Boston CIO is authorized and directed to regularly consult with experts, thought leaders and key stakeholders for the purpose of exploring options for the implementation of policies and practices arising under or related to this Executive Order.

Can NewsGenius make annotated government documents more understandable?

Posted on by

1

Rap-Genius-640x325

Last year, Rap Genius launched News Genius to help decode current events. Today, the General Service Administration (GSA) announced that digital annotation service News Genius is now available to help decode federal government Web projects:

“The federal government can now unlock the collaborative “genius” of citizens and communities to make public services easier to access and understand with a new free social media platform launched by GSA today at the Federal #SocialGov Summit on Entrepreneurship and Small Business,” writes Justin Herman, federal social media manager.

“News Genius, an annotation wiki based on Rap Genius now featuring federal-friendly Terms of Service, allows users to enhance policies, regulations and other documents with in-depth explanations, background information and paths to more resources. In the hands of government managers it will improve public services through citizen feedback and plain language, and will reduce costs by delivering these benefits on a free platform that doesn’t require a contract.”

This could be a significant improvement in making complicated policy documents and regulations understandable to the governed. While plain writing is indispensable for open government and mandated by law and regulation, the practice isn’t exactly uniformly practiced in Washington.

If people can understand more about what a given policy, proposed rule or regulation actually says, they may well be more likely to participate in the process of revising it. We’ll see if people adopt the tool, but on balance, that sounds like a step ahead.

600-x-320-GSA-Mentor-Protege-Program-subpart-519-70-on-cell-phoneWhat could this look like? As Herman noted, Chicago’s SmartChicago Collaborative uses RapGenius to annotate municipal documents.

Another recent example comes from DOBTCO founder and CEO Clay Johnson, who memorably put RapGenius to good use last year decoding testimony on Healthcare.gov.

The GSA’s first use is for a mentor-protege program.

Here’s hoping more subject matter experts start annotating.

[Image Credit: Huffington Post]

On data journalism, accountability and society in the Second Machine Age

Posted on by

1

On Monday, I delivered a short talk on data journalism, networked transparency, algorithmic transparency and the public interest at the Data & Society Research Institute’s workshop on the social, cultural & ethical dimensions of “big data”. The forum was convened by the Data & Society Research Institute and hosted at New York University’s Information Law Institute at the White House Office of Science and Technology Policy, as part of an ongoing review on big data and privacy ordered by President Barack Obama.

Video of the talk is below, along with the slides I used. You can view all of the videos from the workshop, along with the public plenary on Monday evening, on YouTube or at the workshop page.

Here’s the presentation, with embedded hyperlinks to the organizations, projects and examples discussed:

For more on the “Second Machine Age” referenced in the title, read the new book by Erik Brynjolfsson and Andrew McAfee.

Obama administration seeks to increase community resilience against climate change

Posted on by

1

Today, the White House launched an effort to engage the nation’s private sector to create tools and resources that increase the resiliency of communities to extreme weather events.

20140319-051043.jpg

In the pilot phase, more data related to coastal flooding are now on Data.gov, with more on projected sea level rise and estimated impacts to follow. More government data from NOAA, NASA, the U.S. Geological Survey, the Department of Defense, and other federal agencies will be featured on climate.data.gov. NOAA and NASA have will host a challenge for researchers and developers “to create data-driven simulations to help plan for the future and to educate the public about the vulnerability of their own communities to sea level rise and flood events.”

20140319-143621.jpg

As the Associated Press reported, another effort plans to add sensors on city buses in Philadelphia to collect data. Should the effort go forward and be expanded, it will provide an important focus for sensor journalism.

A number of private sector companies have announced their involvement. part. Later today, The World Bank will publish a new field guide for the “Open Data for Resilience Initiative.” Esri will partner with a dozen cities across the USA to challenge developers to use its ArcGIS platform. Google will donate one petabyte of storage for climate data and 50 million hours of processing time on the Google Earth Engine.

In sum, the focus on this component of the initiative is on helping people understand and plan for potential changes to their communities, as opposed to using data to make a case to the public about the source or science of climate change. While it is no substitute for increased public understanding of the latter, improving local resiliency to severe weather through data-driven analyses is a sufficiently pragmatic, useful approach that it might just have an impact.

The White House will host a forum at 5 PM in DC today featuring talks by officials and executives from the agencies and companies involved. More details and context on the new climate data initiative are available at the White House blog.

[Image credit: NASA]