In the video embedded above, Pollock talks about his involvement with OGP and how civil society will be involved in holding government accountable. He also explains what open data means to him, including a definition and how it relates to traditional open government goals of transparency and accountability. Pollock recommends the Open Data Handbook as a resource to learn more and put data to work in the service of better government.
As TechnicallyPhilly reported this morning, the city of Philadelphia, Pennsylvania has joined the ranks of municipalities putting more public data onto the Internet.
“Transparency is a cornerstone of good governance, and it is vital for the City to be open and available to our citizens,” said Mayor Michael Nutter in a statement posted to the city of Philadelphia’s Facebook page. “Philadelphia was recently named at the seventh most social media savvy city in the nation. The Open Data policy furthers many of the policies and initiatives already put in place by the City.”
“The Open Data Policy puts in place the necessary framework, structure and governance that will increase collaboration among City departments and bring citizens closer to their government,” said Chief Innovation Officer Adel Ebeid. “This policy is the first installment in Mayor Nutter’s vision for Philadelphia to become a model for increasing transparency and removing barriers to information sharing and collaboration.”
As NBC Philadelphia reported, the executive order also establishes an internal social media policy for Philadelphia municipal government.
The city now has 90 days to select or hire a chief data officer (a position that Logan Clier called for allcities to establish on the Code for America blog earlier today) and 120 days to establish a “data governance advisory” board, both of which will be in entrusted with established standards and means of publishing open data, along with periodically evaluating the releases to date.
Philadelphia may soon have an opportunity to compare notes with other cities that have pursued open data platforms around the United States, including San Francisco, Chicago, Seattle, Portland, San Francisco, Boston and New York City. NYC has set up a wiki to help implement its landmark open data legislation, an example that Philadelphians might draw inspiration from, with respect to forming more collaborative and transparent processes online.
There’s much to like in this executive order, for open data advocates, but one phrase in particular jumps out: “Each City department and agency shall develop a schedule for making information available to the public and updating it on a regular basis.”
This could go a long way to addressing key concern that has been extant in other cities and states, where data sets go online but are not subsequently updated. That will only be true, however, if political will is coupled with policy clout to drive more release and public engagement with media, academy and Philadelphia’s technical community to put the data to work for the public’s good.
The good news on that count is that Philadelphia has a partner in Technically Philly, which has been an active participant in driving this change:
As with every open data effort, the devil will be in the details. Or, to put it another way, the devil will be in the datasets, including the quality and relevance of what’s posted. That said, it’s impossible to see today’s action as anything other than a watershed for the city that I grew up in, from 1984 to 1994, and I can’t help but hope that everyone in the City of Brotherly Love collaborates in making the most of the opportunity that now lies before Philadelphia to apply data for the public good.
Go make stuff that matters.
As of 7:43 PM ET this evening, the city had not yet posted the executive order to Phila.gov, the city’s official website. I’ve published the EO on open data and government transparency in full below.
EXECUTIVE ORDER NO. __ -12
OPEN DATA AND GOVERNMENT TRANSPARENCY
WHEREAS, the City of Philadelphia is committed to creating a high level of openness and transparency in government; and
WHEREAS, the three principles of transparency, participation, and collaboration form the cornerstone of an open government; and
WHEREAS, the City’s participation as a founding and vital partner in the open data consortium has provided a model for transparency on which the City should continue to build; and
WHEREAS, more City data sets should be published and made available via an Open Data Portal which will provide access to information and a mechanism for public feedback and participation; and
WHEREAS, the demands of an across-the-board open government framework require the dedication of a new position, of Chief Data Officer, to direct these initiatives; and
WHEREAS, social media tools have become a part of everyday life for City employees and City residents, such that social media can be a means of increasing government transparency and civic engagement; and
WHEREAS, timetables should be established for development and implementation of an overall Open Government Plan to enhance and develop transparency, public participation, and collaboration in all City activities;
NOW THEREFORE, I, Michael A. Nutter, Mayor of the City of Philadelphia, by the authority vested in me by the Philadelphia Home Rule Charter, do hereby order as follows:
SECTION 1. OPEN DATA WORKING GROUP AND CHIEF DATA OFFICER
A. As soon as practicable, the Mayor and the Chief Innovation Officer (CIO) will establish an Open Data Working Group to focus on transparency, accountability, participation, and collaboration within City government. The Working Group, which will include senior level representation from program and management offices throughout the City, will assist the CIO in selecting a Chief Data Officer. The Working Group will also provide a forum to develop innovative ideas for promoting open government goals, including collaborations with researchers, the private sector, and the public, and for developing resolutions to issues raised through the public feedback mechanisms of the Open Government Portal.
B. Within 90 days of the Effective Date of this Order, the CIO, with assistance from the Open Data Working Group, shall hire or designate an individual to serve as Chief Data Officer (CDO). The CDO will lead the Open Data and Transparency initiatives outlined in this Order, including working with City departments and agencies to establish standards for publication of data and the most effective means for making such data available. The CDO will report to the Chief Innovation Officer.
SECTION 2. DATA GOVERNANCE ADVISORY BOARD
Within 120 days from the Effective Date of this Order, the Mayor shall appoint a Data Governance Advisory Board. The Board shall consist of nine members, including the Chief Innovation Officer and the CDO, and shall be chaired by an individual designated by the Mayor. The Open Data Working Group shall solicit nominations for members of the Advisory Board, and shall recommend appointments from the public, private, academic and nonprofit sectors. The Advisory Board shall meet regularly at such times as the Board decides, and its members shall serve at the pleasure of the Mayor.
SECTION 3. OPEN GOVERNMENT PLAN
A. Development of Plan. Within six months of the Effective Date of this Order, the CIO and the CDO, in conjunction with the Advisory Board, shall develop and publish an Open Government Plan. The plan will detail, including specific actions and timelines, the steps that the City will take to incorporate the principles of open government into its daily activities.
B. The Plan shall be formulated with the input of senior policy, legal, and technology leadership in the City; open government experts; and the general public.
C. Components of the Plan shall include:
(1) Transparency: Steps the City will take to conduct its work more openly and publish its information online, including ready public access to ordinances and regulations, policies, legislative records, budget information, crime statistics, public health statistics, and other information. Where possible, publication shall be in an open format, subject to privacy, confidentiality, and security concerns, and to the City’s Social Media Use Policy. Additionally, the Plan will identify high value data sets not yet available to the public, and establish a reasonable timeline for their publication online in open formats.
(2) Public Participation: Description of how the City will enhance and expand opportunities for the public to participate throughout each City agency’s decision-making process, including instructions for online access to published information and opportunities for comment; methods for identifying stakeholders and other affected parties and encouraging their participation; links to appropriate websites where the public can engage in the City’s existing participatory processes; and proposed changes to internal management and administrative policies to increase public participation.
(3) Collaboration: Steps the City will take to enhance and expand cooperation among City departments and agencies, other governmental agencies, private and nonprofit entities, and the public, to fulfill City goals and obligations; including proposals to use technology platforms and links to appropriate websites to improve, and inform the public about, existing collaboration efforts, and use of innovative methods to obtain ideas from and to increase collaboration with those in the private sector, nonprofit and academic communities.
SECTION 4. OPEN DATA POLICY
A. Open Government Portal. Within 90 days of the Chief Data Officer’s assumption of responsibilities, the Office of Innovation and Technology shall establish a Portal that will serve as the source for Citywide and departmental activities with respect to this open government initiative. The Chief Innovation Officer, in his discretion, may build on previous open data initiatives, or may establish a new portal.
B. Identification of Barriers, Guidance and Revisions. Within 120 days of the Effective Date of this Order, the City Solicitor, in consultation with the Chief Innovation Officer, shall review existing city policies to identify impediments to open government and to the use of new technologies and, where necessary, issue clarifying guidance or propose revisions to such policies, where greater openness can be promoted without damage to the City’s legal and financial interests.
C. Department and Agency Open Formats. Each City department and agency shall develop a schedule for making information available to the public and updating it on a regular basis. To the extent practicable and subject to valid restrictions, agencies shall publish information on line (in addition to other planned or mandated publication methods), and in an open format. The open format will provide data in a form that can be retrieved, downloaded, indexed, searched and reused by commonly used web search applications and software. Such information shall, subject to legal and practical restrictions and to the City’s Social Media Use Policy, be made available to the public without restrictions that would impede re-use of the information.
D. Open Data Catalog. Within 90 days of the CDO’s assumption of duties, each City department and agency shall create a catalog of its public information. The catalog shall be made accessible through the Open Government Portal. The determination of what shall constitute “public information” and “high value data sets” for purposes of this Order, as well as what “high value data sets” should be shared as set forth in paragraph 4.E hereof, shall be made by each department or agency head in consultation with OIT and the Law Department.
E. High Value Data Sets. Within 120 days of the CDO’s assumption of duties, each Deputy Mayor shall identify and publish online, in an open format, at least three high-value data sets, not currently available on line or not available in a downloadable format.
F. Public Feedback. The Open Government Portal shall include a mechanism for the public to give feedback on and assess the quality of published information, provide input about what information should be a priority for publication, and provide input on the City’s Open Government Plan.
G. Legally Protected Information. Nothing in this Order shall be construed to supersede existing requirements for review and clearance of information exempt from disclosure under the Pennsylvania Right to Know Act and other applicable laws, regulations, or judicial orders.
H. Evaluation. The City’s progress toward meeting the open government goals set forth in this Order shall be evaluated six months from the Effective Date of this Order, again one year from the Effective Date, and annually thereafter. The evaluation shall be released on the Open Government Portal, and shall include criteria to be developed by the Advisory Board.
SECTION 5. SOCIAL MEDIA POLICY
A. The City of Philadelphia’s Social Media Use Policy is, by this Order, simultaneously adopted and incorporated herein by reference as if fully stated.
B. Going forward, the Mayor’s Director of Communications and Strategic Partnerships and the CIO, or their designees, shall consider any additional issues that arise concerning standards for the acceptable use of social media by City employees, as well as by members of the general public who comment on or otherwise interact with the City through its social media websites, and shall, with the review and approval of the Law Department, make such amendments as may be advisable to the Social Media Use Policy.
SECTION 6. EFFECTIVE DATE
This Order shall be effective immediately.
Date: ___________________ ____________________________
MICHAEL A. NUTTER, MAYOR
58 ministers, officials, members of civil society and the media explained what “open government” means to them at last week’s Open Government Partnership conference in Brazil. You can watch them all in the video player below. (Each clip is under 30 seconds.)
Since I heard that last year’s Gov 2.0 and Open Government Events Calendar was useful to the broader community, here’s this year’s version. There will be many other places around the globe for people to gather, talk and learn about Gov 2.0 in 2012 — just take a look through the many Govloop event listings. There will be any number of citizen-generated unconferences and hackathons, where the attendees generate the program. They’ll include CityCamps, BarCamps, PodCamps or MobileCamps. Check out the CityCamp calendar to find one near you and keep an eye out for CityCamp meetups in February.
The following listings are by no means comprehensive but should serve as a starting point if you’re wondering what’s happening, when and where. If you know about more Gov 2.0 events that should be listed here, please let me know at alex@oreilly.com or @digiphile.
Have ideas for improvement to the open government declaration I’ve embedded below? Edits to the document below should make their way back to the original draft.
In 2012, making sense of big data through narrative and context, particularly unstructured data, is now a strategic imperative for leaders around the world, whether they serve in Washington, run media companies or trading floors in New York City or guide tech titans in Silicon Valley.
While big data carries the baggage of huge hype, the institutions of federal government are getting serious about its genuine promise. On Thursday morning, the Obama Administration announced a “Big Data Research and Development Initiative,” with more than $200 million in new commitments. (See fact sheet provided by the White House Office of Science and technology policy at the bottom of this post.)
“In the same way that past Federal investments in information-technology R&D led to dramatic advances in supercomputing and the creation of the Internet, the initiative we are launching today promises to transform our ability to use Big Data for scientific discovery, environmental and biomedical research, education, and national security,” said Dr. John P. Holdren, Assistant to the President and Director of the White House Office of Science and Technology Policy, in a prepared statement.
The research and development effort will focus on advancing “state-of-the-art core technologies” need for big data, harnessing said technologies “to accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning,” and “expand the workforce needed to develop and use Big Data technologies.”
In other words, the nation’s major research institutions will focus on improving available technology to collect and use big data, apply them to science and national security, and look for ways to train more data scientists.
“IBM views Big Data as organizations’ most valuable natural resource, and the ability to use technology to understand it holds enormous promise for society at large,” said David McQueeney, vice president of software, IBM Research, in a statement. “The Administration’s work to advance research and funding of big data projects, in partnership with the private sector, will help federal agencies accelerate innovations in science, engineering, education, business and government.”
While $200 million dollars is a relatively small amount of funding, particularly in the context of the federal budget or as compared to investments that are (probably) being made by Google or other major tech players, specific support for training and subsequent application of big data within federal government is important and sorely needed. The job market for data scientists in the private sector is so hot that government may well need to build up its own internal expertise, much in the same way Living Social is training coders at the Hungry Academy.
“Big data is a big deal,” blogged Tom Kalil, deputy director for policy at White House OSTP, at the White House blog this morning.
We also want to challenge industry, research universities, and non-profits to join with the Administration to make the most of the opportunities created by Big Data. Clearly, the government can’t do this on its own. We need what the President calls an “all hands on deck” effort.
Some companies are already sponsoring Big Data-related competitions, and providing funding for university research. Universities are beginning to create new courses—and entire courses of study—to prepare the next generation of “data scientists.” Organizations like Data Without Borders are helping non-profits by providing pro bono data collection, analysis, and visualization. OSTP would be very interested in supporting the creation of a forum to highlight new public-private partnerships related to Big Data.
The White House is hosting a forum today in Washington to explore the challenges and opportunities of big data and discuss the investment. The event will be streamed online in live webcast from the headquarters of the AAAS in Washington, DC. I’ll be in attendance and sharing what I learn.
“Researchers in a growing number of fields are generating extremely large and complicated data sets, commonly referred to as ‘big data,'” reads the invitation to the event from the White House Office of Science and Technology Policy. “A wealth of information may be found within these sets, with enormous potential to shed light on some of the toughest and most pressing challenges facing the nation. To capitalize on this unprecedented opportunity — to extract insights, discover new patterns and make new connections across disciplines — we need better tools to access, store, search, visualize, and analyze these data.”
Speakers:
John Holdren, Assistant to the President and Director, White House Office of Science and Technology Policy
Subra Suresh, Director, National Science Foundation
Francis Collins, Director, National Institutes of Health
William Brinkman, Director, Department of Energy Office of Science
Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.
The hot IT buzzword of 2012, big data has become viable as cost-effective approaches have emerged to tame the volume, velocity and variability of massive data. Within this data lie valuable patterns and information, previously hidden because of the amount of work required to extract them. To leading corporations, such as Walmart or Google, this power has been in reach for some time, but at fantastic cost. Today’s commodity hardware, cloud architectures and open source software bring big data processing into the reach of the less well-resourced. Big data processing is eminently feasible for even the small garage startups, who can cheaply rent server time in the cloud.
To learn more about the growing ecosystem of big data tools, watch my interview with Cloudera architect Doug Cutting, embedded below. @Cutting created Lucerne and led the Hadoop project at Yahoo before he joined Cloudera. Apache Hadoop is an open source framework that allows distributed applications based upon the MapReduce paradigm to run on immense clusters of commodity hardware, which in turn enables the processing of massive amounts of big data.
Details on the administration’s big data investments
A fact sheet released by the White House OSTP follows, verbatim:
“National Science Foundation and the National Institutes of Health – Core Techniques and Technologies for Advancing Big Data Science & Engineering
“Big Data” is a new joint solicitation supported by the National Science Foundation (NSF) and the National Institutes of Health (NIH) that will advance the core scientific and technological means of managing, analyzing, visualizing, and extracting useful information from large and diverse data sets. This will accelerate scientific discovery and lead to new fields of inquiry that would otherwise not be possible. NIH is particularly interested in imaging, molecular, cellular, electrophysiological, chemical, behavioral, epidemiological, clinical, and other data sets related to health and disease.
National Science Foundation: In addition to funding the Big Data solicitation, and keeping with its focus on basic research, NSF is implementing a comprehensive, long-term strategy that includes new methods to derive knowledge from data; infrastructure to manage, curate, and serve data to communities; and new approaches to education and workforce development. Specifically, NSF is:
· Encouraging research universities to develop interdisciplinary graduate programs to prepare the next generation of data scientists and engineers;
· Funding a $10 million Expeditions in Computing project based at the University of California, Berkeley, that will integrate three powerful approaches for turning data into information – machine learning, cloud computing, and crowd sourcing;
· Providing the first round of grants to support “EarthCube” – a system that will allow geoscientists to access, analyze and share information about our planet;
Issuing a $2 million award for a research training group to support training for undergraduates to use graphical and visualization techniques for complex data.
Providing $1.4 million in support for a focused research group of statisticians and biologists to determine protein structures and biological pathways.
· Convening researchers across disciplines to determine how Big Data can transform teaching and learning.
Department of Defense – Data to Decisions: The Department of Defense (DoD) is “placing a big bet on big data” investing approximately $250 million annually (with $60 million available for new research projects) across the Military Departments in a series of programs that will:
*Harness and utilize massive data in new ways and bring together sensing, perception and decision support to make truly autonomous systems that can maneuver and make decisions on their own.
*Improve situational awareness to help warfighters and analysts and provide increased support to operations. The Department is seeking a 100-fold increase in the ability of analysts to extract information from texts in any language, and a similar increase in the number of objects, activities, and events that an analyst can observe.
To accelerate innovation in Big Data that meets these and other requirements, DoD will announce a series of open prize competitions over the next several months.
In addition, the Defense Advanced Research Projects Agency (DARPA) is beginning the XDATA program, which intends to invest approximately $25 million annually for four years to develop computational techniques and software tools for analyzing large volumes of data, both semi-structured (e.g., tabular, relational, categorical, meta-data) and unstructured (e.g., text documents, message traffic). Central challenges to be addressed include:
· Developing scalable algorithms for processing imperfect data in distributed data stores; and
· Creating effective human-computer interaction tools for facilitating rapidly customizable visual reasoning for diverse missions.
The XDATA program will support open source software toolkits to enable flexible software development for users to process large volumes of data in timelines commensurate with mission workflows of targeted defense applications.
National Institutes of Health – 1000 Genomes Project Data Available on Cloud: The National Institutes of Health is announcing that the world’s largest set of data on human genetic variation – produced by the international 1000 Genomes Project – is now freely available on the Amazon Web Services (AWS) cloud. At 200 terabytes – the equivalent of 16 million file cabinets filled with text, or more than 30,000 standard DVDs – the current 1000 Genomes Project data set is a prime example of big data, where data sets become so massive that few researchers have the computing power to make best use of them. AWS is storing the 1000 Genomes Project as a publically available data set for free and researchers only will pay for the computing services that they use.
Department of Energy – Scientific Discovery Through Advanced Computing: The Department of Energy will provide $25 million in funding to establish the Scalable Data Management, Analysis and Visualization (SDAV) Institute. Led by the Energy Department’s Lawrence Berkeley National Laboratory, the SDAV Institute will bring together the expertise of six national laboratories and seven universities to develop new tools to help scientists manage and visualize data on the Department’s supercomputers, which will further streamline the processes that lead to discoveries made by scientists using the Department’s research facilities. The need for these new tools has grown as the simulations running on the Department’s supercomputers have increased in size and complexity.
US Geological Survey – Big Data for Earth System Science: USGS is announcing the latest awardees for grants it issues through its John Wesley Powell Center for Analysis and Synthesis. The Center catalyzes innovative thinking in Earth system science by providing scientists a place and time for in-depth analysis, state-of-the-art computing capabilities, and collaborative tools invaluable for making sense of huge data sets. These Big Data projects will improve our understanding of issues such as species response to climate change, earthquake recurrence rates, and the next generation of ecological indicators.”
Further details about each department’s or agency’s commitments can be found at the following websites by 2 pm today:
“I think that government is always going to need help, and that’s part of the message that we’re trying to spread… government not only will need help but will become an institution that lets people help, that encourages people to help out, and has a strong connection to the citizens its supposed to serve.”-Jennifer Pahlka, talking in a new interview with CNN on geeks helping open government.
Earlier this winter, Pahlka (aka @pahlkadot) delivered a TED Talk, “Coding a Better Government,” that now has over 300,000 views at TED.com and another 40,000+ at YouTube:
That talk and her SXSWi keynote — which was nearly three times as long and perhaps that much better — aren’t just about Code for America or civic coding or the impact of the Internet on society. It was about how we think about government and citizenship in the 21st century.
Jen’s voice is bringing the idea of civic coding as another kind of public service to an entire nation now. If America’s developer community really wakes up to help, city and state government IT could get better, quickly, as a network effect catalyze by the “Code for America effect takes off.
“To build resilient, peer-to-peer cities these precarious economic times demand, these conversations and collaborations need to be facilitated top-down, ground-up, and between every other decentralized community node that can contribute to weaving a diverse tapestry of a city’s political, cultural, historical, and socioeconomic data. …
To those of us who don’t think of ourselves as hackers but find ourselves applying that ethos to other trades—journalists, community organizers, field researchers, social justice activists, lawyers and policy wonks, and many more groups—let’s join the conversation, contribute our skills to the civic hacker community, and see what we can build together for our cities.”
If millions of non-coders collaborate with the geeks amongst us, learning from one another in the process, it could transform “hacking as a civic duty” from a geeky pursuit into something more existential and powerful:
21st century citizenship in which an ongoing digital relationship with government, services, smarter cities and fellow citizens is improved, negotiated and delivered through mobile devices, social media and open data.
E Pluribus Unum 