Last week, I had the opportunity to attend a lecture given by Dr. Timothy Knowles, the John Dewey Director of the Urban Education Institute (UEI) at the University of Chicago. Dr. Knowles talked about lessons learned from twenty years of data collected about Chicago Public Schools (CPS). With this data, CPS is now able to make informed decisions regarding the development and implementation of initiatives aimed at improving outcomes – they know the problem and the factors that contribute significantly to success.
The next day, I found myself at the Rayburn Building learning about the fantastic results found in out-of-school-time program provider Higher Achievement’s independent, longitudinal study. The study showed that Higher Achievement’s program helps close the opportunity gap during the critical middle school years—the time experts say is the pivot point for future success in both school and life. This report validates Higher Achievement’s work and makes the organization one of the first out-of-school-time programs to demonstrate proven academic impact for program participants.
What struck me when hearing about these evaluations is that all the data was collected, compiled, verified, and analyzed by outside parties—CPS by the U of Chicago and Higher Achievement by Public/Private Ventures. In an era when all organizations are encouraged to use data to inform decision making, can we not be trusted to measure our own impact?
Currently, most research and evaluation is being done internally – completed by those that are least likely to provide unbiased analysis. We see the associated negative outcomes all the time. Entities measure those things with which they can show positive results, leaving us with little knowledge about what actually works. Independent research bodies can hold those providing services accountable in a way that cannot be done internally. So, why don’t more of these research bodies exist?
Through a simple reallocation of resources, it seems possible. The creation of independent research institutes could provide a needed boost to many organizations. Organizations would pay for membership in the community – simply shifting money that will no longer be spent doing all of this work in house. To be accepted into this research partnership, organizations would need to have a well-articulated theory of change and logic model – important steps that are often overlooked. Membership in this partnership would lend credibility to both new and existing organizations.
In recent years, an increased emphasis has been placed on impact assessment. I believe most of the people doing this work to be well intentioned, but we know the pitfalls of self-assessment. Our current approach uses data to confirm, but we need data to inform. To take that step, we must all be courageous enough to hear the results.
But courage alone is not enough. This work is extremely expensive, so we must find ways to create data systems in which costs can be shared. Industries must have honest discussions about valued outcomes, allowing for the sharing of evaluation costs across organizations. By taking these steps, we could truly begin collecting data that could inform future decisions and encourage improved delivery of needed services.
Photo credit: geishaboy500
