An Alternate Approach to Nonprofit Evaluation

a. 2016 State of Evaluation Report, https://stateofevaluation.org/  b. Data based on convenience sample of 38,000 nonprofits with email addresses in Guidestar's database.

a. 2016 State of Evaluation Report, https://stateofevaluation.org/

b. Data based on convenience sample of 38,000 nonprofits with email addresses in Guidestar's database.

When I look at evaluation in the nonprofit sector, the same thought keeps coming to mind, “Why aren’t more people asking for outside help?” There is a lot of data being collected out there that ranges from being certifiably bad to somewhere in the area of “OK”. Good data collection by nonprofits seems to be on the rarer side. What strikes me, though, is that when I look at the issues that are the source of the bad data (e.g. poorly written surveys, mis-specified outcomes, outcomes that are incorrectly measured, etc), I see problems that could quickly and easily by fixed by a skilled mind. It doesn’t have to be this way! Or so I often like to shout at the world from inside my head.

The question I have to ask, though, is, “Does it have to be this way? Is this really the best we can do?” My feeling is that we can do better; however, this raises another question, “Is ‘better’ worth it?” I will be exploring these questions over a series of posts, bringing in some of my personal experience, a touch of educated conjecture, and outside research. The end goal is to paint a picture of what is possible, why we might consider doing it, what it would take to get there, and whether it would be worth it.

First, my proposal. What I would like to do that would be different from the status quo is create a centralized evaluation organization- a nonprofit that does evaluation for other nonprofits. As I envision it, the way the model would work is that once an organization or a program reaches a certain size or age, we would work with that nonprofit to build effective and efficient evaluation plans into their programming. The nonprofit would, in most cases, then be solely responsible for their own data collection, using the tools and following the protocol we developed together. At the end of a set period, the nonprofit would turn the data over to us for analysis. The process would conclude by producing an independent report after working with the organization to identify strengths and areas for improvement. It would be up to the nonprofit to take next steps and build on what was learned in the evaluation cycle. The cycle would then repeat itself, with changes made to the evaluation protocol only when necessary.

Screen Shot 2017-10-26 at 7.51.43 PM.png

In this system, the skilled work is in skilled hands while the burden to the nonprofit is minimized relative to the level of quality being achieved in their evaluation process. The system would be paid for primarily by funders, with minimal to no cost-sharing with the organizations being served. No cost-sharing would remove one of the most significant barriers to many nonprofits around seeking outside help: not having it in their budget. Because of the proliferate number of nonprofits in the state, we would have to use a system for prioritizing who gets served and who does not. This would most likely be tied to the size or cost of the program or organization, its age, and its intended longevity.

Some additional opportunities with the model include the possibility to standardize reporting across funders (at least at the state-level), the potential to standardize evaluation methods across programs within a certain program type, the ability to set benchmarks for comparing similar programs across organizations, and even the ability to begin to rate or rank nonprofits according to how well they conduct and use evaluation (e.g. do they feed insights from the data back into their programing to make improvements?). In short, we would be in a position to help streamline application and reporting processes across funders, streamline some of the evaluation process for nonprofits, and solve a basic information problem that funders always face when reviewing grantee applications: “Are they really as good as they say they are?” In theory, this would make both nonprofit’s and funder’s lives easier in the long-run (at least for those who are predisposed to liking evaluation).

To summarize, the model has the potential to help a lot of nonprofits improve their ability to meaningfully understand their programs and improve them; to streamline the reporting aspect of the funding cycle for funders and recipients alike; to remove barriers to and increase incentives for nonprofits to invest staff time and energy in the evaluation cycle; and to solve an information problem for many funders when trying to assess whether funds given to an organization are going to be well spent.

That’s the proposal in four paragraphs. There are a number of challenges to overcome here. One is the resistance many nonprofits have to evaluation. Another is connecting data collected and analyzed to actual change and improvements within an organization. A third is the logistics of the whole thing- operating efficiently at scale. And then there is, of course, the cost. An approach like this is going to be costly due to the skilled labor hours needed to execute it. I will spend the remainder of the post addressing this issue.

Cost is a substantial factor. Funders, such as foundations, individual donors and government, would likely be footing the bill. Let’s say the evaluation organization needed to service 1,000 nonprofits across the state and we were able to get the whole process down to taking an average of 100 consulting hours per program per evaluation cycle (assuming all organizations have only one program, which is not true). If we are paying an average of $35 per hour, then labor costs alone are around $3.5 million per year. In actuality, labor costs are probably going to be a lot higher.

After labor costs, we have to have office space, pay for hardware and software, pay for administrative staff, and cover travel costs to send consultants to more remote organizations (assuming the evaluation organization does not opt to have hubs in key regions of the state, like Asheville, Wilmington, & Charlotte). Further, some nonprofits will need additional funding to upgrade their systems and fully benefit from our help. In short, it is an expensive undertaking that, in order for it to be justified, would have to save the nonprofit sector at least as much money in improved program efficiency as well as in saved nonprofit and funder staff time and energy.

If it is worth it, can the industry afford it? Well, the combined income of about 1,250 organizations in North Carolina that might be beneficiaries of such an effort was roughly $1.93 billion dollars, based on IRS reported income in September 2017. These are organizations that are devoted to charity work of the sort we typically see applying for grants and needing to do evaluation. They also have reported incomes between $500,000 and $5 million dollars in fiscal year 2015 or 2016- large enough to be worth investing in but small enough that they might not yet have dedicated evaluation staff. At this amount, even if the evaluation organization cost $10 million per year to run, it would still only be 0.5% of the income of the organizations it was serving.

Funders have been asking more and more to ‘see the data’ when taking grant applications and issuing funds. However, few funders appear to be paying any attention to the quality of that data or how it is used by the nonprofit- and how can they? It takes a trained eye to spot the good from the bad from the ugly and few funders employ in-house evaluators. As a consequence, while nonprofits are collecting data as requested, the quality of that data is seldom adequate to do proper assessments and make improvements. Moreover, the current system does not enable or incentivize many nonprofits to go beyond simple reporting and actually use their data to improve their programming. I believe a centralized evaluation organization could go a long ways to improving many of these issues, and at a cost that is a small fraction of the worth of the nonprofit sector.

The next post will look at my theories around why it is better to outsource this work than rely on internal staff to do it. Later, I will break down the process more and talk through where it has efficiency gains and where it may just simply be slow. We will also look at who will benefit from this service and who won’t- certainly not everyone can. Then I will pick apart how it might be paid for and how nonprofits might be incentivized to take advantage of the service. Finally, we will look at some of the secondary benefits of having a centralized evaluation organization.