One of the most common and significant questions we are asked by social impact organizations of all sizes (from large global foundations to small regional nonprofits) is: How can we accurately and reliably measure the impact of our work?
Comparatively speaking, the private sector has it easy. Private enterprises can use two simple financial metrics – revenue and profit – as a baseline measure for their overall performance.
For social impact organizations, measuring impact often results in a sea of proxy metrics that requires significant time and effort for most comprehend. Too often, developing the social measurement strategy becomes a time-consuming exercise rooted in abstract concepts and opinions that requires several rounds of edits only to result in a mediocre output that scratches an itch, rather than solves a problem.
Since the output of these measurement strategies are often used to attract and retain funding, the stakes are too high for nonprofits to not have a well-defined approach to measuring their outputs, outcomes and impact. Here’s how we suggest you start (or restart, if things aren’t working for you today):
First, Set Your Expectations
Due to the complexity of social impact measurement, many seek a repeatable metric or set of metrics to measure impact across programs and organizations. The truth is, there is a great deal of subjectivity of throughout the process of creating social impact. That subjectivity combined with a variable definition of success, means there is not, and there likely never will be, a ‘one-size-fits-all’ metric or set of metrics for social impact.
Additionally, it is important to accept the fact that it may take between a year and a decade to assess overarching outcomes and impact.
Next, Define or Reconfirm Your Theory of Change or Practice
The Theory of Change or Theory of Practice logic models provide basic outline for how your program will deliver upon long-term objectives. And they don’t just stop there: if constructed carefully, these logic models should also provide a basis for monitoring organizational performance and process effectiveness.
Identify a Small and Simple Set of Measures
Rather than attempting to identify a constellation of related metrics and prove causality straight out of the gate, Formative recommends organizations looking to measure impact of their digital efforts start small and simple – between five and ten KPIs. The fewer metrics, the less data and collection required to get measurement off the ground.
Research from Stanford’s Environmental and Social Responsibility Global Supply Chain forum identifies four best practices to select a short-list of KPIs:
- Can be measured regularly (e.g., quarterly) with high degree of accuracy
- Consistent with organization/program mission and objectives
- Mutually exclusive, uniquely relevant indicator
- Comprehensive: covers inputs, outputs and quality measures
For those that already have a long set of KPIs being measured, prioritize each of them to make your list more manageable.
Organize Measures into the Four C Framework
With constant love and attention, over time your short list of measures will grow into a larger set of KPIs that will require organization. In the more likely scenario, you already have a long set of KPIs that lacks any sort of formal organization.
In either scenario, Formative recommends organizing digital social impact measures around the four C’s: Creation, Consumption, Credibility and Change.
- Creation – How effective and efficient is your program or organization at creating digital outputs (e.g., content, tools)? Understand your organizational performance by documenting your inputs (e.g., total budget, staff) and measuring your direct outputs in terms of both time (e.g., frequency of output) and money (e.g., cost/unit of output).
- Consumption – To what degree are target audiences consuming your output? Look deeper into traditional reach metrics to understand the type and quality of reach attained, cut by multiple factors (e.g., organic/paid, channel, satisfaction).
- Credibility – How credible and valuable is your output and brand? Consider the credibility of not only the program as a whole, but each component of the program itself (e.g., content credibility, partner credibility, program brand credibility).
- Change – What intended or unintended changes have occurred in your target audience? Identify intended or unintended changes in not only target audiences’ situations or behaviors, but also within the larger construct of the issue (e.g., community satisfaction, incident rate, legal/policy change).
While the Four ‘C’ framework may not address key questions such as causality, we believe it will get you more than 80% of the way towards defining a robust, repeatable measurement framework.
Case in Point
For example, a recent client aimed to improve primary and secondary school student outcomes by driving increased teacher collaboration, nationwide. Rather than directly attempting to measure for student improvement or increased teacher collaboration, we took a far more piecemeal approach that directly aligned with their Theory of Change. We began with assessing basic reach of their program and engagement with their content (Consumption). As a next layer, we measured platform-level collaboration (e.g., shares, comments) and partnership referrals (Credibility). Finally, as a beginning to the last layer, we recommended point-in-time surveys to understand general perceptions (e.g., depth of engagement, incentives, peer learning) of teachers that were active in the program versus those that were not involved in the program at all (Change). Still, none of these measures tie directly to measuring the impact the program had on teacher collaboration nationwide, but they served as indirect proxies to provide an honest assessment of activities to date, until larger longitudinal studies and correlations could be completed over the next several years.
Putting it Together
The “end outcome” of your Theory of Change may be difficult and/or impossible to measure. However, indicators of the intermediate steps within the Theory of Change will provide a good, representative method to show progress on end outcomes. Tying these outcomes to the Theory of Change is critical for measurement and also critical to test the theory itself. Good measurement ensures that you are reviewing and revising your Theory of Change over time to reflect the “facts on the ground”.
- This article is only scratches the surface of digital social impact measurement.
- Many deeper questions were left unanswered, including:
- How can you measure the ROI of social impact efforts?
- Where does all this data come from?
- How can you convey measures consistently (e.g., dashboards and reporting)?
- What is the right size of personnel to focus on measurement within my organization?
- What skills should I look for?
Subscribe and stay tuned to future Formative posts on measurement that will dive deeper into some of these questions and more in the coming months.