Research Update: Measurement and Evaluation Practices in Canada’s Charitable Sector

Note: this is excerpted from the May 2020 edition of our “Research Update” publication. The Edmonton Social Planning Council, in collaboration with our volunteers, strives to provide stakeholders and community members with up-to-date reviews, prepared by our volunteers, on recently published social research reports and publications.

Reviewed by Shawna Ladouceur

This article summarizes findings from a 2018 survey conducted by Imagine Canada. Imagine Canada is a national organization providing programs and resources to strengthen charities, promote corporate giving, and support the charitable sector through accreditation, corporate recognition, public policy advocacy, and research.

Organizational stakeholders, policymakers, the media, and individual Canadians want to see how the charities they support are affecting the populations they serve, as well as the impact on broader systems. But given the diverse range of potential approaches to measurement and evaluation, an even wider range of possible uses for the results produced, and the incredible diversity of stakeholders and audiences with whom to engage, charities seek a common understanding around norms and expectations. Imagine Canada works with charities to establish a cohesive narrative for this increasingly important conversation, and this survey laid groundwork for the development of this common understanding with an in-depth look at the current state of evaluation.

Seeking to benchmark current practices, categories for this survey included: aspects measured and evaluated; methods employed; how results were used; the role of funder relationships; enablers and barriers affecting evaluation capacity and ability to make use of findings; opinions about evaluation and measurement; the role of evaluation-related networks; the effect of dedicated evaluation staff; and the role and effect of employing external evaluators.

An overview of survey results indicated 96% of charities evaluate their work in some way. Method and resources employed can be affected by: the size of the organization—small (<$150,000 annual revenue), medium ($150,000 -$5 million), or large (>$5 million); charitable subsector (grant making, fundraising and voluntarism; arts, culture, & recreation; education; social service; and health); and region of Canada in which the charity operates. Aspects measured include: outputs, outcomes, quality, long-term/systemic impacts, return on investment, and other measures. A range of both quantitative (administrative data, statistical compilations, surveys, web statistics, and experimental studies) and qualitative methods (interviews, focus groups, logic models / theories of change, and case studies) are employed, with measurement tools used seemingly correlated to aspects being measured.

Results from evaluation can be used in many different ways. For the purpose of this survey, these were subdivided into four major categories and eighteen distinct uses. Categories included: stakeholder reporting/ communications; organization/ program-planning and decision-making; learning; and organizational performance. The most common of the eighteen distinct uses were: reporting to the organization’s board of directors, funders, or other supporters; evaluating achievement of objectives; learning from outcomes; revising existing programs; and supporting the development of new ones.

When examining evaluation capacity and ability to make use of findings, the survey exposed both enablers and barriers. Listed in order of influence, these included: support from organizational leadership; staff buy-in; staff knowledge, skills, tools, etc.; stakeholder buy-in; funder support; external evaluators/ consultants; capacity of funded/ supported organizations; staff time; and financial resources. These factors function as barriers when absent, and enablers when present.

An examination of funder relationships found that 90% of charities surveyed received some form of external funding. Charities that received funding dedicated to evaluation tended to a more comprehensive review. Regular communication between funders and fundees appeared to be key to making best use of evaluation results, and served to promote a positive view of this relationship.

Key survey findings indicated that satisfaction with evaluation capacity and ability to make use of results were strongly correlated with enablers and barriers. Of note, opinions about evaluation did not seem to vary according to what charities evaluate, how they evaluate it, or how they use evaluation results. Involvement with evaluation networks increased overall satisfaction. Charities with dedicated evaluation staff were more likely to draw on a number of specific methods and resources, have significantly more positive opinions about evaluation, and to see potential enablers and barriers as enablers. While this survey found that charities working with external evaluators reported being less satisfied with evaluation capacity, they actually tended to demonstrate greater capacity, evaluating more involved aspects of their work and drawing on more sophisticated methods. Speculation as to this phenomenon is presented as multifactorial and seemingly complicated.

This survey provided a starting point on which to base a coordinated narrative for the conversation charities are having with Canadians. Such narrative will certainly serve to increase trust and confidence, strengthen charities, and promote giving in support of the charitable sector. Given the heavy statistical nature of this report, this review can only provide a broad overview of the large repository of valuable statistical information presented in this survey. Please see the full report for detailed analysis.

Publication Source: Lasby, David. (2019). The State of Evaluation—Measurement and Evaluation Practices in Canada’s Charitable Sector. Imagine Canada. Retrieved from: https://www.imaginecanada.ca/en/research/state-of-evaluation

Volunteer Research Reviewer: Shawna Ladouceur is a Registered Nurse who sees the impacts of the social determinants of health in ways that demand action. She has extensive experience working directly with vulnerable populations in the inner city. Her personal interests include skiing, hiking, biking, running, reading and travelling.