Start Here: A Step-by-Step Guide to Impact Evaluation

Tuesday, June 16, 2020

By Andrew Dunn
Manager, Financial Health Network

A commonly asked question after designing a new product or solution is “Does it work?” The best way to provide tangible answers to questions around impact and “what works” is through impact evaluation. 

“Impact evaluation” is a broad term that different stakeholders use in different ways. Academics think of impact evaluation as experimental, causal research, but there is a broader suite of methods that can be informative for industry leaders – it’s all about conducting the right type of research to provide your organization with the answers it needs. Understanding why you want to conduct impact evaluation is the first step in determining how you should do it.

Brightside is a financial assistance platform helping employees with their finances to improve their quality of life.

Manifest works with employers and retirement providers to help workers keep their retirement accounts consolidated as they switch jobs.

Brightside and Manifest are conducting impact evaluations using their existing data to understand how their product offerings impact their users. 

  • Pros: Data already exists and is fairly accessible for analysis. Can be quick, as the only limit is internal time/capacity.
  • Cons: Lack of data on nonusers, which means lack of insight into existing external trends. Limited in causal claims, meaning there may be greater uncertainty about whether the product is the reason for the outcomes observed.

For those interested in bringing impact evaluation to their organizations, the Financial Health Network can share lessons learned from experience and walk you through the steps to get started. In this article, we also outline examples of impact evaluation projects with several participating companies in our Financial Solutions Lab, as well as the pros and cons of different approaches to impact evaluation. 

HoneyBee is a PTO small-dollar loan provider for employees.

HoneyBee is analyzing its existing internal data and user surveys to understand how users’ financial situations change as they use loans from HoneyBee.  

  • Pros: Fairly quick and simple to conduct. HoneyBee can ask a wide variety of survey questions to better understand its user demographics and gather data beyond what it already collects.
  • Cons: Limited to users. Limited in causal claims.

Steps to Conducting Impact Evaluation

  1. Determine what you want to measure. Organizations often conduct impact evaluation analysis to understand how to refine their products and services to better serve their customers, or to develop success stories about their current offerings. You should plan how you can effectively use the knowledge you’ll gain from an impact evaluation.
  2. Design your evaluation project. Do this by understanding what data is already available for use, how you can collect new data, your internal capacity to work with data, your timeframe for data collection and analysis, and the desired level of complexity for your evaluation. The answers to these types of questions will help you determine what your evaluation will look like at a high level. 
  3. Design an evaluation plan. This detailed plan specifies exactly what you’ll be measuring, how you’ll measure or analyze it, and which data will play a role. One important consideration at this stage is the project timeline. How much time do you have available to track changes over time? How much time do you think is realistic for you to see impact? Only you know the best timeframe to see impact, but when in doubt, the longer the timeline, the better. That way, you can be certain you’re capturing all the necessary data.
  4. Implement your evaluation plan. During this phase, you’ll collect the necessary data over the specified time period, and then analyze your results. Check that your data is accurate and complete and determine if your hypotheses were met. Often, these evaluation projects will bring questions as well as answers, but having a better understanding of the impact of your products and services will ultimately mean you’re better informed.

MedPut is an employee benefit that pays for, and negotiates, unpaid healthcare bills.

Onward allows employees to make regular contributions to a savings account directly from payroll and, based on their contributions, gain access to a secured credit line for use during unexpected emergencies

MedPut and Onward supplement their internal data with survey data from both users and nonusers of their products to understand how their financial circumstances change over time.

  • Pros: Can compare data on users and nonusers to help understand the differences between them. 
  • Cons: Response and selection bias can limit causal claims. Participants opt in to take the survey, meaning the companies cannot be certain if the changes they see are due to their products or to something unique about the survey-takers.

Impact evaluation is a valuable tool for understanding the impact and efficacy of your organization’s products and services. Impact evaluation is most valuable when you have an effective plan and organized accounting of your available data. By following the steps in this guide, your organization can use impact evaluation as a valuable tool to understand the impact and efficacy of your products and services.

Even’s Daily Money Platform gives employees visibility into their earnings, access to pay when they need it, and the power to plan for the future – all long before payday.

Even is conducting a “randomized controlled trial” in which new users are randomized to see different versions of Even’s onboarding flow (also known as “A/B testing”). Even is tracking whether differences in onboarding flow impact the amount that users save. The randomization in the project controls for known issues, like selection bias, where only users who are already financially healthy engage in savings behaviors. This evaluation design will allow Even to say with certainty that changes in savings they see are caused by differences in the onboarding flow.

  • Pros: Rigorous. Supports causal inference, meaning we can be fairly certain that changes we observe are due to the onboarding flow.
  • Cons: Requires some technical knowledge to implement. In this instance, access to different onboarding flows is randomized.

Want to Learn More?

Learn about the Financial Health Leaders Lab, featuring organizations that are measuring the impact of products and services on financial health.

Stay tuned to hear more about the Financial Solution Lab’s impact evaluation project here.

Contact Financial Health Network Consulting to discuss your own impact evaluation project.

Join Us

Subscribe to a Better FinHealth Future, Today

Stay on the forefront of financial health news. Subscribe for access to content by leaders and innovators and invitations to digital and live events.