Measuring impact
What is Impact?
Impact is the long term effect of the activities you carry out and the differences you want to make in the community. When planning and designing projects and activities for your organisation the first thing you will need to consider is what the problem is that you are trying to solve, who for, and in what sort of ways. Your impact is the effect that your activities will have on solving this problem.
Impact is difficult to demonstrate as it goes beyond simply stating what you do, the amount of something you do, or the amount of people you support.
What impact is not?
Impact is not what you do. These are the activities that you carry out in order to work towards achieving a goal. Impact is not the amount of something that you do, for example, running 10 play sessions for 12 children per session. That is an ‘output’ and is generally numerical.
Examples of impact?
It is important to think about who your interventions are for and what they are for.
Examples of the impact of running a series of children’s play sessions could be:
- Increased confidence in the children.
- Increased integration into other services for the parents.
- Improved educational attainment for the children.
Why measure your impact?
- To understand the difference that your project is making.
- To understand the achievements of the project.
- To understand the achievements of the people participating in the project.
- To use as a marketing tool /get your group known for the work that you do.
- To use project learning to develop new funding relationships or sustain funding from your current funder.
Outcomes
The difference between impact and outcomes
Impact and outcomes are related but different.Outcomes refer to the changes directly resulting from an organisation’s activities. They are generally short to medium term in nature and directly relate to the activities.
Impact is different as it refers to broader, long-term change. It is harder to measure than your outcomes.
Smart Outcomes
An outcome is the change that happens for people during your project. Most funders use the SMART method to appraise outcomes. This is a method which assesses the effectiveness of your intended outcome in bringing about your intended change.
Specific: Your outcome needs to be clear and directly related to the issue you are exploring
Measurable: Your outcome needs to be something which you can measure so you can track your progress
Achievable: Your outcome needs to be realistic – something that can be done
Relevant: Your outcome needs to be directly relevant to the change you are trying to make
Time-bound: Your outcome should have a finish date by which it should be achieved
Example of a SMART outcome for children’s play sessions:
The percentage of children who have attended 3 or more play sessions over a 6 month period who report that they feel more confident as a result of attending.
An evaluation framework
An evaluation framework is something that you can create that will help you to think about your impact and how you demonstrate it. A simple evaluation framework will have 3 elements. It is helpful to work backwards through these:
- The problem: What is the issue or question you are trying to solve?What impact do you want to have
- Outcomes: What are the changes you want to bring about that will contribute to solving the problem
- Indicators: What are the things that will show you that you are working towards your outcomes?
| Indicators |
Outcomes |
Impacts |
|
Children make repeat visits to play sessions.
More play sessions are scheduled.
|
Children show increased confidence and self esteem.
Children show increased creativity and problem solving.
|
Children have the best start in life and are ready to succeed. |
|
Increased number of local spaces are used for play sessions.
Parents take active part in play sessions.
|
Stronger connection and appreciation of space and place.
Stronger connections between community members.
|
Stronger and more resilient communities. |
A theory of change
Developing a theory of change is an effective way of mapping out the steps and activities that you will need to deliver in order to have the impact you want to see.
It is a collaborative and participatory process which ideally involves stakeholders and service users. Part of its effectiveness is that it includes all participants in active involvement in both developing the outcomes, the planned impact of a project and then evaluating whether it is meeting those intended outcomes.
A theory of change starts with identifying the desired impact of a project and then works backwards to fill in the activities that will lead to that impact. Start by identifying the ‘problem’, the issue or question that your project aims to tackle. By going through this process you will develop an evaluation framework which will demonstrate your impact.
A theory of change often has 5 stages
- Inputs: The resources (time, people, money, equipment) needed to achieve your desired outcomes.
- Activities: The interventions (the things you do) which will bring about your desired change.
- Outputs: The results or deliverables that lead from your activities and contribute to your desired outcome.
- Outcomes: The short and medium term changes which will support your long term goal.
- Impact: Your long term goal which will be related to solving the ‘problem’.
Find out more information about what a theory of change is and how it can be used.
Measuring your impact
Before you start planning your project you should think about how you will monitor and evaluate your outcomes. This can support you in your fundraising efforts.
If your indicator is the number of socially isolated older people who go out of their flat more than they used to, you will need to know how many do not go out more than once a week at the start of your project. Then if they go out two or three times a week by the end of your project you can demonstrate that you have achieved your outcome.
Other ways to gather information to measure your outcomes:
- Starter forms (to gather baseline information).
- Feedback forms during the project.
- Feedback forms at the end of the project.
- Interviews with individuals or groups.
- Film and photographic evidence. You must obtain permission from individuals featured in this evidence. For children, parental or guardian consent is required, and for certain vulnerable adults, guardian consent is necessary.
- Observations by staff and volunteers.
- Case studies.
- Other data (e.g local stats and survey results, NHS fingertips database, government information on benefits claims and employment levels).
Monitoring and evaluation
Monitoring is the ongoing process of collecting, analysing, and using information to track a project’s progress toward its goals. This is different to evaluation which is the systematic assessment of a project’s overall effectiveness, impact, and relevance after a certain period.
- Answer:
In order to measure the impact that your work has you will need to collect information at the beginning and end of a project to measure progress. One way to do this is to design your own feedback form which asks questions from your service users. You should let them know how you are going to use the findings. If it contains any personal data you will also have to explain how you are going to store the data securely. Survey Monkey and Microsoft Forms can help you to create easy online forms. There are many existing tools that you can use to capture information such as:
- Strengths and difficulties questionnaires. These are behavioural screening questionnaires for children aged 3 to 16 which can be used at the beginning and the end of a project to measure progress. All versions can be found on the Youth in Mind website.
- Warwick-Edinburgh scale. This widely-used scale can be used to evaluate projects which aim to improve mental wellbeing. It is free to registered charities and Community Interest Companies and can be used at the beginning and end of a project to measure progress.
- The Outcomes Star. This is a family of evidence-based measurement tools using a person-centered approach, tailored to specific sectors such as mental health, homelessness, community, education or adult care. You can see a full list on their website. Everyone using the Outcomes Star must have a licence and have completed their core training, so it can be expensive to use in small projects.
Try to set aside enough time to analyse the data you collect. You will need systems in place to respond to the findings if you find anything challenging. If the data shows that you aren’t making the difference you hoped, think about what steps you can take to make changes to the project.
- Answer:
The actual act of collecting data can be done by the project workers and/or volunteers. This can include recording attendance at activities or events, conducting surveys and doing interviews.
This data can then be analysed by project workers or service managers and included in reports or fundraising bids. There are no rules here about who does what; it is very much dependent on you and your organisation. It is also possible to commission an external organisation to do the work of monitoring and evaluation.
The work of monitoring and evaluation can be done both internally and externally to the organisation.
- Answer:
An external evaluation can offer a thorough and independent evaluation of your project. This can be very useful to demonstrate your impact and has the following benefits:
- It is independent.
- It would be run by experts if you engage the right external evaluators.
- It allows you to focus on the delivery of your project.
The main drawback of this method is that it can cost more money than running your own evaluation. If you are planning an external evaluation then you should write this into your budget at the fundraising stage. Funders are often happy to see that you have considered this at an early stage as it means they will have the information they need to know if their funding has been impactful. Even if you have an external evaluator you will still need to agree your outcomes and outputs, how to measure them and regularly collect the data to evaluate. If you don’t it is like inviting a chef to help you make dinner but not providing any ingredients.
Resources
External evaluators
Tower Hamlets CVS has used these evaluators before:
You should always research providers and seek quotations to make sure you get a service that is suitable for your needs.
If you have any questions about any of the information in this resource, THCVS can give 1-1 support and advice. Please email us at info@thcvs.org.uk.