Measuring Impact Toolkit – Planning

1. Understanding the issue

Understanding the issue you want to address is the starting point for any impact planning. Doing this brings clarity about who you are trying to help and why.

To fully understand the issue you should ask yourself the following:

  • What is the scale and scope of the issue?
    Think about how many people are affected by the issue and who they are. Are some groups more affected than others? How widespread or significant is the problem?
  • What is the cause behind the issue?
    Think about how is it affected by individual capacities and relationships (e.g. being a child or having a health condition), institutions and professionals (e.g. schools, hospitals, government, doctors, and teachers), and the wider environment it belongs to (e.g. policies, rules and regulations, and public opinion).
  • What are the consequences of this issue existing?
    Who is affected by these consequences? Who is particularly vulnerable?

An important step in the process after answering these questions is to identify what facts or research you have to back up your claim that there is an issue to be solved. This is called evidencing your need. Evidence can range from what you’ve already collected in your own evaluation, to large-scale national research. If you’re getting started, national public datasets are a good place to begin. Below are some good sources of information:

2. Your target group

Your target group, or beneficiaries, are the individuals who will benefit from the work you do. Defining these individuals can clarify who exactly you are targeting with your interventions, helping you to design your services more appropriately and effectively. The work you will have done on your need in Step 1 will help you to describe your target group in this step.

To begin, think about whether you work with beneficiaries themselves such as individuals, families or communities to achieve positive change, or do you instead target the systems and infrastructure that surround those who are affected to campaign for change?

If working with the former, defining your target groups is all about noting individual’s key characteristics and their specific needs. These characteristics can also be used as inclusion and exclusion criteria for your beneficiaries. If you are working with young people for example, what is the upper limit on age that you will work with?

Basic information you could use to define your group are:

  • Age range
  • Gender
  • Cultural and religious demographics
  • Where they live
  • Qualifications they hold/do not hold
  • Attitudes
  • Behaviours

To improve your description further, delve deeper into the detail of your target groups by including more contextual information about their profile such as socio-economic group and employment status. Pull out different sub-groups such as whether they are single parents, children excluded from school or homeless.

Further advice on segmenting and targeting your group can be found on NCVO’s Knowhow pages.

3. Your mission statement

Your organisation’s mission statement should be aligned to your overall long-term aim that you want to achieve for your beneficiaries or society as a whole. It is the distilled essence of your organisation’s values, beliefs and what you are working towards.

This statement should clearly articulate what you do while avoiding jargon and being excessively long. The best mission statements are clear, concise and above all, useful to the reader.

A mission statement is simultaneously your own internal guide to your long-term ambitions as well as the public’s – so make sure it is comprehensible.

The reason creating a mission statement is part of impact measurement is that we cannot start measuring until we have clear achievable goal in front of us.

The best mission statements are usually comprised of three elements:

  1. A cause or who you serve
    What matters? Who is important?
  2. An action
    What are you doing?
  3. A result
    What change can you see?

One of the key points that organisations often miss is that your mission statement should enable you to say when you would be happy to close your charity’s doors, or in other words when you have achieved your goal. If you don’t have an obvious end goal, your mission statement might be too vague.

Here are some good example mission statements:

charity:water: “Bringing clean, safe drinking water to people in developing countries.”
Invisible Children: “To bring a permanent end to LRA atrocities”

Further advice on creating impactful mission statements can be found on NCVO Knowhow pages.

4. Defining success

So far we have defined an issue for a particular target group and outlined our organisation’s values and long term goals. Now it is time to define what success looks like for your target group in the context of the particular issue you are tackling. These successes are what we call outcomes.

A clear articulation of what you want your project or service to do is the only way to plan activity that will help you measure and achieve those goals. To agree your goals ask yourself these questions:

  • What long-term change or impact are you aiming for?
    Think about what sustained effect you would like to have on individuals, families communities or the environment beyond the project’s scope.
  • What shorter-term changes or outcomes are needed to produce that long-term change?
    Think about what changes you expect to see in beneficiaries along the way to the long-term goal. For example you may see changes in their behaviour, skills, attitudes and knowledge.
  • Does existing research tell you anything about how to achieve these changes?
    Investigate whether there are any studies you could draw on by government, academics, or other organisations delivering similar services to you. Think about what you need to equip your users with so that they can achieve the long-term change you are aiming for.
  • What best practice can you draw upon?
    Other organisations delivering similar services may share their insights and learning. There may also be useful examples of best practice outside of your specific area of work or sector too.

Further useful resources on outcomes:

5. Defining your activities

A clear definition of success should be followed by a plan of how to get there. Planning out the activities your organisation needs to deliver to your beneficiaries to bring about change will help you see any missing links in your service as well as help to frame your evaluation.

Ask yourself the following questions to help define your activities:

  • What type of service do you provide?
    Is it best described as counselling, skills training, education or campaigning? Think about how often you offer your service and where. Is this service run by volunteers, staff or external organisations? Are there any specific qualifications or skills they require?
  • How do you find individuals to engage with your service?
    Describe whether they are referred to you by other organisations or whether you conduct outreach activities in certain locations.
  • How would you like people to experience your service?
    Think about how you would like delivery to feel to those individuals in practice? Would you describe the need for people to feel safe, listened to, supported and able to be themselves?
  • What is distinctive or unique about your service?
    Think about what makes your particular activities successful. What are your organisation’s strengths? Do you offer things such as tailored one-to-one support or are your services more flexible than others?
  • What quality standards will you work towards and measure?
    For example, if you deliver educational activities are there national or even international standard you should be complying with? Or if you work on mental wellbeing, are there existing measures you could use to identify change and compare with other interventions?

Collecting engagement data and feedback data will help you to establish the answer to question two – it will show you how your users currently experience your service. For more information on data types:

6. Collecting outcome data

Having identified your goals and defined success for your beneficiaries, you now need to consider what further information you need to collect to help you evaluate your service.

Your first port of call should be to review your existing data. Think critically about what you already have – how consistent and accurate is it? How are you using the data and do you really need it? Are you missing anything important that will tell you more about your service user’s progress to your outcomes?

Next, you need to decide and prioritise what to collect. What information would help you to understand and improve what you do?

  • Think about WHO you are working with
    Are you reaching the right people?
  • Think about HOW people are engaging with the project
    What do you want to know about what people think of it or how they use it?
  • Think about WHAT difference it makes
    What would you need to know to tell if your activities are working? How often would you ideally review this in order to respond?

Think about your data in 5 categories. The first three are routine day-to-day data, the last two are occasionally collected.

  1. User data
  2. Engagement data
  3. Feedback data
  4. Outcome data
  5. Impact data

More information on data types from NPC.

To help you map out the data you intend to collect it can be helpful to create a data collection plan using the five types of data as a framework.
Inspiring Impact have created a useful data collection plan worksheet that you can work from as well as data prioritising worksheet.

NCVO – Collecting Quantitative Data
NCVO – Collecting Qualitative Data
Take Inspiring Impact’s Data Diagnostic questionnaire

7. Information for funders

One of the key things not to forget when developing your impact measurement is those who fund you. Your priority in the first instance is to be clear about what information you need to prioritise for your funders as well as your own internal learning.

Ensure you review your grant or contract documents to collate your external reporting requirements with your own meaningful internal reporting metrics. This is best done at Board level to ensure oversight and that the entire organisation is involved.

If you’re reporting to a funder check their requirements before you get stuck into impact measurement. Do they want a large, full report on the whole project? Or do they want something short and outcome focused – perhaps even more of a case study? This will direct the rest of your measurement.

Remember: sharing your evaluation findings with the organisations and individuals that fund your work isn’t just about bureaucratic reporting; it is our collective responsibility to account for how we spend other people’s money and to demonstrate that it is has been used well. The more lucid detail and value that you can showcase through effective impact management, the better.

More information about ensuring your Board is tracking the right things can be found here.

8. Data collection tools

Different data collection methods and their benefits will be covered in part two of these series of guides. However, before you fully decide which data collection methods to choose you need to think carefully of the needs of your project, organisation and stakeholders.

  • Depth of information required
    Think about how much information you need, and at what level of detail – you can look back at your outcome indicators to help with this. If you need in-depth, qualitative data, interviews may be helpful. If not, you might choose a method that collects quantitative data such as a questionnaire.
  • Sensitivity and complexity of the issues
    Do you need information on sensitive or complex issues such as people’s life style choices or behaviours? Consider which method will be appropriate for asking questions on these.
  • Time and skills required
    How much time and skill are you and your staff able to allocate to designing and using data collection tools? For example, focus groups need skilled facilitators. Also consider how much time you will be asking people taking part in your data collection to spend on this, and whether this is realistic and proportionate depending on the nature of contact you have had with them.
  • Ease of collection and analysis
    Think about how straightforward it will be for you to collect and analyse data. Interviews and focus groups are likely to be more complex to record and analyse, but you may decide it’s important to spend time on this if you want to collect qualitative data.
  • Credibility
    Will the methods you use be acceptable to the stakeholders you will be reporting to? If not, the information you collect may be questioned. Some tools are well established. For example, interviews and questionnaires. Consider if these methods are appropriate.
  • Is the collection method appropriate to the service and its values?
    It’s important to consider which tools are most appropriate to your organisation and its values.

In some situations, more informal methods or anecdotal data may be acceptable. In others you may need to use methods that are tried and tested, such as a validated questionnaire recognised and used in a particular field.

  • User views and rights
    Before you collect information from people, you should make sure they know why you are collecting it, and how you will use the information. Make sure that users have consented to the way that you are collecting the information and that they understand you will treat information confidentially.
    Think about any other needs that users might have. These could relate to their learning, sight or hearing abilities, cultural background or other circumstances
  • Reliability
    A reliable method is one that can be applied consistently each time you use it, in different situations and with different people. It is essential when you are comparing information over time or between different participants and/or within different situations. If the question you ask can be interpreted differently by different people, your data may not be reliable.
  • Validity
    Are you measuring what you intend to measure? To what extent, for example, are you relying on selective perception, rather than cross-checking through a number of data collection methods? How far do the questions you ask through your data collection tools provide valid evidence?

Choosing the right method of collection takes a bit of thought, but selecting the correct one will benefit you immensely through the quality of data you collect. Always ensure you refer back to your definition of success and ask yourself: will this data show us that this has been achieved and to what extent?

9. Measurement framework

The next and final step is to bring together all that we’ve discussed into a coherent measurement framework. This is where we will outline what indicators we are using for outputs and outcomes, who will be in charge of measuring and at what point in time. This is the very practical result of your planning work.

First, write out your projects aims and activities related to those aims. We’ll use an example of a service to support ex-offenders into work:

Short term aims and activities

Next, define these activities into outputs, things produced as a result:

Activities and outputs

After this you can think of how you measure this output, or the indicators to show that something has been delivered:

Outputs and products

Now we will move on to outcomes and do this in a similar way. Take your organisation’s aims and write out the outcomes or the change you need in your beneficiaries to get there:

Short term aims and outcomes

Some of the most important outcomes from your work may be ‘soft’ outcomes – internal changes to the way people feel or think that are impossible to observe directly such as the above – increased motivation.

Measuring these changes is difficult to do directly, so instead think about what outward behavioural changes you can readily see from the individual and measure those.

For example, we assume that those ex-offenders with increased motivation to job search will spend more time doing it but will also feel more hopeful and positive about the future. These are our measurable indicators.

Outcomes and indicators

One thing to remember with outcome indicators is that you need some sort of baseline to rate the change against. For example with the amount of time spent on job searching, you will need to know how much time was spent on this before your intervention. Similarly when rating how positive ex-offenders feel you would need to ask that question before you start activities with your beneficiaries. How you choose to pose those questions is up to you to decide.

Long-term Impact

Measuring long-term impact is a much more complicated process than measuring your outcomes. You will need to commit a much lengthier amount of time to do it and to account for other factors that may have contributed to it.

In our example, the long-term impact is improved access to employment for individuals leaving prison in the UK. It is easy to see with this long-term aim how other factors may come into play beyond our own project such as other organisations programmes, the status of the job market and the changes to government policy among others.

In order to track impact in a meaningful way, you will need to situate your work in its context and understand the role that other organisations and wider factors had in contributing to impact (attribution).

It also includes understanding how much change would have happened anyway even if your organisation did not exist (sometimes referred to as the counterfactual, or deadweight). Getting to grips with these issues often requires a more sophisticated approach to evaluation and a more detailed research design.

However, you may decide to set indicators for keeping track of your impact if you feel that you can measure longer-term change in a meaningful way.
In this case, review your impact or mission statement and think carefully about which indicators you would need to collect information against. For example, suitable impact indicators for tracking improved access to employment for people leaving prison might be:

  • Percentage of ex-offenders entering the workforce after leaving prison
  • How soon ex-offenders entered work after leaving prison
  • The type of work ex-offenders take on after leaving prison

Long-term impact measurement is a constant learning exercise, so do not worry if you do not have all of the solutions to reporting it straight away.

Finally, now that you are clear about which pieces of information you need to collect, to finish off you measurement framework you will need to decide:

  1. How to collect that information (data collection methods covered in part 2 of this guide)
  2. When you will collect it
  3. Which staff member(s) will be responsible for collecting it
  4. How it will be used

The first three questions are crucial to building a clear action plan for data collection. The fourth is also important to help you double-check that you are not setting out to collect information that may end up not being used.

For further guidance please see our next two guides in the series covering data collection and analysing/assessing data.