1. Routine Data Collection
Collecting data will help you to understand if your programme is working for your service users as well as achieving the aims you set out in your planning (see vol.1 of the Impact Toolkit). Analysing this data will also help you to improve your service and maximise the impact your organisation is having on people’s lives.
Routine data collection is exactly that; routine. You will collect this data often, if not every time you work with your service users. Often it is not seen as valuable as it does not explain the short or long-term effects of a programme. This is a mistake however, as it is crucial to understanding your impact.
Routine data tells you who your service users are, how many you’re reaching, and what they think and feel about your project. Without this insight you won’t know how many of the right group you’re targeting and whether they are experiencing your project as you might expect them to.
Routine data comprises three out of the five data types:
- User data
What: Information about the people you are reaching.
Why: To check whether your service is reaching the intended target group and tell you about the people you are currently helping
How: Collected when an individual applies to take part in your programme. You may ask someone their age for example to assess their suitability - Engagement data
What: Information about how service users are using your service and to what extent.
Why: To understand whether or not you effectively deliver the service to your intended users. Key questions include how often people attend, for how long and how engaged are they?
How: Collected on an ongoing basis such as tracking attendance at events. - Feedback data
What: Information about how people think or feel about the service
Why: To establish whether your service gets the reaction you want and whether it is beginning to work in the way you intended.
How: Could be captured via social media or surveys. You may have an instant feedback mechanism where people record their interaction immediately afterwards.
A note on data systems
Effective use of data isn’t all about collection, how you record that data is an important element too.
A good data system can help with:
- Data entry
Making data collection easier and more efficient, in addition to ensuring consistency and accuracy. - Data storage
Can help to reduce paperwork, as well as giving you a place to store sensitive data in one place - Data use
Ensuring data is easily accessible to staff, and can be compiled or aggregated, and presented with reports or dashboards
When developing or updating a data system, it is important to think about your needs. What data do you want to collect? What do you need the system to do? And how can it be integrated with your day-to-day work?
If you don’t already have a data system, it’s worth considering off-the-shelf options, rather than building your own. Speak to others who have been through a similar process. If you do opt for something new, provide potential suppliers with a detailed overview of your requirements to get a sense of the cost.
Be clear about the data you plan to collect, and the outputs you want the system to generate. Outline any ongoing support you may need, particularly if you don’t have in-house IT expertise.
Further information on routine data
2. Data Collection Methods
Data collection methods are the different ways in which we gather information whether it is for routine data types or for outcome and impact data.
Not all data collection methods were created equal; some will be easier to use, some will be more suited to certain data types than others. It is important to pick the right method for the questions you want answered and the data you want to collect.
In the next few steps we will describe a selection of the most commonly used data collection methods, discussing the pros and cons of each and tips about using
3. Surveys
Surveys can be a potent tool to methodically gather both quantitative (numerical) and qualitative (non-numerical) information about a particular group. In simple terms, quantitative data provides the numbers to prove the broad points of your project, whereas qualitative research offers insight into why and how people behave in the way that they do.
Why use surveys?
Pros
- Can reach a large number of people
- Generate data that is consistent and easy to analyse
- Allows people to respond anonymously
- Easy to compare before and after
Cons
- Can be difficult to get responses
- Risk of questionnaire and form-filling fatigue
- Literacy and numeracy challenges, such as learning difficulties or those for whom English is a second language
Open and closed questions
When developing a survey, it is helpful to have both open and closed questions. Closed questions collect quantitative data and are quicker for users to answer and the results are easier to analyse. Closed questions could be choosing from a list of options, ranking options in order or selecting a point on a scale.
Open questions collect qualitative data in the form of descriptive information; for example, by asking ‘what do you feel you need support with?’ They enable users to answer freely and in detail, while providing insights you perhaps hadn’t expected.
Bear in mind the following when designing a survey:
- Keep your questions simple, focused and easy to understand
- Keep your survey short – 15 questions is a good size
- Avoid leading questions
- Ask one thing at a time
- Focus on the objectives of the survey
More information on developing a survey
4. Interviews
An interview is ‘a conversation with a purpose’, conducted in person or on the phone. Interviews enable you to gather detailed information about people’s attitudes, motivations, beliefs, and perspectives. They also allow you to investigate beyond the headline findings of a survey to explore particular issues in greater depth or seek explanation for any unexpected answers.
Why use interviews?
Pros
- Can map out a range of experiences by talking to different people
- Gain an in-depth understanding of individual perspectives
- Easier to handle sensitive topics compared with other methods
Cons
- Can be labour intensive and time consuming
- Can be intimidating for some people who feel put on the spot
Structured and unstructured interviews
Structured interviews are interviews with set questions where every participant is asked the same question and are generally shorter in length. Unstructured interviews are flexible and can be tailored to individuals, enabling the interviewer to scope out different areas of interest. Typically they can last up to two hours. These approaches can be combined into semi-structured interviews, such as allowing certain questions to be explored in more detail while others remain structured.
Bear in mind the following when designing an interview:
- Keep your questions simple, focused and easy to understand
- Avoid leading questions for closed questions
- Encourage full responses for open questions
- Ask one thing at a time
- Focus on the objective of your interview
More information on developing an interview
5. Focus groups
Focus groups are facilitated discussions with small groups of individuals. Similarly to interviews, they can be a valuable way to explore different perspectives in more depth, capturing information about personal and group feelings, perceptions, and opinions.
A unique quality about Focus Groups is that they also allow participants to interact with each other, building on ideas and exploring a broader range of issues. They are best suited to feedback, outcome, and impact data.
Why use focus groups?
Pros
- Able to understand points of debate as well as similarities/contrasts in people’s experiences
- Good for generating ideas for changes and improvements
- Stimulates a lively and interesting discussion
- Can help people feel more engaged with your organisation
Cons
- Results can’t be reported for individuals
- Discussions move fast so can be difficult to get into detail
- May not be appropriate for sensitive topics
- The most vocal participants can dominate the session
Six to ten participants is a decent size, allowing you to explore a range of experiences while also exploring issues in detail. Focus group facilitators need to be skilled in managing group dynamics and have a good grasp of the issues being discussed.
Bear in mind the following when running a focus group:
- Let participants respond on their own terms, and define the issues important to them, but keep the group focused on the key topics
- Give everyone the opportunity to comment, particularly those who may find it difficult to contribute
- Ask for feedback periodically to summarise and clarify key points
- Observe body language. If someone seems uncomfortable or visibly seems to disagree with something, ask them about it
- Confirm how and in what form you will communicate the findings of the focus group
More information on conducting a focus group
6. Observations
Observations help you consider your service from the perspective of your users. Researchers observe users as they engage with your service or in another activity in their everyday lives, developing a holistic understanding of their context. Rather than interviewing people about their experience, the researcher joins in and sees it for themselves. The aim of observations is to get ‘under the skin’ of a particular issue or viewpoint by taking a detailed and personal view.
Why use observations?
Pros
- Can be helpful in initial service design
- Helpful to understand user’s point of view
- Gain a better understanding of an unfamiliar setting or group of people
- Obtain insights where more in-depth methods may not feel appropriate
Cons
- Resource-intensive. Require a lot of time and produce a lot of data to analyse
- Requires high level of skill
- Researchers need to minimise biases and preconceptions and extract robust data and insight
A good researcher is essential when observing and interacting with service users in their real-life environment. They need to avoid all potential drawbacks, such as the detail and completeness of their observations, as well as potential bias in data collection or analysis. They need to be open-minded and able to put someone else’s viewpoint before their own. They also need to understand the purpose of why the observation is being conducted.
Different styles are open to researchers. Some may try to present an entirely neutral manner and let the subject take the lead. Others are more engaged and participatory. Both can be productive but it is important that the researcher can adapt to the subject.
Recording observations can help guarantee that nothing is missed and give subjects the opportunity to observe their own behaviour. If consent is given, they can provide valuable material for external communication. However, some participants may feel recording devices are intrusive, or they may make the observation less natural.
More information on observations
7. Creative Methods
In addition to traditional research methods, there are creative ways to gather data and feedback using drawings and videos, diaries and storytelling, and online software. These methods are less formal and, instead of a researcher asking questions, participants may be asked to document their own thoughts and feelings.
Creative methods can be less time consuming compared to interviews or questionnaires. They can be valuable in transferring power to the participant and engaging them in evaluation.
Why use creative methods?
Pros
- Makes evaluation more fun and less formal
- Put participants in control
- Helps you to better understand the user journey
- Reaches groups who may struggle with interviews or questionnaires
- Can provide a more engaging record of impact
Cons
- Self-selection of participants may bias the findings
- Difficult to conduct at scale
- Can be difficult to interpret visual or other representations
- Partial perspective because people choose how to represent themselves
- Aggregating the data is difficult
When using creative methods, it is important to reflect on the particular needs of your service users. Keep in mind the challenges of creative methods; for example, some participants may lack confidence in drawing or storytelling. Be prepared to facilitate by seeking out voices that are not being heard and making sure you can make sense of the information being provided.
When analysing the information, interpretation may require a high level of skill and grouping the data may be difficult. It is important to remember that there may be an element of self-selection in who participates, and the data may also only represent a partial construct that requires contextualisation to make sense.
More information on different creative methods
8. Case Studies
Case studies look in-depth at particular service user journeys. They are commonly used in a communications or fundraising context, but can also be used to understand your impact by exploring how change occurs and under which circumstances. They are well suited for outcome and impact data.
Why use case studies?
Pros
- Gives a detailed understanding of a particular setting or process
- Allows for comparisons between individual experiences or service delivery settings
- Can be an emotive and powerful storytelling tool
Cons
- Can be time consuming
- Examples need to handled and selected carefully as they will not be representative
Case studies for understanding impact require a different approach to marketing or fundraising case studies. Rather than looking for stories about people who most vividly illustrate the needs and benefits of your service, look for stories that give a more balanced view. You want to explore both the typical and the range of experiences of people who use your service, to gain a better understanding of the difference your project is making and how people experience it.
Your focus should be on the difference that has been achieved. Evidence is important for justifying your claims about impact, as is an understanding of context. Service users need to understand the role they will play and the ways in which their story may be used, before giving written approval via a consent form. Explain what you’re trying to achieve. Reassure them that their contact details will not be shared without their consent.
Bear in mind the following when writing a case study:
- Think about your audience: What information would they like to know and how would they like that to be presented to them?
- Reflect on the positive and negative learnings: It is as useful to learn from your story as it is to convey the good things. This also helps to make your case study feel more genuine and honest.
More information about case studies
9. Involving Staff
A key element of data collection is ensuring that your own staff or volunteers understand what they are collecting and how to do it effectively.
This is all part of building an impact culture within your organisation; getting staff to think in terms of outcomes and impact rather than activities.
Your evaluation plans, tools and processes will be more effective if they are informed by staff, users and other key stakeholders. They are also more likely to see the point of the process and collect decent data if you bring them with you.
Don’t just assume that everyone can monitor and evaluate. There will usually be some skill-building required, which will take time and usually resources. While many people learn to self-evaluate on the job, you can save time later on by having some support along the way, to help people collect the right data and know what to do with it.
Keep an eye out for skills gaps in other areas too, not just monitoring and evaluation. Some organisations struggle with implementing good impact practice because staff lack the skills in one-to-one interviewing or IT, for example.
Many organisations make the mistake of not integrating their evaluation work with other internal initiatives, like strategic planning. This may seem obvious, but it’s amazing how often people forget that impact work influences all aspects of organisational life, from job descriptions to staff meetings to strategic plans.
Impact culture is about being open to finding out what works and what doesn’t, and using that learning to make better decisions about how to target your resources. It’s also a chance to celebrate success; staff and users often really value seeing what their organisation has achieved.
More information on impact culture and practice
10. Data Protection & GDPR
Whichever methods you choose to use, the data you collect must be kept secure in order to comply with data protection legislation. All data you collect from service users will be classified as personal data and so it is important that you take into consideration why you have chosen to collect that data.
Every organisation should have a written policy and procedure that is specific to the context about how they handle personal data and enact privacy principles.
Data protection and The General Data Protection Regulation (GDPR) is a lengthy topic and could easily fill a guide in itself. There has been extensive guides and templates created and as such we will not cover the topic more in this guide.
More information about GDPR
11. Get in touch
If you need any further help with measuring your organisation’s impact, evaluating projects or would like us to run a training session for your staff, please get in touch with us.
0116 257 5025
impact@valonline.org.uk
www.valonline.org.uk/economic-impact
Twitter: @jamesS_VAL