Involving stakeholders at each stage of an evaluation is one of the fundamental principles of SROI, and should be celebrated as one of its greatest strengths.
By Sarah Arnold, Economist, NEF Consulting.
The common thread underpinning NEF Consulting’s work is co-production. Our participatory research methods ensure our work reflects the input of those most affected, involving stakeholders at each stage of an evaluation. This is one of the fundamental principles of SROI, and should be celebrated as one of its greatest strengths as a methodology.
The very first principle of SROI is to ‘involve stakeholders’. They should be involved at each stage: determining the programme’s impact, defining its outcomes, choosing indicators to measure those outcomes, choosing financial proxies to value them, and sense checking all results to ensure they are reasonable.
In SROI, what people say matters. SROI requires a real trust in people. You must have faith in the changes they themselves identify, to take their experiences over your own assumptions.
A major strength of our work is that it is underpinned by the real experiences of those who matter most to the programme. We use participatory research methods to ensure that our work is led by the voices and experiences of those directly affected by a programme or organisation. We talk to people. We work with people.
When done well, SROI’s participatory approach is its most compelling and unique selling point.
The importance of co-production
I have seen time and again why it is important to work closely with stakeholders at all stages of an SROI or evaluation:
Identifying unintended outcomes. In-depth research with stakeholders may identify outcomes of a programme that those running or evaluating that programme might not have intended. For example, an SROI of a parenting programme in Ecuador found that, as a result of the programme, social norms around women and children’s roles in the family had begun to shift, even though they had not been working directly to achieve this.
Measuring outcomes. Stakeholders can help develop indicators that are relevant to participants. Wording is important, and stakeholders can co-produce questions that will have meaning and resonance.
Understanding impact. To understand the additional impact of a programme, beyond what would have happened anyway, we usually ask participants to self-evaluate. We might ask: “Considering your past behaviour, and how this would have changed over time, if you had not joined the programme, how many times this month would you have visited the GP?” We trust individuals to be able to judge this, and it has been shown to be a valid alternative to experimental approaches such as randomised control trials. (See the American Journal of Evaluation)
Valuation. Where no market values exist, we can work with stakeholders to understand the value of an outcome. Standard economic techniques such as willingness to pay (WTP) approaches, ask stakeholders directly how much an outcome is worth to them. Even where market values exist, they may not fully capture the social value of an outcome. For example, there are market values for health, such as the value of health insurance, but many people will consider their health to be more valuable than the cost of insurance. Such approaches are also useful for less tangible outcomes that are hard to value. See this approach in our work with CARE .
As SROI involves co-production throughout the process, each SROI is unique. We recently conducted an evaluation of two employment programmes. Both worked with mothers to get them into the job market. One programme worked through helping mothers re-join the labour market by identifying flexible jobs that paid a living wage. The other worked with mothers who were very far from the labour market, many of whom didn’t speak English as a first language, and had never written a CV or had a job interview. Although the final intended result was the same, the programmes worked differently: the first signposted opportunities and provided support with job applications; the second focussed on group social sessions and support, before introducing job opportunities. We worked with participants of each programme to generate a unique set of outcomes demonstrating the pathways people take towards the ultimate goal.
Bespoke is beautiful
Of course, there is also a flipside. SROI’s focus on participatory research can be expensive, time-consuming or both. But, although conducting an SROI involves a lot of effort, it is worth it. The stakeholder participation that makes the process costly also gives the work legitimacy, and provides the most important insights.
SROI only values what matters to stakeholders, it does not dictate a specific set of outcomes or benefits that could be valuable. Any SROI will involve multiple judgments and assumptions, and these should be transparent, but not standardised. This may lead to difficulties when comparing SROIs and can add to the complexity of undertaking an SROI. Currently, SROIs are only directly comparable when they share a common underlying outcomes framework and all results and judgments have been checked and calibrated against each other.
As SROI develops, some are working towards standardisation. This should make SROIs more accessible and in that sense should be welcomed. But standardised frameworks often miss why a programme is important, or how activities were delivered. These factors are critical to a project’s success and understanding how and why a programme works is just as important as quantifying the extent of success.
Standardisation should not come at the cost of SROI’s greatest strength. SROI must continue to be flexible and allow the voices and experiences of those who matter to be heard. After all, we already have standardised cost-benefit methodologies – why not celebrate SROI for what it does best?