I was flattered to have been invited back to Japan in April to speak at the 2016 Social Investing and Corporate Social Responsibility Forum, held at Meiji University in Tokyo
Japanese colleagues specifically asked me to touch on the following issues:
- How do the various Social Impact Bond (SIB) players identify and structure outcome metrics?
- Can you structure outcome metrics in a way that is not motivated directly by budgetary savings but by social wellbeing?
- What data do the various SIB players need to collect, and how do they analyse and use the information?
- What are some lessons that OPM have learned from evaluating SIBs?
This is the first, in a series of blogs, summarising some of my key messages.
How do the various SIB players identify and structure outcome metrics?
There are a number of factors to consider when identifying metrics for SIB-funded programmes. However, this is an art not a science and it should support – not obscure – the achievement of meaningful outcomes.
The ‘3 Ms’
I work to three key principles called the ‘3 Ms’, namely, metrics should be:
a) While SIBs are focused on outcomes, for contracting purposes, the duration of SIBs has to work for the outcomes payers, social investors and service providers. Hence, metrics are usually some form of intermediate or proxy indicators. There should be a compelling rationale to believe that if these intermediate/proxy outcomes are generated, then it is plausible that the longer term desired outcomes are likely to be achieved.
b) Metrics should be easily interpretable. What does it mean if an indicator goes up, stays the same, or comes down? Take ‘reporting of crime’ as an example; if crime reporting goes up, is it because there is more crime (which is bad) or is it because people are getting better at reporting crime (which is good)?
a) Can the outcome be measured consistently and robustly? Where it is not already collected routinely, what are the resource implications for collecting the data, and are there tools and processes for collecting the data well.
b) Do we have the systems in place to support good measurement?
As SIBs are based on the ability to pay for stated outcomes, there needs to be some mechanism for pricing those outcomes. It is common to find outcomes being priced based on some projected savings resulting from those outcomes being achieved, but outcome pricing does not always have to stem from budgetary savings.
Roles in identifying and structuring outcome metrics
Social investors, outcome payers, service providers and intermediaries are all very diverse and have different motivations, so it can be hard to generalise. Crudely speaking, their roles and significance of their roles can vary depending on the type of SIB.
Outside of the UK, individually-negotiated SIBs are most common. This type of SIB means that the outcome payers often work very closely with service providers and sometimes with the help of external intermediaries to help them define and structure outcome metrics.
In the UK, we similarly have individually-negotiated SIBs (for example, the Essex SIB that OPM has been evaluating). However, UK is unique because we have many SIBs developed through an Impact Bond Fund model (e.g. the Innovation Fund, Fair Chance Fund, Youth Engagement Fund). Under this model, government departments (as outcomes payers) spent a lot of time analysing data and came up with what is known as a ‘rate card’ that specifies the different outcomes that the government is interested in, how the outcomes should be measured, and the maximum price that the government will pay for each outcome.
There are now SIBs that are developed by service providers and sometimes intermediaries. In these cases, the service provider or intermediary led the development of the outcome metrics. For service providers, it is usually because they have a long history of delivering a specific intervention and have been measuring its effectiveness in a particular way.
Structuring outcome metrics and payment
Crudely speaking, this is done at the individual level or at the group level. At the individual level, outcomes are specified for the individual participant/beneficiary. It may be one outcome per participant, or could be a series of outcomes for that person. This is the approach used in the Impact Bond Fund SIBs. In the first round of the Innovation Fund, the rate card issued by the UK Department for Work and Pensions specified that one of the desired outcomes was ‘improved behaviour at school’. This outcome was to be measured by ‘letter from teacher’. The achievement of this outcome for the pupil triggers a payment of up to £800.
In comparison a group-level approach can be structured in two ways. First, you focus solely at the intervention group and define the number or the percentage within that group that needs to demonstrate the outcome. This is the approach used in Germany’s SIB which specified that at least 20 individuals out of the group of 100+ must experience the outcome for payment to be triggered.
Alternatively, you can compare the intervention group against a control group. In these scenarios, there are often thresholds set for outcome levels. For example, the Peterborough SIB structured its outcome metrics in a way that supported two different payment triggers. The first is when an intervention cohort demonstrates at least 10% reduction in reoffending compared with the control group. If this condition is not met, a second way for triggering payment is if all three intended cohorts have an average reduction in reoffending of at least 7.5%.
It strikes me that while the principles underpinning outcome metric selection are clear, the act of identifying and structuring them in support of SIBs is as much an art as it is a science. There is no single approach that works in all cases. I think it is important that we do not get lost in the technicalities and forget about what is really important. Instead, we must always keep a clear eye on outcomes and make sure that we identify and structure metrics in a way that supports meaningful achievement of those outcomes.