BEAT 2 Project: Final Evaluation Report


Download BEAT 2 Project: Final Evaluation Report


Preview text

BEAT 2 Project: Final Evaluation Report, NSC, 2018
MAS What Works Fund
BEAT 2 Project
Final Evaluation Report
On behalf of Changing Lives and Oasis Aquila Housing April 2018
1

BEAT 2 Project: Final Evaluation Report, NSC, 2018
Prepared for Changing Lives and Oasis Aquila Housing BEAT 2 Project: Final Evaluation Report FINAL April 2018
New Skills Consulting is a specialist regeneration and research consultancy. Combining expert knowledge with a practical hands-on approach, we help our clients develop and deliver successful projects. Peter Graham New Skills Consulting Spaceworks Benton Park Road Newcastle, NE7 7LX
[email protected] 0191 223 6720
2

BEAT 2 Project: Final Evaluation Report, NSC, 2018

Contents

1 Executive Summary

4

2 Overview of project

8

3 Overview of the evaluation approach

12

4 Key findings: Outcome/impact evaluation

16

5 Key findings: Process evaluation

32

6 Key findings: Economic evaluation

39

7 Limitations of the evaluation and future evaluation

43

8 Implications and recommendations for policy and practice

46

9 Sharing and learning activity

51

Annex 1 - Client Case Studies (attached)

3

BEAT 2 Project: Final Evaluation Report, NSC, 2018
1 Executive Summary
PROJECT SUMMARY
Focused on supporting vulnerable adults resident within both the urban and rural areas of Tyne & Wear and Northumberland, the aim of the BEAT 2 project was to break the cycle of financial instability and crisis, enabling financially excluded adults to gain financial independence and build resilience to future financial crisis. Delivery was undertaken by a small team of specialist advisors employed by Changing lives and Oasis Aquila Housing. They provided one-to-one financial advice/support on an outreach basis from a wide range of community settings e.g. recovery centres, food banks, or homelessness centres. Project delivery followed a two-stage process of short-term crisis support, followed by long-term development of financial skills.
By the end of March 2018, BEAT 2 had supported 496 adults age 18+ that were either homeless, or exhibited at least one of the characteristics placing them at risk of homelessness (e.g. 20% with rent arrears, 32% with a reduction in income, 56% with substance abuse issues, 66% mental ill health). All clients were within the ‘struggling’ population segment, as defined by MAS. The main support needs of clients were: help to apply for benefits; help to appeal a benefit decision or sanction; support to increase income; debt advice; and help to improve money management skills.
BEAT 2 also provided financial capability training to 134 frontline staff/volunteers working in other services supporting vulnerable adults (e.g. recovery centres, health organisations, employability services). The training equipped staff/volunteers with the skills and knowledge needed to provide basic financial capability support to their clients, and to refer clients on to BEAT Team Advisors for more specialist support.
EVALUATION APPROACH
The overall research question to be answered by the evaluation was: ‘To what extent does the BEAT 2 project help to improve the financial capability of clients most at risk of facing financial exclusion, within both urban and rural settings across Tyne & Wear and Northumberland?’ The evaluation focused primarily on gathering outcomes evidence from project clients, and from frontline staff/volunteers undertaking training, using pre and post-intervention surveys. The research took place between March 2017 and March 2018.
 Clients: 444 clients (90%) completed a baseline survey on first registering with the project, of which 329 received two or more interventions. 181 of these clients (55%) completed follow-up surveys between one and nine months after receiving support.
 Staff/volunteers: 134 staff/volunteers completed a baseline survey prior to undertaking training, of which 61 (46%) completed a follow-up survey between one and six months after completing training.
The surveys captured data to measure change against the following outcomes for clients (NB. these project level outcomes were mapped against four MAS outcomes: financial wellbeing; behaviours; ability; mindset).
 Outcome 1: Clients experience a reduction in benefit sanctions, rent arrears and debt.  Outcome 2: Clients have an improved understanding and awareness of their own financial situation, in
particular dealing positively with financial difficulties.  Outcome 3: Clients feel less anxious about their financial situation.  Outcome 4: Clients can keep track of income and spending, and create and stick to a viable budget.  Outcome 5: Clients have improved knowledge of financial services and products.
And for staff and volunteers:
 Outcome 6: Staff and volunteers have a better understanding of the importance of financial capability and its impact on clients.
 Outcome 7: Staff and volunteers have improved financial capability skills and knowledge.  Outcome 8: Staff and volunteers have improved ability to deliver financial capability support to clients.
4

BEAT 2 Project: Final Evaluation Report, NSC, 2018
SUMMARY OF KEY FINDINGS
Outcomes evaluation
 Financial wellbeing (Ability to live adequately within means) - The project supported clients to resolve benefit problems, restore or increase benefits income, and reduce rent arrears and debt. This improved the ability of clients to live adequately within their means e.g. 51% more clients were resolving benefit problems at the follow-up stage, compared to the baseline.
 Financial wellbeing (Finances and emotional/mental wellbeing) - The project enabled clients to improve their financial and emotional wellbeing, and reduce their anxiety levels. 37% fewer clients worry a great deal about their financial situation after support.
 FinCap Behaviours (Managing money well day-to-day) - The project supported clients to improve how they manage their money day-to-day, including taking action to reduce rent arrears and debt. As a result, clients feel they are managing better financially. 75% of clients at the baseline said they were finding it very difficult or quite difficult to manage financially, but by the follow-up stage this had fallen to 43%.
 FinCap Behaviours (Dealing with financial difficulties) - The project supported clients to take positive steps to deal with financial difficulties, by increasing their income. 28% of clients had applied for new/additional benefits during the past six months, but by the follow-up stage this had risen to 64%.
 Mindset (Financial attitudes to managing money) - The project improved the mindset of clients, improving their attitudes about managing money and helping them feel more in control financially. At the baseline stage, 43% of clients tended to agree, or agreed strongly, that nothing they do will make much difference to their financial situation, but by the follow-up stage this had fallen to 24% of clients.
 Mindset (Financial confidence) - The project helped clients become more confident in their ability to manage their finances. 31% of clients reported a significant improvement in their approach to keeping track of their finances (income and spending), between the baseline stage and the follow-up.
 Ability (Financial knowledge and understanding) - The project helped clients improve their financial knowledge, skills and understanding e.g. % of clients reporting they have a clear idea or some idea about how to create a household budget increased from 56% at the baseline, to 70% by the follow-up stage.
 Outcomes for staff/volunteers - The training had a positive impact on improving the financial capability, skills and knowledge of staff/volunteers, across a range of topics matched to the needs of clients including benefits, rent arrears, debt, and money management. The percentage of staff/volunteers reporting good or reasonable skills and knowledge in these areas increased by between 10% and 20% from the baseline to the follow-up.
 Urban and rural impacts - There was no evidence from the evaluation that the financial capability outcomes delivered by the project have differed between clients in urban areas and those in rural settings.
Process evaluation
What worked?
 Face-to-face advice provided by skilled and experienced advisors - By building trusting relationships over a period of several months, advisors were able to help clients address a financial crisis (e.g. benefit reduction, debt), then continue to work with them to build financial capability and skills).
 Working alongside specialists to achieve sustainable change - Financial capability support for this client group should be delivered in conjunction with support from other specialists, such as addiction recovery centres, or mental health counselling. This allows clients to improve their financial capability, at the same time as addressing the underlying causes of their financial situation.
 Importance of outreach delivery model - Providing financial advice to vulnerable adults where they are (e.g. recovery centres, supported accommodation, food banks) is the key to engaging and retaining vulnerable clients in financial capability support.
5

BEAT 2 Project: Final Evaluation Report, NSC, 2018
 Using benefits advice as a ‘hook’ to engage clients - Most clients were attracted to the project to get help to resolve a benefits problem. By offering support to resolve benefits problems (and related issues, such as debt and housing), clients can be engaged in a longer-term relationship with an advisor who can then help to develop their underlying financial capability and skills over time.
 Important role of frontline staff and volunteers - Working in partnership with frontline staff and volunteers in other projects and services (e.g. addiction recovery centres, mental health counselling) is an effective way to identify clients in need, encourage referrals, and allow frontline staff and volunteers to deal with more straightforward financial problems. Financial capability training helps give frontline staff/volunteers the skills and knowledge needed to provide basic support.
Other key findings  Complex needs of client group and significant time required to achieve real change - Adults who
are homeless, or at risk of homelessness, face a range of complex issues (e.g. physical and mental health problems, addictions and issues with personal safety), as well as financial exclusion. These underlying problems must be addressed, in parallel with building financial capability skills, if lasting change is to be achieved. This type of change takes longer than the 13 months available for this pilot project.
Economic evaluation  The economic evaluation has shown that the project delivered a positive return on investment, with
every £1 spent on the BEAT 2 project generating £3.63 in financial benefits.
METHODOLOGICAL LIMITATIONS  The mixed methods approach adopted, including outcome, process, and economic evaluation, is
considered to have been appropriate and effective. However, there were some limitations.  The 13 month period available for pilot delivery has been too short for some clients to achieve
meaningful change in financial capability, and for this to be measured by the evaluation.  The outcomes questionnaires, based on the MAS outcomes framework, did not always work well for this
group of vulnerable adults. Many clients found the questions confusing and the survey process too long, resulting in some clients becoming stressed and anxious.  While the evaluation has been able to demonstrate positive changes in the skills and understanding of staff/volunteers undertaking financial capability training, the sample size upon which the analysis is based (61 matching pre and post-intervention survey responses) was smaller than originally planned.  In line with the MAS outcomes framework, all of the survey data gathered to measure progress against project outcomes is based on self-reporting by clients at the pre and post-intervention stages. As is the case with all research based on self-reporting, there is a risk that some respondents may overstate the impacts of the support to show themselves in a good light, or they may understate the true impacts, for example where they lack self-confidence and tend to underestimate their achievements. This should be taken into account when considering the evidence of project outcomes.  It is important to highlight that the resources available for the evaluation research were insufficient to allow for significance testing to be undertaken. While the outcomes evaluation indicates that clients made positive progress against all of the outcomes defined in the Theory of Change, it is not possible to claim that the outcomes achieved are statistically significant because no significance testing has been undertaken. It is important to consider the evidence of project outcomes in this light.
6

BEAT 2 Project: Final Evaluation Report, NSC, 2018
IMPLICATIONS AND RECOMMENDATIONS  On the basis that the support is clearly needed, it makes a demonstrable difference to some of the most
vulnerable adults in society, and delivers a positive return on investment, there is a strong rationale for policy makers and funders to consider how this type of support could be continued in future.  The BEAT 2 delivery model has proven to be an effective solution to engaging the most financially excluded, vulnerable adults in financial capability support. This could be a valuable engagement tool for policymakers to consider, when seeking ways to boost the financial capability of the most excluded people in society.  MAS and the financial capability community are asked to consider the following recommendations.  Consider supporting a wider roll-out and scaling up of the BEAT 2 delivery model, with a specific
focus on meeting the financial capability needs of vulnerable, financially excluded adults, arising directly from current and ongoing welfare reforms.  Any roll out should take as its starting point the effective and successful delivery model which has been pilot tested through the BEAT 2 project. However, the delivery model should be improved to include the enhancements identified above. Any commissioning exercise should be co-designed in the first instance with the What Works partners focusing on this specific client group, who can then coordinate input from their wider network of support providers and intermediaries.  Depending on the outcomes of the co-deign exercise there may be options for either a national programme of financial capability commissioning for vulnerable, financially excluded adults; or local commissioning approaches, where services would be supported by local funders and commissioners, but within a supportive national policy framework that encourages and incentivises the local delivery of financial capability support for this vulnerable, financially excluded client group.
7

BEAT 2 Project: Final Evaluation Report, NSC, 2018
2 Overview of project
2.1 Aims and outcomes
New Skills Consulting1 was appointed by Changing Lives to undertake an independent evaluation of the ‘BEAT 2’ Project, which was funded through the Money Advice Service (MAS) What Works Fund (WWF). BEAT 2 was a 13 month pilot, commencing March 2017 and ending March 2018. It was delivered by Changing Lives and Oasis Aquila Housing. The main focus of the evaluation was to address the following key research question:
‘To what extent does the BEAT 2 project help to improve the financial capability of clients most at risk of facing financial exclusion within both urban and rural settings across Tyne and Wear and Northumberland?
2.2 Project activities
Theory of Change
The project aimed to break the cycle of financial instability and crisis experienced by financially excluded adults, delivering a two-stage process of short-term crisis support, underpinned by long-term development of financial skills.
Stage 1 Helping clients achieve stability in the short-term by dealing with immediate financial crisis.
Stage 2 Improving clients’ capability to manage and avoid future financial crisis, by developing their
skills, knowledge, motivations and attitudes.
In doing so, the project aimed to help clients to gain financial independence and build resilience to future financial crisis. The overarching goal was achieving sustainable financial stability, which enables people to better meet goals, reduce stress and improve quality of life. Improved financial capability will help clients to gain employment, tackle addiction, find suitable accommodation, leave exploitative relationships and remove themselves from harm. It was expected that the project would result in clients experiencing a range of positive outcomes including: reduced rent arrears; reduced benefit sanctions; reduced debt; reduced anxiety about their financial situation; improved understanding of their own financial situation; and improved skills to manage their own finances effectively (see Theory of Change diagram on page 11).
To achieve this the following activities were delivered:  A small team of BEAT Team Advisors provided specialist, one-to-one financial advice and support to
clients facing financial problems.  Other frontline staff and volunteers from Changing Lives, Oasis Aquila Housing, and other intermediary
organisations, received specialist financial training to equip them with the skills and knowledge to provide financial capability support to the clients they support in their day-to-day roles.  Provision of an e-financial telephone advice service, available to frontline staff and volunteers at Changing Lives and Oasis Aquila Housing, to provide specialist advice about more technical financial issues.
Clients were identified by BEAT 2 project staff undertaking outreach work at a range of community venues, such as recovery centres, food banks, Job Centres, and community centres. Clients experiencing financial crisis were identified at these locations and offered support. Potential clients were also referred to the project by partner agencies and by frontline staff and volunteers at Changing Lives and Oasis Aquila Housing.
1 Section 6 of the Report – Economic Evaluation – was undertaken by Nick O’Shea, an independent consultant. 8

BEAT 2 Project: Final Evaluation Report, NSC, 2018

2.3 Project context
The National Audit Office (NAO) has found that the North East has been hardest hit by the economic downturn and austerity. On average the reduction in welfare benefits between 2010 and 2015 is £551 for each working-age adult in the region (NAO 2016). The North East also has the highest unemployment rate in the country (Office of National Statistics, 2016) and the lowest average wage (NAO, 2016). For those experiencing homelessness/at risk of homelessness, this exacerbates the financial difficulties they face. Vulnerable groups experiencing addiction, domestic violence, long-term unemployment, or mental health issues are particularly badly affected. This client group is severely marginalised and unable to access the mainstream services designed to help people in financially difficulty. They suffer from extreme financial exclusion and lack the necessary skills and knowledge to improve their financial situation. Financial crisis is a common feature in their lives and is either a cause or result of the issues they face. Clients’ lack of financial capability can lead to rent arrears, benefit sanctions, and the need for supported accommodation. This creates cycles of instability and crisis, with financial insecurity at their heart.
2.4 Characteristics and needs of the target audiences
The primary target audience for the BEAT 2 project was vulnerable adults aged 18 plus that are most at risk of financial exclusion, in particular, those who are homeless or ‘at risk’ of homelessness. Homeless includes those rough sleeping, living in supported accommodation, or ‘sofa surfing’. The project defined individuals as being ‘at risk’ of homelessness where they face one or more of the following: rent arrears; debt; benefit sanctions; or reduction in income (e.g. due to benefit cuts, or loss of job). Other important factors contributing to an individual being placed at greater risk of homelessness include substance abuse, mental ill health, and offending behaviour. It was expected that the majority of clients would be single people, but the project would also support families and couples with the characteristics described above.

By the end of March 2018 BEAT 2 had supported 496 clients. The characteristics and support needs of clients have been analysed using responses to the baseline evaluation survey completed on initial engagement by 444 clients (90%), and the project database (Inform/CPM) which has records for 438 clients matched to the baseline survey respondents. This is the evaluation dataset. A summary of the characteristics and needs of the clients supported, matched to the project’s key targeting criteria, is provided in Table 1. It shows that the project has been targeted effectively on the intended groups. All clients supported were either homeless, or exhibited at least one of the characteristics placing them at risk of homelessness (e.g. 20% with rent arrears; 32% reduction in income; 56% substance abuse). However, most clients exhibited multiple risk factors. 97% of clients were aged 18 plus, and all were within the ‘struggling’ population segment, as defined by MAS.

Table 1: Characteristics and needs of BEAT 2 clients compared to project targeting criteria

Age

18-24 (9%); 25-40 (37%); 41-60 (44%); 60+ (7%); Under 18 (3%)

Homeless (or threatened with

Homeless (23%); threatened with eviction (5%)

eviction)

Other factors placing people at risk of homelessness

Rent arrears

20% Substance abuse

56%

Debt

23% Mental ill health

66%

Benefits sanctioned

3% Offending behaviour

14%

Reduction in income

32%

Personal / family situation Geographic area (urban / rural) 2

Single (78%); family (15%): couple (7%) Urban (78%); rural (15%); unknown (7%)

Gender

Male (58%); female (42%)

2 It is estimated that approximately 18% of the North East’s working age population live in rural areas (North East Farming and Rural Advisory Network (NEFRAN), Rural Growth Prospectus, Sept 2013). At least 15% of BEAT 2 project clients live in rural areas, which broadly reflects the North East average.
9

BEAT 2 Project: Final Evaluation Report, NSC, 2018

Employment status

Permanently sick/disabled (36%); temp sick (18%); unemployed (34%); employed (6%); other (6%)

Support needs / what did clients want support with on first engagement with project?

Apply for benefits

34% Increase income

13%

Appeal a benefit

29% Housing or tenancy issues

13%

decision/sanction

Debt advice3

24% Understand financial products

2%

Money management

17% Other

8%

Source: Inform/CPM data (438 clients), Baseline evaluation survey (444 clients)

The BEAT 2 project had a secondary audience, which was frontline staff and volunteers working for Changing Lives, Oasis Aquila Housing, and other support organisations. Staff and volunteers were targeted for training based on two criteria. Firstly, their day-to-day role involved them supporting vulnerable adults within the BEAT 2 target audience; and secondly, that they wanted to improve their understanding of financial capability, and their knowledge/skills to deliver financial capability support to clients.

By the end of March 2018, the project had provided training to 134 staff / volunteers. Their characteristics and support needs have been analysed using responses to the baseline evaluation survey completed by 134 staff/volunteers prior to undertaking the training. This shows the project has been targeted effectively on the intended groups. 100% of those trained work in frontline roles with vulnerable adults. 77% work for Changing Lives or Oasis Aquila, while 23% work for other partner organisations providing support to vulnerable adults (e.g. recovery services, employability providers, health organisations). A greater proportion of trainees were paid staff, and fewer were volunteers, compared to expectations at the start of the project.

Table 2: Characteristics of staff and volunteers compared to project targeting criteria

Status Organisation

123 paid staff (92%); 11 volunteers (8%) Changing Lives and Oasis Aquila (77%); other support organisations (23%)

Job role (highest frequency) Training support needs

100% working in frontline roles - Employment; Asset Coach; Housing Support; Recovery Service; Mental Health; Independent Living; Peer Mentor; Women’s Service Worker. Benefits: Helping clients apply for benefits (79%); how to appeal a benefit decision (78%) Debt: How to help clients manage debt/negotiate with creditors (64%) Other: Better-off in work calculations (57%); basic financial concepts (51%)

Motivations

Provide better service to clients (76%); update existing financial knowledge to help clients (48%)

Source: Baseline evaluation survey (134 responses)

2.5 Project changes
Overall, the project was delivered as intended, and there were no significant changes during delivery. The main delivery issue encountered was that the number of staff and volunteers taking up the opportunity to participate in the specialist financial capability training was lower than originally anticipated. There are a number of reasons that may have contributed, including: internal structural changes in the two partner organisations during the project delivery period; lack of time for staff and volunteers to attend the training; and for some individuals financial capability is not high on the agenda in the context of their work so the training was not a priority. Nevertheless, feedback was gathered from those that had participated in the training, and all of those who provided feedback reported that their financial capability skills and knowledge had improved as a result. In addition, three quarters had already been able to use these skills in successfully supporting clients with financial problems.

3 In the context of this project ‘debt advice’ means signposting to specialists and not providing FSA regulated debt advice. 10

Preparing to load PDF file. please wait...

0 of 0
100%
BEAT 2 Project: Final Evaluation Report