In an area as complex as healthcare, it may seem a daunting task for an individual practitioner to make a difference in terms of improving healthcare services. Nevertheless, some organisations make small grants available to enable practitioners to experiment with or pilot ideas for service innovations and improvements. The hope with these will be that, if successful, the innovation will be adopted by mainstream services, either locally or further afield. However, such a result is dependent on a whole range of factors, varying from the willingness of the individual's employing organisation to welcome innovation to whether a project leader can find influential champions to advocate for the project's adoption.
To maximise the chances of wider adoption, it is essential that projects can demonstrate successful outcomes, which requires projects to be evaluated. The Health Foundation (2015) suggests that evaluation is concerned with:
‘… practical assessment of the implementation and impact of an intervention. It is conducted in a spirit of discovery, rather than management or monitoring. It is concerned with developing understanding and supporting more strategic judgement and decision-making, such as whether and how an intervention should continue, and continue to be funded.’
Evaluations can focus on outcomes, assessing and preferably measuring what the project achieved, and/or on processes, examining and understanding the development and implementation of the project over time and in its local context. In this paper, we focus on evaluating the latter, drawing on an evaluation of 10 community nurse-led specialist innovation projects funded specifically to provide support to people experiencing homelessness (Bryar, 2020). These were funded in 2017 by the Queen's Nursing Institute (QNI) in partnership with the Oak Foundation (www.theoakfoundation.org.uk/) and implemented over the period of 1 year in 2018. The QNI possesses extensive experience in funding and supporting projects undertaken by nurses working in the community and occasionally advertises project funding opportunities on its website (www.qni.org.uk).
Poor health is both a contributing factor to and a result of people becoming homeless. The population that is homeless includes people of all ages, as well as families with children. Their health needs are the same as those of the general population, but are exacerbated by their precarious living situation (Local Government Association (LGA), 2017; Public Health England, 2019). Poor access to healthcare, combined with ill health, is responsible for the high mortality at an early age among people experiencing homelessness (Menezes et al, 2020).
The 10 projects discussed were expected to have the following outcomes (wording as in original):
- Improve the care provided to homeless people;
- Improve homeless people's access to care;
- Ensure that patients felt their concerns were listened to and that they had a leading role in their care;
- Improve patients' quality of life and wellbeing (QNI, 2015).
The funded projects focused on specific groups of people experiencing homelessness and other inclusion health groups: rough sleepers and homeless hostel residents; prisoners, asylum seekers and vulnerable migrants; and Gypsy, Roma and Traveller communities. They offered services that included health checks, drop-in clinics, health education, prevention of self-harm, tuberculosis (TB) screening, and provision of new clothing. Table 1 presents more detail about each project. Projects are referred to in the text by their corresponding number in the left-hand column of the table.
Table 1. Characteristics of the projects
| Number | Project | Location | Clients | Services provided |
|---|---|---|---|---|
| P1 | Drop-in and NHS health check | Birkenhead | Hostel residents and non-residents who are experiencing homelessness | Provision of NHS Health Checks to residents aged 40-75 years, and a drop-in service for minor injuries and ailments for residents and non-residents |
| P2 | Five Ways to Wellbeing | Bristol | Hostel residents and non-residents who are experiencing homelessness | Series of groups of five sessions on the Five Ways to Wellbeing for people experiencing homelessness (Connect, Be Active, Keep Learning, Take Notice and Give to others) |
| P3 | GRT health outreach project | East Surrey | Gypsy, Roma and Traveller communities in East Surrey | Community staff nurse outreach, screening for diabetes, stroke, heart disease and children's dental health, promoting uptake of immunisations and developing visual materials to support public health messages |
| P4 | Health champions for the homeless | Newham, London | People experiencing homelessness in the borough | Setting up a peer support group (health champions) for people experiencing homelessness and running a Big Health Day to identify undiagnosed conditions, with a focus on diabetes, mental health and respiratory conditions, and, where relevant, to start treatment plans |
| P5 | Health Inclusion Team Plus | Southwark, London | People sleeping rough | To increase uptake of pneumonia, influenza and Hepatitis A and B vaccinations; improve identification and treatment of clients with bloodborne viruses, human papillomavirus, diabetes and sexually transmitted diseases; connecting clients to primary health care, housing and homelessness support services |
| P6 | Latent tuberculosis screening and awareness | Birmingham, West Midlands | Male prisoners | Screening and treatment for latent tuberculosis infection in a nurseled clinic at the prison and education sessions for staff to increase their knowledge of tuberculosis |
| P7 | Leap Ahead | Darwen, Lancashire | Residents in a hostel | Improve access to general practice; increase uptake of flu, pneumococcal, MMR and meningitis immunisations; reduce smoking rates; reduce the risk of diabetes and heart disease through targeted education |
| P8 | New clothing for rough sleepers | Croydon, London | People sleeping rough, asylum seekers, hostel residents and non-residents who are experiencing homelessness | To respond to cases of scabies and other infestations by new clothing provision and dermatological treatment |
| P9 | Self-harm | Weston-super-Mare, North Somerset | Hostel residents and non-residents who are experiencing homelessness | Setting up a regular self-harm support group, providing one-to-one health education on self-harm reduction and self-harm emergency first aid kits |
| P10 | Touch Base | Brighton, East Sussex | Hostel residents and non-residents | Testing, diagnosis and treatment of hepatitis C. Training of non-health staff to undertake saliva swabs |
The nurses leading the projects were supported by the following QNI personnel, who were the staff of the QNI Homeless Health Programme (which is now called the Homeless and Inclusion Health Programme): the homeless health programme manager, the homeless health programme administrator, the director of programmes (to early 2018) and the director of nursing programmes (from April 2018). QNI also provided a programme of workshops on certain related topics, including planning, communicating, networking, creative thinking, economic evaluation and sustainability.
Evaluation methods
The evaluation was informed by a realist evaluation approach (Pawson and Tilley, 1997). This evaluates interventions by considering and distinguishing context, mechanisms and outcomes. It seeks to answer the question: ‘What works, for whom, in what circumstances, how and why?’ (Wong et al, 2017).
The outcomes of an innovation are dependent on the interaction between the elements of context and mechanisms, represented as: Context + Mechanism = Outcomes. Aspects of the context and the mechanisms can support or restrain implementation (Herens et al, 2016).
Realist evaluation is particularly useful in understanding innovations in complex social situations, such as those found in community and primary care settings. Contextual data also helps to identify specific local factors that make the innovation effective or ineffective, thus providing useful guidance to anyone wishing to replicate the innovation elsewhere.
To inform the evaluation, data were collected using different methods:
- Interviews with the former QNI director of programmes (by telephone), the former QNI homeless health project manager and one of the founders of the QNI Homeless Health Network (by Skype)
- Review of the project reports written by project leads
- Visits to the projects
- Interviews with project leads, project clients, and other staff
- Collection of any resources and evidence on the visits
- Internet searches for evidence of dissemination of the findings from the projects.
Interview schedules were developed, drawing on an initial meeting with the interim QNI homeless health programme manager and the QNI director of nursing programmes, and informed by a previous report on QNI-funded projects (Bryar, 2015). The visits to project leads were undertaken over a period of 4 weeks in the autumn of 2019.
Formal research ethics approval was not required for an evaluation; however, the work was undertaken throughout with reference to research ethics standards—for example, respect for confidentiality and informed consent by participants. The findings presented are drawn from the project evaluation report (Bryar, 2020).
Findings
Contextual factors
We first outline contextual factors that were reported by project leads as either enabling or disabling. As each enabling factor implies a corresponding disabling factor, Table 2 summarises both reported and implicit context factors. The enabling contextual factors can be categorised as follows:
- Relevant experience and skills
- Effective interprofessional collaboration
- Collaboration with service users
- Support for projects and project staff.
Table 2. Summary of contextual factors
| Enabling | Disabling |
|---|---|
| Relevant experience and skills | Lack of relevant experience and skills |
| Effective interprofessional collaboration | Lack of effective interprofessional collaboration |
| Effective collaboration with service users | Ineffective collaboration with service users |
| Support for projects and project staff | Lack of support for projects and project staff |
| Adequate nursing resources | Inadequate nursing resources |
| Reorganisations |
Relevant experience and skills
The majority of the project leads had extensive experience and expertise in working and building trusting relationships with people experiencing homelessness. Many also had well-established working relationships within the setting—for example, hostels—where the projects delivered services.
Effective interprofessional collaboration
The majority of the projects were embedded in wider networks of homeless health or specialist health and other services. This meant that project leaders were able to draw on a range of knowledge and skills. P5 drew on long-standing local networks between services for people sleeping rough, and P9 referred to a ‘strong network between the project leads, other services and volunteers in the area’. P2 drew on the expertise of services in the local mental health NHS trust in working with people experiencing homelessness. Some project evaluations mentioned the benefits of deliberately bringing together project staff with expertise from different areas—NHS experience and local authority knowledge of traveller communities (P3), for example.
Collaboration with service users
Some projects identified the role of service users in creating an enabling context. P4 was based on the willingness of some people experiencing homelessness to become health champions, and the project leader praised their commitment in the evaluation. P1 also noted that hostel residents had engaged in supporting the project. Other projects benefited from the information they had gleaned through consultation with service users; an approach summed up by P3:
‘Meeting the members of the community where they were, rather than following the health professionals' agenda.’
Support for projects and project staff
Respondents often mentioned the support they had received within their own or from other organisations for their projects—for example, from the board of trustees of a hostel, senior management, line managers, influential individuals, and the local clinical commissioning group. One example given included the staff of at a hostel for people experiencing homelessness, who encouraged residents to receive NHS health checks or to attend health promotion sessions (P1). Other assistance was provided by the TB nursing team and prison staff (P6), and by organisations that referred people experiencing homelessness to the project (P9). P3 was able to draw on the skills and support of a consultant when children's dental needs emerged as an unanticipated concern. P5 reported strong organisational support from middle and senior managers and from a GP practice, while P6 benefited from reduced charges for TB tests provided by the local NHS trust.
All the project leads valued the support they had received from the QNI, which included workshops, networking opportunities with the other project leads and learning from the other projects. They also appreciated the QNI's flexibility in relation to how project funding was used.
Disabling contextual factors can be categorised as follows:
- Practical problems
- Inadequate nursing resources
- Re-organisations
- Ineffective collaboration with service users.
Practical problems
Several leads reported that project progress was impeded by very practical problems that they themselves were not in a position to solve, such as locating a room to hold regular client sessions (P2) or finding storage space for materials between client sessions (P8). P1 was held back by a lack of access to NHS computer record systems, and P4 and P9 experienced significant delays in getting materials printed. P4 and P8 experienced delays in setting up financial systems within NHS trusts to ensure that project funds were released promptly. P8 pointed out that such projects do generate additional work for finance and other departments in NHS trusts, which needs to be recognised by project personnel.
Inadequate nursing resources
Some projects had difficulty recruiting nurses with the requisite skills (P2, P4, P8). P2 faced difficulties because nurses could not be recruited to cover usual clinics, as it was difficult to find skilled nurses who were prepared to work a 4-hour session, rather than a 12-hour hospital shift. In particular, once the projects came to an end, the majority of project leads did not have time to undertake further development of the project or dissemination of what had been learnt. This limited the degree to which such learning could be shared with the host, partner, and other organisations.
Reorganisations
Inevitably, some projects had to cope with local NHS and local council reorganisations, which affected their own staffing or how services could be delivered (P3, P5, P6, P10). For example, P6's relationship with the host prison was made problematic by a change of managing organisation, which meant that the prison was in a ‘state of crisis’ during the project year. Some sites (P3, P5) reported that staff numbers were reduced as part of such reorganisations.
Ineffective collaboration with service users
All the projects found that the engagement and support of clients in the development and ongoing running of the projects took a great deal of time. For example, P4 had not anticipated quite how much lifestyle issues would get in the way of the health champions fulfilling their role, and the nurse lead lacked the time to provide as much support to them as was needed. P3 found that some Gypsy, Roma and Traveller communities moved away from the area for periods of time. P6 had difficulties in following up with prisoners after discharge. In one case (P7), delays seemed to be due to a staff member's lack of skills in engaging hostel residents, and lack of appreciation of the complexity of their lives and health needs. One result of this was that clinics were initially held in the morning, which were not convenient for hostel residents and, were therefore, poorly attended. Similarly, in P5, the nurses changed the timing of client visits in the community from the morning to the afternoon, to enable better engagement.
Mechanisms
As discussed above, realist evaluation considers the interaction of the context and mechanisms on the outcomes or achievements of intervention projects. Mechanisms can be defined as: ‘… an element of reasoning and reactions of (an) individual or collective agent(s) in regard of the resources available in a given context to bring about changes through implementation of an intervention.’ (Lacouture et al, 2015).
In Table 1, the project interventions that were provided by each project are shown. The mechanism—that is, the reasoning and reactions of participants, both clients and staff of the projects, to the intervention and within the unique project contexts-resulted in the outcomes, which are shown in Table 3.
Table 3. Achievement of planned outcomes
| Number | Interventions | Outcomes |
|---|---|---|
| P1 | Provision of NHS Health Checks to residents aged 40−75 years, and a drop-in service for minor injuries and ailments for residents and non-residents | Some 39 drop-in clinics, 117 residents and 62 non-residents seen; NHS health checks for 60 residents, who had never previously had a check |
| P2 | Series of groups of five sessions on the Five Ways to Wellbeing for people experiencing homelessness (Connect, Be Active, Keep Learning, Take Notice and Give to others) | Five courses of five sessions attended by 22 clients whose anxiety was reduced; willingness to attend groups increased, overall mental and emotional wellbeing had improved; clients reported that they were more willing to prioritise their own healthcare |
| P3 | Community staff nurse outreach, screening for diabetes, stroke, heart disease and children's dental health, promoting uptake of immunisations and developing visual materials to support public health messages | Some 54 adults were offered screening and 22 accepted; discussion of health- and non-health-related issues lead to 16 referrals, 44 signpostings and 40 instances of direct advocacy. Identification of childhood dental issues and referral of 26 children via Vulnerable Children's Dental Pathway |
| P4 | Setting up a peer support group (health champions) for people experiencing homelessness and running a Big Health Day to identify undiagnosed conditions, with a focus on diabetes, mental health and respiratory conditions, and, where relevant, to start treatment plans | Seven health champions recruited and trained; champions worked with the nurse lead to host the Big Health Day and deliver health information to people experiencing homelessness |
| P5 | To increase uptake of pneumonia, influenza and hepatitis A and B vaccinations; improve identification and treatment of clients with bloodborne viruses, human papillomavirus, diabetes and sexually transmitted diseases; connecting clients to primary health care, housing and homelessness support services | Some 109 people seen on the streets; 77 same-day health checks and 69 health interventions carried out. Worked with the Street Population and Outreach Team to identify people living on the streets and the focus on health by the nurses enabled the outreach team to work with clients who had been difficult to engage in other ways |
| P6 | Screening and treatment for latent tuberculosis infection in a nurse-led clinic at the prison and education sessions for staff to increase their knowledge of tuberculosis | Some 100 men screened using an assessment tool and tested; 10 were identified with latent tuberculosis. Increased knowledge among staff of tuberculosis led to an increase in referrals to the project lead |
| P7 | Improve access to general practice; increase uptake of flu, pneumococcal, MMR and meningitis immunisations; reduce smoking rates; reduce the risk of diabetes and heart disease through targeted education | Some 11 NHS Health Checks, 31 influenza vaccinations and a number of other vaccinations carried out, along with prevention of a fatality through the identification of a resident in renal failure. Staff at the hostel now encourage residents to register with the local GP practice, and almost 50% of the hostel residents were registered in November 2019 |
| P8 | To respond to cases of scabies and other infestations by new clothing provision and dermatological treatment | Some 10 clients provided with new clothes. Reduction in numbers of clients with infestations reduced need during the project year. |
| P9 | Setting up a regular self-harm support group, providing one-to-one health education on self-harm reduction and self-harm emergency first aid kit | Some 45 self-harm discussions held and kits distributed over 9 months; being able to talk about self-harm valued by clients in one-to-one meetings with the nurse; reduction in the A&E visits by clients who had self-harmed or with infected wounds; awareness of service and packs increased referrals from other services |
| P10 | Testing, diagnosis and treatment of hepatitis C. Training of non-health staff to undertake saliva swabs | Team of non-health care staff in day centres and hostels trained to undertake initial screening. Some 76 swab tests were carried out during the project year; 28 people were referred to the project lead; 6 people with new cases of HCV were referred to specialist services, had further investigations and treatment. Micro-elimination of HCV in individual hostels tested |
The reactions of the clients to the projects varied from full to little engagement, influenced, as discussed above, by the contextual factors of their lives. Positive reactions occurred in many of the projects, including P2, where clients formed friendships within the groups, learnt from each other, recommended the group sessions to others and had a feeling of achievement when they gained their certificate of attendance. In P3, discussions between the community staff nurse and the Gypsy, Roma and Traveller communities about childhood immunisations were initiated and the project lead commented: ‘… the opportunity to challenge myths and beliefs about immunisations has a value and impact that is difficult to measure but valuable nonetheless.’ In P9, one man commented that ‘… it was good to have someone to talk to who listened and cared’, illustrating the importance of the reaction or response to an intervention.
Implementation of the projects was influenced by the reactions and reasoning of everyone involved. P10, for example, was supported by the enthusiasm of a hostel worker who had experienced hepatitis C some years prior and was a strong advocate for the screening programme due to this experience. Increasing knowledge helped to build the confidence of prison nursing staff in P6 to undertake assessments and refer people with potential TB to the project lead.
Outcomes
All the projects achieved the planned outcomes to a greater or lesser extent. In addition, some of the projects had unexpected outcomes for both the clients and services. An important outcome is whether they continued locally, and if findings were disseminated to other places.
Achievement of planned outcomes
Project leads identified specific project outcomes in their project plans, summarised in Table 3. Improved access to healthcare was achieved, for example, in P4 by the training of peer health champions and the provision of health checks on the Big Health Day, and in P5 through a nurse working with a street outreach team. Ensuring that people's concerns were listened to is evident in all the projects—for example, in P3 and P9. Improvements in quality of life and wellbeing is again evident for all projects, but is particularly illustrated in P8 and P2.
Unexpected outcomes
A number of projects had unexpected outcomes. In P8, hostel staff commented on the positive impact of the new clothes on the self-esteem and mental health of the clients. As a result of attending the groups in P2, clients formed friendships, learnt from each other and attained a feeling of achievement when they gained their certificate of attendance.
Of the 179 people who attended the drop-in clinics provided by P1, 40% said they would have gone to A&E and 30% would have called an ambulance if the drop-in clinics had not been available. Thus, the project reduced the demand on local services. For the primary care practice in which the project lead in P4 was based, an unexpected outcome of organising the training and Big Health Day event was an increase in local health networking. As a result of P6, the nurse lead became a keyholder, so she could respond to referrals from prison staff and carry out screening more quickly.
Longer term impact
At the end of the initial project year, 7 of the projects continued, as shown in Table 4; two of these, P3 and P10, were extended across their respective counties. Several projects, including P1, P3, P5, P8 and P10, had engaged with commissioners and governing bodies and gained ongoing support and funding to continue or extend their operations. A number of the projects are showcased on the QNI website, alongside resources to support nurses working with people experiencing homelessness and other inclusion health groups (www.qni.org.uk/resources/?aow=homeless-health).
Table 4. Impact of projects
| Number | Project | Location | Local? | Expanded? | Ongoing? |
|---|---|---|---|---|---|
| P1 | Drop-in and NHS health check | Birkenhead | Yes | No | Yes |
| P2 | Five Ways to Wellbeing | Bristol | Yes | No | No. Local interest in others running groups |
| P3 | GRT health outreach project | East Surrey | Yes | Yes | Yes. Extended to the whole of Surrey |
| P4 | Health champions for the homeless | Newham, London | Yes | Link on ELFT website to Queen's Nursing Institute film of project | No. Suspended due to lack of nursing staff |
| P5 | Health Inclusion Team Plus | Southwark, London | Yes | Yes | Yes. Extended to other London boroughs |
| P6 | Latent tuberculosis screening and awareness | Birmingham, West Midlands | Yes | Yes, through tuberculosis networks | Yes |
| P7 | Leap Ahead | Darwen, Lancashire | Yes | No | No. Annual flu clinic at the hostel |
| P8 | New clothing for rough sleepers | Croydon, London | Yes | No | Yes |
| P9 | Self-harm | Weston-super-Mare, North Somerset | Yes | No, but used in at least one other project | Yes; continued on one-to-one-basis |
| P10 | Touch Base | Brighton, East Sussex | Yes | Yes, through liver networks | Yes. Extended across the whole county |
Discussion
The enabling and disabling context factors noted above will be familiar to those experienced in running projects of many kinds, as most of these project-associated difficulties are not specific to this client group (Bryar and Griffiths, 2003). Nevertheless, it is interesting to compare these projects to a study of how four different local areas deliver effective health and care services to people sleeping rough, which noted five comparable shared ‘principles’ (Cream et al, 2020):
- Finding and engaging people sleeping rough
- Building and supporting workforces to go above and beyond existing service limitations
- Prioritising relationships
- Tailoring the local response to people sleeping rough
- Using the full power of commissioning.
Of these, only the last was not a prominent theme in the QNI projects, reflecting the fact that funding was for individuals, none of whom were commissioners.
The fact that the nature of the client group was sometimes experienced as a disabling contextual factor could perhaps have been better anticipated by those designing the projects, and by those choosing which to fund. The challenges of working with this client group are sufficiently well-known (LGA, 2017); indeed, the very existence of this QNI funding is an acknowledgement of the fact that routine services are not adequate to serve the health needs of people experiencing homelessness. Nevertheless, some project leaders underestimated, during the planning stage, the problems of working with people with relatively disorganised lifestyles and often complex care needs. A key piece of learning from the evaluations is that the challenge of reaching and engaging such populations must never be underestimated.
Another key piece of learning from the evaluations is that no individual nurse leader, however dynamic and skilled they may be, can progress such projects without the collaboration of a range of partners, encompassing individuals, departments or organisations. In planning a project, one cannot take for granted the capacity of such collaborators to adhere to the agreed scheduling and objectives. One might expect, for example, that printing could be performed promptly, but two projects found that this was not the case. QNI's flexibility in overseeing how the grants were spent meant that time lost for this reason was never too detrimental to a project; however, project planners need to devise timetables that allow for delays. The need to develop project management skills was identified by the project leads as part of the evaluation.
Conclusion
A requirement of the funding for the projects provided by the Oak Foundation was that an evaluation of the programme of projects should be undertaken. Use of a realist evaluation approach enables the identification of context factors and mechanisms that had an impact on the separate projects, but also allows for comparison across the projects and with other studies to identify common features that enable or disable innovation projects.
Key points
- Even small nursing projects that attempt to innovate should be evaluated, so that positive results can be disseminated to services and personnel planning similar initiatives
- Sharing locally and more widely also inspires other nurses to consider similar initiatives and illustrates how innovative and imaginative nurses can be when given the opportunity and support
- Local conditions may be crucial in the success of a project, so the evaluation needs to consider the impact of these
- ‘Realist evaluation’ is a framework that investigates how both the context and the mechanisms of the innovation contribute to outcomes. This paper looks at an evaluation of 10 small community nurse-led projects, to illustrate how this framework can be used.
CPD reflective questions
- Think of a development in your service, possibly one you were directly involved in. What outcomes did it achieve?
- What do you think the effects of local and organisational factors were on the outcomes of the development?
- How might the project have developed differently had local and organisational conditions been different?
- What advice would you give to someone wanting to replicate this development in a very different context (eg a much larger or smaller organisation than yours; a much more or less deprived population; etc)?