Skip to main content

Integrating patient and public involvement into co-design of healthcare improvement: a case study in maternity care

Abstract

Background

Despite recognition of the importance of patient and public involvement (PPI) in healthcare improvement, compelling examples of “what good looks like” for PPI in co-design of improvement efforts, how it might be done, and formalisation of methods and reporting are lacking. In this article, we sought to address these gaps through a case study to illustrate a principled approach to integrating PPI into the co-design of healthcare improvement.

Methods

The case study aimed to involve maternity service users in the co-design of clinical resources for a maternity improvement programme, using a four-stage approach: 1) establishing guiding principles for PPI in the programme, 2) structuring PPI for the programme, 3) co-designing improvements with PPI, and 4) seeking feedback on PPI in the co-design process.

Results

Partnership-focused frameworks and other literature on PPI and co-design informed the guiding principles. The structure included a five-member PPI group who provided continuous input, and an additional 15-member PPI group who met twice to discuss experiences of obstetric emergency. PPI in the co-design processes shaped the development of the resources in multiple ways, such as strengthening the prominence given to listening to those in labour and their birth partners, ensuring inclusivity of visuals and language, and developing communication principles informing all resources. Feedback suggested that PPI members felt valued, listened to, and supported to provide unanticipated contributions.

Conclusions

The case study demonstrated how a principled approach to PPI enabled service users to play a key role in co-design of clinical resources aimed at improving the quality and safety of maternity care in the UK. Further case studies, across different clinical areas and with varying levels of resources, are needed to validate this approach.

Peer Review reports

Background

Building on the growth of patient and public involvement (PPI) in healthcare research and service delivery [1,2,3,4,5,6,7], an emerging movement seeks to integrate PPI in the improvement of quality and safety of healthcare (i.e. healthcare improvement) [8,9,10,11]. Healthcare improvement efforts can range from large-scale programmes focused on organisational change through to small-scale projects to improve the ways local clinical teams care for patients [12]. However, the question of how improvement activities can authentically and inclusively involve service users throughout has remained under-explored [10, 13, 14]. In part, this vacuum arises because of an absence of compelling examples of “what good looks like” for integrating PPI in the co-design of improvement, how it might be done, and formalisation of methods and reporting [4, 11, 15].

Co-design, as a broad approach, seeks to identify and understand problems from multiple perspectives (e.g. patients, healthcare providers, designers), and generate and evaluate solutions through collective creativity, working in partnership, and mobilising experiential knowledge [13, 16,17,18,19]. Development of approaches to integrate PPI in co-design of improvement can usefully draw on key learning from practices and experiences of PPI in health research, where it is increasingly institutionalised in policy, funders’ requirements, ethical review, and researcher training [1, 20,21,22], and has evolved into a distinct set of practices [1, 20,21,22,23,24,25]. However, PPI in research has also encountered a range of challenges, including those relating power imbalances, limited engagement of minority ethnic groups and those at risk of social and economic disadvantage, inadequate emotional, financial and practical support to PPI contributors, and a tendency towards managerialist or consumerist models of involvement [26,27,28,29,30].

These issues must also be addressed in healthcare improvement, where the evidence base for effective activities has remained relatively under-developed [8, 14, 31]. An additional challenge for improvement efforts is that they may be dominated by simplistic “product-dominant” logic adopted from industrial settings [7, 31] that risks neglecting the relational nature of healthcare and fails to recognise that those who use healthcare services are not simply consumers, but active contributors to their own health [8, 9, 32, 33] PPI may help address these challenges [9, 34], but progress on integrating it in healthcare improvement is patchy and slow [14].

A review of published frameworks for supporting and evaluating PPI in research indicates that a principled, activity-bespoke approach to PPI – rather than a “one-size-fits-all” approach – is the most effective way to integrate PPI in a programme of work [1]. The range and nature of challenges similarly suggest that no single universal methodological framework will be suitable for PPI in healthcare improvement [35]. Instead, a set of guiding principles might offer the required flexibility and ability to customise PPI to the particular health conditions, settings, and contexts of each improvement activity or effort [11, 19, 36]. This kind of approach can also help to make concrete the values underpinning the activity [36], address the need to respect and value inclusion and diversity [11] and serve as a practical and actionable tool for setting expectations and supporting conflict resolution [36]. Despite its potential advantages and calls for pilots and case studies [1], detailed practical illustrations of a principled approach to PPI remain scarce [19].

In this article, we aim to illustrate the application of a principled approach to PPI in co-design of improvement, by reporting a case study that involved service users in co-design of clinical resources for a maternity improvement programme.

Methods

We employed a pragmatic case study approach, with a view to answer questions of how or why things work in real life contexts [37]. Our case study was pragmatic in that it built on an existing programme of work – the Avoiding Brain in Childbirth (ABC) programme [38,39,40]. After presenting the case study, we outline the four stages of work we used to apply a principled approach to integrating PPI in the case study (Table 1).

Table 1 A principled approach to integrating PPI in co-design of a healthcare improvement programme, as applied to a case study

Case study: Avoiding Brain Injury in Childbirth (ABC) programme

The UK Department of Health and Social Care funded in 2021 the Avoiding Brain Injury in Childbirth (ABC) programme, a collaboration between the Royal College of Midwives (RCM), Royal College of Obstetricians and Gynaecologists (RCOG), and The Healthcare Improvement Studies Institute (THIS Institute) at the University of Cambridge [38,39,40]. The ABC programme aimed to co-design, with healthcare professionals and service users, standardised approaches to two key contributors to avoidable harm: (1) detecting and responding to possible fetal deterioration during labour [38, 41], and (2) management of the obstetric emergency of impacted fetal head at caesarean birth [42,43,44,45]. The ABC programme sought to involve those who use maternity services, through PPI, in the co-design of the following prototype resources, ready for deployment in a future national programme of testing, implementation and evaluation:

  1. (1)

    a clinical practice tool that combines assessment of fetal heart rate features with other intrapartum risk factors [38],

  2. (2)

    a multi-professional training package to support detecting and responding to possible fetal deterioration during labour [41],

  3. (3)

    a clinical management algorithm to support training and guide practice for managing impacted fetal head at caesarean birth [44], and

  4. (4)

    a multi-professional training package to support management of impacted fetal head at caesarean birth [42, 44].

The ABC programme was a particularly appropriate case study because of its focus on co-design involving service users in maternity care improvement – an area where the lack of explicit approaches for co-design was most concerning. This gap meant that those who stand to benefit the most from improved maternity care may be at risk of exclusion from the process [26, 46,47,48,49,50,51] It required inclusive PPI in co-design of improvement, given evidence of persistent problems in safety of maternity care remains a persistent challenge [52,53,54,55], and an established link between poor outcomes to social disadvantage and minority ethnicity [49, 56]. Several reports have identified enduring suboptimal engagement with women and their partners before, during and after birth [48, 50, 51]. Though these reports highlight the importance of listening to the voices of those who use maternity services [48, 50, 51], how to achieve this has remained unclear.

Another reason the ABC programme was a compelling case study is that improvement resources like clinical tools and training packages frequently suffer from problems of poor implementation and variations in use [57,58,59]. These problems are often related to sub-optimal design and development processes [60, 61], including inadequate involvement of healthcare professionals and service users prior to implementation [59, 62,63,64,65]. Involving maternity service users in co-design of clinical resources for a maternity improvement programme was therefore an especially apt setting to explore the application of a principled approach.

We gathered insights from the case study using data collected in the programme, including workshops and surveys with various PPI groups and clinicians involved in the ABC programme as well as anecdotal feedback from PPI members, and the authors’ lived experiences of contributing to the programme as facilitators or participants. Richness of the case study’s data was enhanced by the use of user-centred design methods and co-design principles [38, 42], as well as survey questions developed using GRIPP2 reporting guidelines [66, 67].

Tables 2, 3 and 4 and the Results section provide further detail on data collection and analysis relating to PPI as used in the case study. Data collection and analysis relating to co-design of the ABC resources are detailed elsewhere [38, 41].

Table 2 Structure of the two PPI groups involved in the ABC programme
Table 3 Processes and outcomes of co-design of clinical resources for detecting and responding to possible fetal deterioration during labour
Table 4 Processes and outcomes of co-design of clinical resources for managing impacted fetal head at caesarean birth

Stages of work used to apply a principled approach to the case study

We employed four stages of work to apply a principled approach to integration of PPI into co-design of the clinical resources for the case study (Tables 1):

  1. (1)

    establishing guiding principles for PPI in the programme,

  2. (2)

    structuring PPI for the programme,

  3. (3)

    co-designing improvements with PPI, and

  4. (4)

    seeking feedback on PPI in the co-design process.

The first stage comprised the drafting of a set of guiding principles to PPI customised to the needs and context of the ABC programme. In the second stage, the guiding principles were used to inform setting up flexible structures for PPI in the ABC programme, such as what types of PPI groups to establish, how to ensure inclusivity and diversity in and across the groups, clarifying the roles of the groups, and establishing how to compensate PPI members for their contributions [72]. The third stage comprised the co-design of the clinical resources with PPI, which included building of relationships among those involved in the programme while remaining cognisant of potential power dynamics [5, 24, 30, 73,74,75] The fourth stage focused on seeking feedback from those involved in the programme, with a view to learn what did and did not work in the co-design processes, demonstrate transparency, and help create a learning culture [66, 67].

Results

Below, we illustrate how each of the four stages of the applied principled approach guided integration of PPI in the co-design of prototype resources for the ABC programme.

1. Establishing PPI principles for the ABC programme

In developing guiding principles for involving maternity service users in the ABC programme, we started by reviewing relevant literatures, including those relating engaging with women and birth partners on experiences of maternity care and safety in the UK [76, 77], engagement and involvement of Black and Minority Ethnic groups in healthcare research [78, 79], and co-design principles [13, 16,17,18,19, 80]. The review demonstrated the relevance and potential of collaborative partnerships between researchers and lay people, especially those that emphasise transparency and accountability [6] we therefore also reviewed literature on partnership-focused frameworks [5, 81], in particular those developed for the English NHS’s INVOLVE [82] and the Canadian Institutes of Health Research [83].

The literature reviews clarified that no pre-existing framework for PPI was exactly suited to the needs of the ABC programme. This informed a collective decision – discussed together with the ABC PPI group (see Stage 2 below) – that a set of customised guiding principles would be most appropriate. Based on the review of the literature, the research team drafted a set of principles as a first step, and then refined these in discussion with the ABC PPI group members (see Stage 2 below):

  1. (1)

    be transparent and open,

  2. (2)

    be equitable by providing access and support,

  3. (3)

    work towards diversity,

  4. (4)

    listen and respect,

  5. (5)

    be open to constructive challenge, and

  6. (6)

    provide feedback and recognition.

The principles aimed to optimise the collaborative partnership between the team developing the improvement resources for the ABC programme and maternity service users, with a particular focus on: (i) enabling equity and inclusion, (ii) facilitating co-design including healthcare professionals and those with lived experiences, and (iii) ensuring that maternity service user experience and perspective would be reflected in the resources co-designed in the programme. These principles shaped the subsequent steps of the project.

2. Structuring PPI for co-design in the ABC programme

The guiding principles for PPI in the programme and available resources for the programme were important in informing the structures for PPI. In recognition that different phases of the programme would have different needs and opportunities for PPI input, the PPI facilitation leads (LH and JL) ensured the establishment of two PPI groups: one that included five members who supported the whole programme (see Table 2 and Supplement 1), and an additional group (n = 15) who met twice to share and discuss relevant lived experiences of obstetric emergency (Table 2). We considered a relatively small “core” group to be the most productive mechanism for contributing to the programme. They attended regular meetings that took place throughout the lifecycle of the programme (July 2021 to April 2022). The members of the core ABC PPI group were selected based on their lived experience and the expertise they could bring to the case study. They included representation of women from Black and Asian minority backgrounds as well as expertise in advocating of maternity safety. The additional PPI group was invited via several local Maternity Voices Partnerships—aiming to maximise diversity and representation of under-represented voices – with a wide take-up for the two planned meetings. For the tasks planned for these meetings, a larger group of about 15 participants was considered manageable.

The design of the groups was different to reflect their different roles in the overall programme of work. The core ABC PPI group met regularly with a rolling agenda and very open/equitable structure. The number of meetings was not pre-defined at the start of the project. In contrast, the larger group met twice and had a very focussed agenda – because of the larger numbers the range of topics covered was narrower to give all women space to share their views.

Members of both groups were paid in accordance with NIHR guidance on involvement [72]. Members of the ABC PPI group co-authored outputs (this paper) or were acknowledged as part of authorship/contributor groups [38, 42].

3. Co-designing clinical resources for the ABC programme with PPI

Development processes for the clinical resources for the ABC programme – a clinical tool, an algorithm and two training packages – drew on user-centred design methods and co-design principles [38, 42]. All meetings (Table 2) and co-design activities (Tables 3 and 4) with the PPI groups were online and organised in an open, informal, flexible, and democratic way to support authentic contribution [75].

A pre-circulated agenda framed the meetings (Table 2), with any required readings shared in advance, but retained enough flexibility for new agenda items to emerge. Expected meeting outcomes or areas of input were not set at the start but determined dialogically through open conversations. PPI facilitators chaired or supported meetings in ways that enabled the service user voice to be heard, while providing clarity about what was in scope for PPI in co-designing the resources. The meetings were tailored to the flexible, iterative approach required for co-design of clinical practice resources [13, 17, 18, 80], while fostering a sense of group identity, trust, and mutual respect. Learning and views from the PPI groups were reported regularly to programme management meetings to ensure maternity service user voices were a constant presence.

Most meetings were held with the five PPI group members and one or more PPI facilitators (BA, JL and/or LH). At certain points, key members of the clinical teams (two obstetricians and two midwives – see Acknowledgements) were invited, as needed, to present materials or ask the PPI groups for advice and input (Tables 3 and 4). The size and experience of the PPI groups (Table 2), and attention to listening ensured by the PPI facilitators, addressed potential power imbalances during the meetings, such as those between healthcare professionals and service users [5, 30, 73,74,75].

Integrating PPI in the co-design processes in this way was highly impactful in the design of the improvement resources (Tables 3 and 4, Supplementary Material 2). For example, the PPI groups strengthened the prominence given to listening to those in labour and their birth partners, ensured inclusivity of visuals and language used in the resources, and supported the development of the strategies for communicating with women and birth partners.(Supplementary Material 2) [38, 42].

4. Seeking feedback on the co-design process in the ABC programme

A discussion between the PPI facilitators and the ABC PPI group – guided by a semi-structured interview guide (Supplemental material 3) – took place during a meeting dedicated to reflecting on the group’s experiences of co-design in the ABC programme. The PPI facilitators collected additional feedback from the ABC PPI group as well as the ABC clinicians who participated in the PPI meetings (two obstetricians and two midwives), using an email survey with questions developed using GRIPP2 reporting guidelines (Supplemental material 3) [66, 67]. Further anecdotal feedback was derived from PPI members and Maternity Voice Representatives taking part in the training evaluations (see Table 4) [42].

The ABC PPI group reported that effective input on co-design of the resources was helped by the flexible process and egalitarian meetings where ideas, different perspectives and challenge were welcomed and encouraged.

‘PPI is built in ABC. It is a culture. PPI was not forced into the programme.’ (ABC PPI member)

The group reflected on the approach to PPI in the ABC programme, indicating they felt listened to and valued, with facilitation of unanticipated contributions to the co-design of resources.

‘Our ideas were included as the programme was being developed.’ (ABC PPI member)

The group reported that frequent meetings, and the inclusion of multiple team members, supported relationship-building, gave time to reflect, served building trust, and helped to challenge power dynamics. The online meetings were seen as enabling involvement from across the country.

‘Have enjoyed working with this great team. Patient advocates often find themselves the sole lay representative in many situations—it’s been a privilege to work with others […]’ (ABC PPI member)

The importance of PPI in the ABC programme was also highlighted at other times, for example during focus groups following pilot-testing of one of the training packages [42].

‘This is one birth and that person probably isn’t ever going to experience it again. So it’s so important in training to keep practicing how to work alongside and with that service user, because that’s their one or maybe second chance of birth. They will never be here again and they will remember all of you for the rest of your lives.’ (Maternity Voice Partnership Representative).

The PPI group also valued the opportunity to co-design with clinicians, including how ABC clinicians joined the PPI meetings when invited to discuss technical or clinical details. The ABC clinicians involved in the PPI meetings reported that involvement of maternity service users had a positive impact on the ABC programme. They also reported developing a deeper appreciation of the knowledge and experience that maternity service users hold.

‘Allow time to really listen. Sometimes we can feel defensive of our profession and maternity services, but before we can begin to make positive changes we need to see the reality of the situation from the perspective of the people who use them.’ (ABC clinician).

The ABC clinicians for example noted that it made them think about their own clinical and research practice, and that they would make changes to ensure that they worked with PPI groups in all future improvement activities.

‘To be able to say we really are listening is invaluable, especially given the findings of recent national reports.’ (ABC clinician)

The PPI group also reported aspects of co-design that were less good, including the very high pace of the programme. Suggestions for improvement included recruiting birth partners, a more transparent process for recruiting PPI members, and further increasing involvement of clinicians during the meetings.

Discussion

Compelling examples of “what good looks like” for PPI in co-design of healthcare improvement, how it might be done, and formalisation of methods and reporting are all needed to support high-quality and impactful healthcare improvement activities [4, 11, 35]. This paper presents and illustrates an approach based around the development and deployment of guiding principles that are bespoke to the context and requirements of specific programmes [1, 11, 36] The case study demonstrates how use of this principled approach enabled service users to play a key role in co-designing clinical resources in a programme addressing an area of pressing need – improving quality and safety of childbirth in maternity care in the UK – by sharing their lived experience and extending the diversity of those involved in design [8, 11, 14]. It shows that a principled approach can effectively be used to co-design healthcare improvement, where integration of PPI has been patchy and slow [14]. The case study illustrating how to apply a principled approach also answers the call for more empirical examples of how to move beyond a “one-size-fits-all” approach to PPI [1], and addresses the limited attention to PPI in reporting of improvement activities [19, 35, 84].

Our four-stage approach (Table 1), with guiding principles bespoke to the programme developed at the outset, helped ensure that service users’ voices and lived experiences could influence all aspects of the programme. Ensuring diversity of voices and perspectives was a core principle, given the evidence linking poor maternal and perinatal outcomes to ethnic and social disadvantage [49, 56]. This was supported by inviting PPI participants from ethnically diverse backgrounds, and having PPI structures that allowed their voices to be heard during PPI meetings. This, in turn, supported a focus on building rapport and trust with those in labour and their partners during co-design of the clinical resources, in recognition this is especially important for women from ethnic minority and socially disadvantaged groups [26, 46].

The principled approach was successfully employed for a high-paced, complex and relatively well-resourced healthcare improvement programme. The success of the case study demonstrates the value of committing time and resource to PPI, developing a coherent team of PPI members, facilitators and other stakeholders, and encouraging mutual trust and sustained involvement throughout the programme. For example, a range of resources enabled investing time in regular meetings over several months, with continued access to the skills needed to support and follow the guiding principles around PPI established in the first stage. Recognising the value of time (including opportunity costs), resources, trust and relationship-building for PPI may help address the need for attention to the relational nature of healthcare, including allowing those who use healthcare services to be active contributors to their own health rather than being “consumers” [8, 9, 85]. Committing to partnering with service users on improvement journeys can help overcome some of the socio-cultural challenges of engaging with clinicians’ social worlds [10], including helping address the risks of a simplistic product-dominant logic that overlook the needs and preferences of service users when undertaking healthcare improvement [8, 9, 86].

Given its flexible nature and its potential for improving the co-design of clinical resources, a principled approach could be of use to a range of improvement activities, including ones with less resources than available in the case study, and in areas other than maternity care. To test this, future work is needed to establish what levels of resources are needed to employ the proposed approach in other case studies, including establishing the trade-offs required to enable positive impacts of PPI in lower-resourced improvement programmes. Further case studies employing the approach could also help to refine it, and could be captured in a repository used to provide further illustrative examples and enable evaluation the approach. This work could help strengthen the methodological infrastructure for PPI in co-design of healthcare improvement [84].

Limitations

The pandemic conditions and relatively short timescales (10 months) in which the case study had to be conducted imposed some limitations. Although the PPI groups included diverse and experienced women, the need for rapid recruitment of the groups challenged optimal transparency of recruitment. It did not allow for purposive selection of representative birth partners. It also precluded seeking feedback from the obstetric emergency PPI group. We were also not able to assess to what extent the absence of in-person meetings and the researchers’ roles in collecting and analysing the data influenced the outcomes. Although the proposed approach appears to have potential to support integration of PPI in co-design of healthcare improvement that can be standardised, replicated, and potentially scaled when needed, it will require further evaluation. Future work should help identify the kinds of applications the approach works best for and where its limits lie, including establishing the resourcing needed for minimal and optimal execution of each stage.

Calls for stronger reporting of impact of PPI on health services are not new and reporting guidelines for PPI (GRIPP) [66] and quality improvement (SQUIRE) [84] are well established. Although these reporting guidelines exist the field would be strengthened if they were drawn on more routinely. We have aimed to build on these guidelines, though acknowledge that further improvement is possible – the latter may be supported by further integration of GRIPP and SQUIRE guidelines.

Conclusions

This paper, with a case study in maternity care, shows how healthcare improvement teams can use a principled approach to systematically integrating PPI into co-design of clinical resources. It demonstrates how use of a principled approach to PPI can enable service users to effectively influence co-design of clinical resources – by acting as intermediaries who share their lived experience and diversify views in a co-design process – while feeling listened to and valued. The case study indicates that a principled approach must be supported by appropriate skills, attention to diversity, and resources. Further evaluation is needed to establish the extent to which the approach can benefit integrating PPI in a wider range of improvement activities.

Data availability

Queries about the dataset should be directed to the corresponding author.

Abbreviations

ABC:

Avoiding Brain Injury in Childbirth

PPI:

Patient and public involvement

References

  1. Greenhalgh T, Hinton L, Finlay T. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019;22(4):785–801. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/hex.12888.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Ostrom E, Parks RB, Whitaker GP. The public service production process: a framework for analyzing police services. Policy Stud J. 1978;7(s1):381–481. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/j.1541-0072.1978.tb01782.x.

    Article  Google Scholar 

  3. Wicks P, Richards T, Denegri S, et al. Patients’ roles and rights in research. BMJ. 2018;362:k3193. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmj.k3193.

    Article  PubMed  Google Scholar 

  4. Mockford C, Staniszewska S, Griffiths F, et al. The impact of patient and public involvement on UK NHS health care: a systematic review. Int J Qual Health Care. 2011;24(1):28–38. https://doiorg.publicaciones.saludcastillayleon.es/10.1093/intqhc/mzr066.

    Article  PubMed  Google Scholar 

  5. Baines RL, Regan De Bere S. Optimizing patient and public involvement (PPI): Identifying its “essential” and “desirable” principles using a systematic review and modified Delphi methodology. Health Expect. 2018;21(1):327–35. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/hex.12618.

    Article  PubMed  Google Scholar 

  6. Greenhalgh T, Jackson C, Shaw S. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94(2):392–429. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/1468-0009.12197.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Batalden M, Batalden P, Margolis P, et al. Coproduction of healthcare service. BMJ Qual Saf. 2016;25(7):509–17. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2015-004315.

    Article  PubMed  Google Scholar 

  8. Armstrong N, Herbert G, Aveling EL, et al. Optimizing patient involvement in quality improvement. Health Expect. 2013;16(3):e36–47.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Batalden P. Getting more health from healthcare: quality improvement must acknowledge patient coproduction—an essay by Paul Batalden. BMJ/. 2018:k3617. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmj.k3617

  10. Alidina S, Martelli PF, Singer SJ, et al. Optimizing patient partnership in primary care improvement: a qualitative study. Health Care Manage Rev. 2021;46(2):123.

  11. Bombard Y, Baker GR, Orlando E, et al. Engaging patients to improve quality of care: a systematic review. Impl Sci. 2018;13(1). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s13012-018-0784-z.

  12. The Health Foundation. Quality improvement made simple: What everyone should know about health care quality improvement. 2013. Available at: https://www.health.org.uk/publications/quality-improvement-made-simple.

  13. Robert G, Locock L, Williams O, et al. Co-producing and co-designing. Cambridge: CambridgeUniversity Press; 2022.

    Book  Google Scholar 

  14. Cluley V, Ziemann A, Feeley C, et al. Mapping the role of patient and public involvement during thedifferent stages of healthcare innovation: a scoping review. Health Expect. 2022;25(3):840–55. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/hex.13437.

    Article  PubMed  PubMed Central  Google Scholar 

  15. McGowan J, Attal B, Kuhn I, et al. Quality and reporting of large-scale improvement programmes: a review of maternity initiatives in the English NHS, 2010-2023. BMJ Qual Saf. 2023:bmjqs-2023–016606. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2023-016606.

  16. Grindell C, Coates E, Croot L, et al. The use of co-production, co-design and co-creation to mobilise knowledge in the management of health conditions: a systematic review. BMC Health Serv Res.2022;22(1). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12913-022-08079-y.

  17. Sanders EBN. Chapter 4: Staging co-design within healthcare: lessons from practice. In: Clausen C, Vinck D, Pedersen S, et al., eds. Staging Collaborative Design and Innovation: An Action-Oriented Participatory Approach. Cheltenham, UK: Edward Elgar Publishing Limited 2020.

  18. Langley J, Wolstenholme D, Cooke J. ‘Collective making’ as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Serv Res. 2018;18(1). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12913-018-3397-y

  19. Masterson D, AreskougJosefsson K, Robert G, et al. Mapping definitions of co-production and co-design in health and social care: a systematic scoping review providing lessons for the future. Health Expect. 2022;25(3):902–13. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/hex.13470.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Boaz A, Biri D, McKevitt C. Rethinking the relationship between science and society: has therebeen a shift in attitudes to patient and public involvement and publicengagement in science in the United Kingdom? Health Expect. 2016;19(3):592–601. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/hex.12295. Published Online First: 20141031.

    Article  PubMed  Google Scholar 

  21. Russell J, Greenhalgh T, Taylor M. Patient and public involvement in NIHR research 2006–2019: policy intentions, progress and themes. National Institute for Health Research: Oxford, UK 2019.

  22. Zibrowski E, Carr T, McDonald S, et al. A rapid realist review of patient engagement in patient-oriented research and health care system impacts: part one. Res Involv Engagem. 2021;7(1):72. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s40900-021-00299-6. Published Online First: 20211010.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Department of Health. Equity and excellence: liberating the NHS (Vol. 7881). The Stationery Office. 2010.

  24. Lauzon-Schnittka J, Audette-Chapdelaine S, Boutin D, et al. The experience of patient partners in research: a qualitative systematic review and thematic synthesis. Res Involv Engagem. 2022;8(1):55. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s40900-022-00388-0. PublishedOnlineFirst:20221003.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Beal AC. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA 2012;307(15):1583. https://doiorg.publicaciones.saludcastillayleon.es/10.1001/jama.2012.500.

  26. Maguire K, Britten N. “How can anybody be representative for those kind of people?” forms of patientrepresentation in health research, and why it is always contestable. Soc Sci Med. 2017;183:62–9.

    Article  PubMed  Google Scholar 

  27. Madden M, Speed E. Beware Zombies and Unicorns: Toward Critical Patient and Public Involvement in Health Research in a Neoliberal Context. Front Sociol. 2017. https://doiorg.publicaciones.saludcastillayleon.es/10.3389/fsoc.2017.00007.

  28. Ocloo J, Matthews R. From tokenism to empowerment: progressing patient and public involvement inhealthcare improvement. BMJ Qual Safety. 2016;25(8):626–32. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2015-004839.

    Article  Google Scholar 

  29. Knowles SE, Allen D, Donnelly A, et al. More than a method: trusting relationships, productivetensions, and two-way learning as mechanisms of authentic co-production. Res Involv Engagem. 2021;7(1):1–14.

    Article  Google Scholar 

  30. Ocloo J, Garfield S, Franklin BD, et al. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Res Policy Syst. 2021;19(1). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12961-020-00644-3.

  31. Dixon-Woods M. Harveian Oration 2018: improving quality and safety in healthcare. Clin Med. 2019;19(1):47–56. https://doiorg.publicaciones.saludcastillayleon.es/10.7861/clinmedicine.19-1-47.

    Article  Google Scholar 

  32. Hunter B, Berg M, Lundgren I, et al. Relationships: the hidden threads in the tapestry ofmaternity care. Midwifery. 2008;24(2):132–7. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.midw.2008.02.003. Published Online First:20080402.

    Article  PubMed  Google Scholar 

  33. Entwistle VA, Carter SM, Cribb A, et al. Supporting patient autonomy: the importance of clinician-patient relationships. J Gen Intern Med. 2010;25(7):741–5. https://doiorg.publicaciones.saludcastillayleon.es/10.1007/s11606-010-1292-2. PublishedOnline First: 20100306.

    Article  PubMed  PubMed Central  Google Scholar 

  34. HQIP. A guide to patient and public involvement in quality improvement. Available at: https://www.hqip.org.uk/resource/a-guide-to-patient-and-public-involvement-in-quality-improvement/. 2019

  35. Nordin A, Kjellstrom S, Robert G, et al. Measurement and outcomes of co-production in health and social care: a systematic review of empirical studies. BMJ Open. 2023;13(9):e073808. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjopen-2023-073808. Published Online First: 20230922.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Wolfe A. Institute of Medicine report: crossing the quality chasm: a new health care system for the 21st century. Policy Polit Nurs Pract. 2001;2(3):233–5.

  37. Baker GR. The contribution of case study research to knowledge of how to improve quality of care. BMJ Qual Saf. 2011;20 Suppl 1(Suppl_1):i30–5. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs.2010.046490.

    Article  PubMed  Google Scholar 

  38. Woodward M, Dixon-Woods M, Randall W, et al. How to co-design a prototype of a clinicalpractice tool: a framework with practical guidance and a case study. BMJ Qual Saf. 2024;33(4):258–70. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2023-016196. Published Online First: 2023/12/21.

    Article  PubMed  Google Scholar 

  39. Royal College of Obstetricians and Gynaecologists (RCOG). Avoiding Brain Injury in Childbirth (ABC). Available at: https://www.rcog.org.uk/about-us/quality-improvement-clinical-audit-and-research-projects/avoiding-brain-injury-in-childbirth-abc/. 2024.

  40. The Healthcare Improvement Studies (THIS) Institute. Next phase of ABC programme to reduce brain injuries at birth commences. Available at: https://www.thisinstitute.cam.ac.uk/blog/next-phase-of-abc-programme-to-reduce-brain-injuries-at-birth-commences/. 2024.

  41. O’Heney J, McAllister S, Maresh M, et al. Fetal monitoring in labour: summary and update of NICE guidance. BMJ. 2022;379:o2854. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmj.o2854. Published Online First: 2022/12/17.

    Article  PubMed  Google Scholar 

  42. van der Scheer JW, Cornthwaite K, Hewitt P, et al. Training for managing impacted fetal head at caesarean birth: multi-method evaluation of a pilot. BMJ Open Qual. 2023;12:e002340. Available at: https://bmjopenquality.bmj.com/content/12/3/e40.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Cornthwaite K, Hewitt P, van der Scheer JW, et al. Definition, management, and training in impacted fetal head at caesarean birth: a national survey of maternity professionals. Acta Obstet Gynecol Scand. 2023. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/aogs.14600.

  44. Cornthwaite KR. Impacted fetal head at caesarean section (doctoral thesis). Bristol: University of Bristol. 2022. Available at: https://hdl.handle.net/1983/2d996ec1-e3bc-4424-b03a-fc7c9c04b80c.

  45. Cornthwaite K, van der Scheer JW, Kelly S, et al. Management of impacted fetal head at cesarean birth: a systematic review and meta-analysis. Acta Obstet Gynecol Scand. n/a(n/a) https://doiorg.publicaciones.saludcastillayleon.es/10.1111/aogs.14873

  46. Boozary AS, Shojania KG. Pathology of poverty: the need for quality improvement efforts to addresssocial determinants of health. BMJ Qual Saf. 2018;27(6):421–4. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2017-007552. Published OnlineFirst: 20180306.

    Article  PubMed  Google Scholar 

  47. Health and Social Care Committee. The safety of maternity services in England. Fourth Report of Session 2021-22: House of Commons, 2022.

  48. Kirkup B. Reading the signals: maternity and neonatal services in East Kent – the Report of the Independent Investigation. Available at: https://www.gov.uk/government/publications/maternity-and-neonatal-services-in-east-kent-reading-the-signals-report. 2022.

  49. Knight M, Bunch K, Tuffnell D, Shakespeare J, Kotnis R, Kenyon S, Kurinczuk JJ (Eds.) on behalf of MBRRACE-UK. Saving Lives, Improving Mothers’ Care - Lessons learned to inform maternity care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2016-18. Oxford: National Perinatal Epidemiology Unit, University of Oxford; 2020.

  50. Knight M, Stanford S. Ockenden: another shocking review of maternity services. BMJ. Publishing Group, 2022.

  51. Ockenden D. Findings, conculsions and essential actions from the independent review of maternity services at the Shrewsbury and Telford Hospital NHS Trust. Ockenden Report, 2022.

  52. NHS Resolution. Five years of Cerebral Palsy claims. A thematic review of NHS Resolution data. London: NHS Resolution. 2017:1–37. https://www.resolution.nhs.uk.

  53. NHS Resolution. Ten years of maternity claims: an analysis of nHS litigation authority data. Available at: https://resolution.nhs.uk/resources/ten-years-of-maternity-claims-an-analysis-of-nhs-litigation-authority-data/. 2018.

  54. Royal College of Obstetricians and Gynaecologists. Each baby counts: 2015 full report. London: RCOG; 2017.

    Google Scholar 

  55. Royal College of Obstetricians and Gynaecologists. Each baby counts: 2020 final progress report. London: RCOG; 2021.

    Google Scholar 

  56. Jardine J, Walker K, Gurol-Urganci I, et al. Adverse pregnancy outcomes attributable tosocioeconomic and ethnic inequalities in England: a national cohort study. Lancet. 2021;398(10314):1905–12.

    Article  PubMed  Google Scholar 

  57. Van Galen LS, Struik PW, Driesen BEJM, et al. Delayed recognition of deterioration of patients in general wards is mostly caused by human related monitoring failures: a root cause analysis of unplanned ICU admissions. PLoSOne. 2016;11(8):e0161393. https://doiorg.publicaciones.saludcastillayleon.es/10.1371/journal.pone.0161393.

    Article  CAS  Google Scholar 

  58. Elliott D, Allen E, McKinley S, et al. User acceptance of observation and response charts with atrack and trigger system: a multisite staff survey. J Clin Nurs. 2016;25(15–16):2211–22. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/jocn.13303.

    Article  PubMed  Google Scholar 

  59. Mackintosh N, Watson K, Rance S, et al. Value of a modified early obstetric warning system (MEOWS)in managing maternal complications in the peripartum period: an ethnographicstudy. BMJ Qual Safety. 2014;23(1):26–34. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2012-001781.

    Article  Google Scholar 

  60. Preece MH, Hill A, Horswill MS, et al. Applying heuristic evaluation to observation chart designto improve the detection of patient deterioration. Appl Ergon. 2013;44(4):544–56. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.apergo.2012.11.003. Published Online First: 20121208.

    Article  PubMed  Google Scholar 

  61. Isaacs R, Smith G, Gale-Andrews L, et al. Design errors in vital sign charts used in consultant-led maternity units in the United Kingdom. Int J Obstet Anesth. 2019;39(60):67. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/j.ijoa.2019.01.001.

    Article  Google Scholar 

  62. Connolly F, Byrne D, Lydon S, et al. Barriers and facilitators related to the implementation of a physiological track and trigger system: a systematic review of the qualitative evidence. Int J Qual Health Care. 2017;29(973):980. https://doiorg.publicaciones.saludcastillayleon.es/10.1093/intqhc/mzx148.

    Article  Google Scholar 

  63. Elliott D, Allen E, Perry L, et al. Clinical user experiences of observation and response charts:focus group findings of using a new format chart incorporating a track andtrigger system. BMJ Qual Safety. 2015;24(1):65–75. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2013-002777.

    Article  Google Scholar 

  64. Ray EM, Smith R, Massie S, et al. Family alert: implementing direct family activation of apediatric rapid response team. Jt Comm J Qual Patient Saf. 2009;35(11):575–80.

    PubMed  Google Scholar 

  65. Rodriguez NM, Burleson G, Linnes JC, et al. Thinking beyond the device: an overview of human-and equity-centered approaches for health technology design. Annu Rev Biomed Eng. 2023;25:257–80. https://doiorg.publicaciones.saludcastillayleon.es/10.1146/annurev-bioeng-081922-024834. Published Online First: 2023/04/18.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, Altman DG, Moher D, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358.

  67. Matthews R, Kaur M, French C, et al. How helpful are patient and public involvement strategicdocuments - results of a framework analysis using 4Pi national involvement standards. Res Involv Engagem. 2019;5:31. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s40900-019-0164-0. Published Online First: 20191104.

    Article  PubMed  PubMed Central  Google Scholar 

  68. National Maternity Voices. What is a Maternity Voices Partnership (MVP)? Available at: https://nationalmaternityvoices.org.uk/mvp-toolkit/1-introduction/test-new-toolkit-content-page/.

  69. NHS Race & Health Observatory. Ethnic Inequalities in Healthcare: A Rapid Evidence Review. Available at: https://www.nhsrho.org/wp-content/uploads/2023/05/RHO-Rapid-Review-Final-Report_.pdf. 2022

  70. Cornthwaite K, Draycott T, Winter C, et al. Validation of a novel birth simulator for impactedfetal head at cesarean section: an observational simulation study. Acta Obstet Gynecol Scand. 2023;102(1):43–50. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/aogs.14432.

    Article  PubMed  Google Scholar 

  71. Landesman R, Graber EA. Abdominovaginal delivery: modification of the cesarean section operation to facilitate delivery of the impacted head. Am J Obstet Gynecol 1984;148(6):707-10. https://doiorg.publicaciones.saludcastillayleon.es/10.1016/0002-9378(84)90551-9.

  72. National Institute for Health and Care Research (NIHR). Payment guidance for researchers and professionals 2022 [updated July 2022. Available from: https://www.nihr.ac.uk/documents/payment-guidance-for-researchers-and-professionals/27392 . Accessed 18 Oct 2022.

  73. Morris L, Dumville J, Treweek S, et al. Evaluating a tool to improve engagement and recruitment of under-served groups in trials. Trials. 2022;23(1). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s13063-022-06747-2

  74. Glover W, Vogus T. Beyond the equity project: grounding equity in all quality improvement efforts BMJ Qual Saf. 2023;32(3):129. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2022-015858.

    Article  PubMed  Google Scholar 

  75. Martin GP, Finn R. Patients as team members: opportunities, challenges and paradoxes of includingpatients in multi-professional healthcare teams. Sociol Health Illn. 2011;33(7):1050–65. https://doiorg.publicaciones.saludcastillayleon.es/10.1111/j.1467-9566.2011.01356.x.

    Article  PubMed  Google Scholar 

  76. Dixon S, Agha K, Ali F F, et al. Female genital mutilation in the UK-where are we, where do we gonext? Involving communities in setting the research agenda. Res Involv Engagem. 2018;4(1):1–8.

    Article  Google Scholar 

  77. Liberati EG, Tarrant C, Willars J, et al. Seven features of safety in maternity units: a frameworkbased on multisite ethnography and stakeholder consultation. BMJ Qual Saf. 2021;30(6):444–56.

    Article  PubMed  Google Scholar 

  78. Darko N. Engaging Black and Minority Ethnic Groups in Health Research:‘Hard to Reach’? Demystifying the Misconceptions: Policy Press. Bristol: Bristol University Press; 2021.

  79. Farooqi A, Jutlla K, Raghavan R, et al. Developing a toolkit for increasing the participation ofblack, Asian and minority ethnic communities in health and social careresearch. BMC Med Res Methodol. 2022;22(1):1–16.

    Article  Google Scholar 

  80. Sanders EBN, Stappers PJ. Co-creation and the new landscapes of design. CoDesign. 2008;4(1):5–18. https://doiorg.publicaciones.saludcastillayleon.es/10.1080/15710880701875068.

    Article  Google Scholar 

  81. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving fromtheory to practice. J Comp Eff Res. 2015;4(2):133–45.

    Article  PubMed  Google Scholar 

  82. Involve N. Public involvement in research: Values and principles framework. Eastleigh, UK: Nihr Involve; 2015.

    Google Scholar 

  83. Pauly B, Urbanoski K, Hartney E, Shahram S, Marcellus L, Wallace B, Macdonald M, Hancock T. What is missing from “Patient-Oriented Research?” A view from public health systems and services. Healthc Policy. 2019;15(2):10.

  84. Ogrinc G, Davies L, Goodman D, et al. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986–92. https://doiorg.publicaciones.saludcastillayleon.es/10.1136/bmjqs-2015-004411.

    Article  PubMed  Google Scholar 

  85. McGowan D, Morley C, Hansen E, et al. Experiences of participants in the co-design of a community-based health service for people with high healthcare service use. BMC Health Serv Res. 2024;24(1):339. https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12913-024-10788-5.

    Article  PubMed  PubMed Central  Google Scholar 

  86. Williams O, Robert G, Martin GP, Hanna E, O’Hara J. Is co-production just really good PPI? Making sense of patient and public involvement and co-production networks. Decentring health and care networks: reshaping the organization and delivery of healthcare. 2020:213–37.

Download references

Acknowledgements

We are most grateful for the contributions of all service users involved in the ABC programme. We thank Rachna Bahl, Katie Cornthwaite, Pauline Hewitt and Wendy Randall for their contributions to the PPI meetings. We thank Philippa Storer and Laura Cowell for their project management contributions. For recruitment and communications support, we thank the Avoiding Brain Injury in Childbirth (ABC) communications team including members from THIS Institute, RCOG and RCM. We are grateful for the many and varied contributions from colleagues across the ABC programme team and external to the team.

Funding

The Avoiding Brain Injury in Childbirth programme (Department of Health and Social Care, UK) supported this work. THIS Institute is funded by the Health Foundation, Grant/Award Number: RHZF/001 - RG88620. The Health Foundation is an independent charity committed to bringing about better health and health care for people in the UK. Mary Dixon-Woods was an NIHR Senior Investigator (NF-SI-0617-10026) during conduct of the study. The funders had no role in study design, data collection and analysis, interpretation of the data, and preparation of the manuscript.

Author information

Authors and Affiliations

Authors

Contributions

Lisa Hinton is the guarantor for this article. All authors read and approved the final manuscript. Their specific contributions, following CRediT (Contributor Roles Taxonomy), are as follows: •Bothaina Attal: formal analysis; investigation; writing – review and editing. Joann Leeding: conceptualisation; formal analysis; investigation; methodology; project administration; writing – review and editing. Jan van der Scheer: project administration; writing – review and editing. Zenab Barry: investigation; writing – review and editing. Emma Crookes: investigation; writing – review and editing. Sandra Igwe: investigation; writing – review and editing. Nicky Lyons: investigation; writing – review and editing. Susanna Stanford: investigation; writing – review and editing. Mary Dixon-Woods: methodology; supervision; writing – original draft preparation; writing – review and editing. Lisa Hinton: conceptualisation; formal analysis; investigation; methodology; project administration; supervision; writing – original draft preparation; writing – review and editing.

Corresponding author

Correspondence to Lisa Hinton.

Ethics declarations

Ethics approval and consent to participate

The consultations with the obstetric emergency PPI group (see Table 2) received ethics approval from The University of Cambridge Psychology Research Ethics Committee (PRE.2022.021), with all participants providing written informed consent. The UK’s Health Research Authority decision tool (http://www.hra-decisiontools.org.uk/research/) showed that ethics approval was not required for the involvement of the ABC PPI group members. Their involvement was conducted in line with best PPI practice, with all members invited to join as authors of this paper.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Attal, B., Leeding, J., van der Scheer, J.W. et al. Integrating patient and public involvement into co-design of healthcare improvement: a case study in maternity care. BMC Health Serv Res 25, 352 (2025). https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12913-025-12423-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doiorg.publicaciones.saludcastillayleon.es/10.1186/s12913-025-12423-3

Keywords