Evaluation of a pediatric navigation program within primary care: a quantitative analysis guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework

Abstract

Background

Pediatric Support Services (PSS) is a Patient Navigation Program designed to address barriers from referral-to-service connection from primary care to health system and community-based services and resources. This study aimed to evaluate PSS’ implementation for mental health services along the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework and identify factors throughout implementation to inform sustainability and delivery.

Methods

This study included descriptive analysis of all patients referred to PSS to assess reach, with a primary cohort analyses of a subset of patients referred specifically to mental health services. Data collection included triangulation of information extracted from electronic health records, direct contact with patients’ caregivers, and follow-up surveys completed by patients’ caregivers. Analyses were designed within each construct of the RE-AIM framework, and assessed for their tiered impact on the patient, provider, and system levels.

Results

From October 2019 to June 2023, 13,109 total referrals for 11,214 unique patients were triaged by PSS. The patient population overrepresented younger, Hispanic, female patients compared to the clinical population included in this health system’s service area. Of these patients, 3,929 were followed-up by trained navigators at two-weeks for mental health service connection, with 50.6% reported being connected to referred services and an additional 27.1% with pending appointments. There was a significant increase in referral connection rate as age increased and for Black patients, compared to other children. For patients considered connected to or pending services, a satisfaction survey found high satisfaction with PSS and the amount of navigator-patient contact (81.5 and 79.6%, respectively).

Conclusion

These findings highlight potential program modifications to optimize quality of care and health for children and families, while enhancing capacity among providers, navigators, and clinics. Further adaptations, including electronic health record integration, patient/family feedback, and automated navigation processes, are suggested next steps for comprehensive navigation.

Trial registration

This study was approved by the Institutional Review Board for Prisma Health, trial number 1,852,794, with the most recent approval for expanded evaluation received on June 15, 2022 (original application approved in 2016).

Background

Prior to the COVID-19 pandemic, an estimated 1 in 5 children (aged 0 to 18) in the United States had a diagnosed mental health condition; however, only 20% received necessary care from mental health professionals [1, 2]. Early evidence estimates a persistent increase in mental health concerns and diagnoses among pediatric populations in recent years, and an exacerbated impact on previously strained pediatric mental health care services with the onset of the COVID-19 pandemic [3,4,5,6,7]. While the prevalence of mental health conditions, including loneliness, isolation, anxiety, and depression, climbed throughout the pandemic, outpatient mental health services utilization and mental health assessments significantly declined [8, 9]. Further, mental health-related emergent visits and inpatient hospital admissions also rose [10]. In conjunction with patients’ and families’ high level of perceived difficulty navigating health care systems, current research and trends highlight a growing need for patients’ connection to mental health services, both timely and tailored appropriately to patients’ needs, and for a system that supports an overburdened provider network [11,12,13,14]. Without adequate support, a growing proportion of children and youth may experience ongoing, untreated mental health conditions across the lifespan, which are associated with a myriad of negative effects on long-term health outcomes [4, 15].

Current research emphasizes the importance of addressing and reversing the trajectory of mental health conditions among children and youth nationwide, particularly through implementation science, early diagnosis and treatment, and community outreach initiatives [16, 17]. Patient Navigation Programs (PNPs) are one strategy to support active referral connection, with demonstrated effectiveness in increasing service access and transition through the health care system [18]. PNPs have demonstrated evidence toward increased referral connection rates, addressed healthcare access inequities, decreased mental health symptoms, increased quality of life, and increased treatment adherence [18,19,20]. However, a lack in standardized, comprehensive, and integrated evaluation and practices limits the generalizability and longevity for PNP findings [21, 22].

The purpose of this study is to evaluate the implementation of an active PNP, Pediatric Support Services (PSS), within a large health system and to establish sustainable measures of implementation outcomes to inform continued assessment of the PNP delivery, adoption, and success. PSS operates through a health system to triage pediatric mental and behavioral referrals from primary care clinics to relevant services based within the health system and community. To organize and plan measures, and to assess limitations in measurable outcomes, this study was guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework [23, 24]. Therefore, the aims of this study include:

• 1. Developing a comprehensive and structured evaluation of PSS; and,

• 2. Testing PSS as an intervention and identifying modifiable program components within the real-world context of a large health system.

Methods

Setting and participants

PSS is an evidence-based, systems navigation model supporting service and resource connection from primary care clinics in one of the largest children’s hospitals in a southeast state in the United States [18,19,20]. By the end of the study, 15 pediatric primary care clinics within the health system network referred patients to PSS. PSS services children and families with behavioral health concerns, at high risk for developmental and behavioral health problems, and social drivers that influence health. Model integration began in two primary care clinics in early 2017. Similarly to other models, PSS employs trained navigators fluent in community outreach, existing community programs, with social work experience, and who are bilingual and available to provide services in both English and Spanish [25,26,27]. These navigators triage and support referrals from primary care clinics through initial patient connections to services based on need, demographics, and availability of services in the local community. In addition to initial patient contact, navigators’ follow-up with patients’ caregivers through referral process to increase connection rates and to decrease barriers to health service access. In this model, navigators have access to patients’ medical charts and an internal resource and service database in which they can link appropriate services based on the patients’ needs. Initial referral to PSS occurs internally through a patient’s electronic health record by primary care physicians which is then opened and assessed by a navigation coordinator. Inclusion criteria for referrals to PSS include: (1) ages 0 to 18, (2) a behavioral health diagnosis, a risk factor or a behavioral or developmental health concern (e.g. experienced stress or trauma, abnormal mental or behavioral health screener, risky health behaviors, familial concern), or an identified social driver of health from their primary care provider.

Study design

The study design and analysis are conducted at three touch points throughout the navigation process (Fig. 1). This study includes data collected from October 2019 through June 2023 as part of an expanded evaluation and comprehensive data collection. Descriptive statistics of retrospective chart review and initial patient caregiver contact are assessed for all patients referred to PSS, regardless of services needed or outcomes (Total Referrals). A subset of all referrals that 1.) received mental health service connections, and 2.) agree to a two-week follow-up are included in a cohort study of service connection outcomes. Just under half of all referrals to PSS are specifically related to patients mental or behavioral health. All other referrals to PSS include various indicators of social drivers for health, which may include food insecurity, housing concerns, legal needs, or medical complexities. Mental and behavioral health referrals were the focus of this study to prioritize an overburdened system, assess an active implementation process (other navigation services had previously been implemented), and to study a relatively homogenous patient population in regard to diagnosis and need. The second phase focuses on patients with mental health service access outcomes only, due to capacity of navigators to follow-up with a subset of predetermined services. An additional subset of the previously described population is included in a satisfaction follow-up based on 1.) service outcome categorized as connected or pending, and 2.) consent to receiving a satisfaction survey for PSS.

Fig. 1

Study design and analysis flow diagram

Data collection

A comprehensive evaluation of PSS was implemented to explore program impact and integration through an implementation science lens using the RE-AIM framework (Table 1). To achieve this goal, a Research Electronic Data Capture (REDCap) database was created to store all data collected as part of both the delivery and evaluation of PSS. Data collection for this study occurs in three ways: 1) medical chart review, 2) manual data entry upon patient outreach via caregivers, and 3) navigator satisfaction surveys sent directly to patients’ caregivers. REDCap is an electronic database designed to collect and securely store patient information to support clinical and translational research [28]. To streamline data collection and patient connection, automated electronic health records reports (pulled from this health system’s Epic System) containing demographic data for all patients referred to PSS are run on a weekly basis and uploaded automatically into the REDCap database. Patient demographics are limited to patient age, gender, race, and insurance status due to completeness of demographic information by providers in medical charts. Upon entry into REDCap, navigators connect with patients’ caregivers to collect data on visit characteristics, reasons for referral and reasons for referral noncompletion, specific service needs and preferences, then connecting and recording referred services and resources. Approximately two weeks after initial patients’ caregiver contact and referral connection, navigators follow up with patients’ caregivers identified as needing services specific to mental health services.

Table 1 Study measures along RE-AIM framework

At two-week follow-up for patients who have connected or are pending (including scheduled appointment) connection to services, are automatically sent a satisfaction survey via e-mail or text message from the REDCap database. The navigation satisfaction survey is adapted from the Navigation Satisfaction (NAVSAT) survey [29], which is validated in use with families receiving system navigation services. The NAVSAT was designed to identify navigator and service provider factors that are associated with patient satisfaction, including navigators’ ability to connect patients and families with appropriate services, the ability of services to improve child and family well-being, navigators’ ability to listen and understand family concerns, and the caregivers’ satisfaction with the navigators’ intake procedures and appropriate frequency of contact [29, 30]. NAVSAT distribution began among eligible participants beginning in June 2022 and is sent via text message or email based on participant’s preference. Data is continuously collected via participant self-report, and participants receive automated follow-up messages from REDCap after two weeks if nonrespondent.

Data analysis

All data was analyzed quantitatively across the RE-AIM constructs. Reach was assessed through frequencies and group comparisons across demographics; chi-square for categorical variables, and unpaired t-tests for continuous. Service connection was calculated from patients referred to mental health or early childhood development navigators approximately two weeks from initial contact with navigators. Connection rate is then calculated from patients connected to services by follow-up over patients recorded as not connected. To estimate differences in patients lost to follow up, follow-up was dichotomized as response to follow-up (regardless of recorded connection to service) and lost to follow-up (i.e. navigator was unable to reach the patient).

Effectiveness was estimated through differences in both service connection and patient response to follow-up through multiple binary logistic regression models, accounting for non-linearity. Both models included patient-level predictors for service connection and response to follow-up, including age as a continuous variable and gender and race as categorical. For interpretation, results are presented and described as odds ratios. The service connection model only included patients who responded to follow-up to account for the uncertainty of service connection for patients who could not be contacted and did not include patients recorded as pending (this included patients with appointments scheduled, but who have not connected with their service by time of follow-up) to account for patients who may not complete the appointment.

Additional data on reasons for referral noncompletion are assessed through frequencies. Implementation and Maintenance were quantitatively assessed via summarized survey results from the NAVSAT patient survey. Statistical tests were performed in R, version 4.3. This study was approved by Prisma Health’s Institutional Review Board.

Results

Reach

A total of 13,109 referrals were made for 11,214 unique patients to PSS from October 2019 through June 2023. Sample representativeness and demographics are presented in Table 2 for each patient at their first referral. The sample was significantly different from the available population of the geographic location served by our primary care clinics for all observed demographics. This study’s sample was younger (Mean age = 8.8) and had higher representation of female (51.0%) and Hispanic (28.5%) patients, with an under-representation of Black (4.7%) patients, compared to the general population. Most patients in this study were covered by Medicaid (61.1%), which greatly surpassed the national average of families covered by Medicaid and the percentage of families in South Carolina covered by Medicaid and the Children’s Health Insurance Program (CHIP) [31].

Table 2 Characteristics of PSS Patient population and pediatric population in Upstate, South Carolina

Primary referral reason is categorized into 8 categories: caregiver/family needs, developmental/behavioral, educational needs, high risk social behaviors, legal, medical specialties, mental/behavioral needs of parents, and mental/behavioral needs of children. Of all referrals made to PSS, the primary referral reason was for mental/behavioral needs of children (64.0%), followed by caregiver/family needs (19.9%).

Effectiveness

At time of two-week follow-up, 3,929 patients consented to follow-up and had outcomes recorded. Of these, 50.6% connected to services, 27.1% were pending connection (categorized as an appointment scheduled), and 22.3% were not connected. An adjusted model to estimate the relationship between service connection of mental health referrals and demographics demonstrated a significant increase in the odds of service connection as age increases (p < 0.001) (Table 3). Referral connection rate was significantly higher for Black patients compared to non-Hispanic White patients (OR = 1.33, p = 0.04), but significantly lower for patients in other racial groups (OR = 0.46, p = 0.01). Additionally, non-Hispanic Black patients had significantly higher rates of response to two-week follow up from navigators, compared to non-Hispanic White patients (OR = 1.26, p = 0.02) (Table 4).

Table 3 Results of binary logistic regression: estimated change in service connection rate based on demographics

Table 4 Results of binary logistic regression: estimated change in navigator two-week follow-up response rate by demographics

Navigators also documented identified barriers to service connection for patients who were not connected to services or who were waitlisted at the time of two-week follow-up. The primary barrier reported by patients’ legal guardian(s) at time of navigator follow-up was ‘service capacity issues/waitlisted services’ (29.2%), preventing patients from connecting with referred mental health services.

Adoption

PSS was implemented simultaneously with updated universal screening guidelines as a referral triage system for early detection of pediatric clinical and community-based needs. Universal screening guidelines were adapted to include the Survey of Well-being of Young Children (SWYC) [32], a comprehensive, evidence-based screening tool for children under age 5, to the existing Patient Health Questionnaire (PHQ-9) [33], which has a 100% systemwide clinic-level adherence rate to date, monitored by the health system (Fig. 2). PSS was piloted in two primary care clinics and has since expanded to 15 clinics systemwide, as of June 2023. Physicians were trained in screening and referral practices onsite and earned Part 4 Maintenance of Certification credits for their referral into PSS. Members of this research team have continuously provided ongoing quality improvement reports and technical assistance at provider and practice level.

Fig. 2

Survey of Well-being of Young Children (SWYC) adoption by percentage of participating primary care clinics by end of each study year

From 2021 to 2023, PSS was implemented in eight additional clinics, facilitated by the transition to online universal screening practices. Referrals have increased each year since PSS’ inception, with the exception of 2022. From 2019 to 2020 and 2020 to 2021, referrals increased by 19.86% and 18.1%, respectively. In 2022, referrals decreased by approximately 19.7%, compared to 2021. Ongoing referral acquisition is generated by provider-level adherence to universal screening protocol.

Implementation

From June 2022 to June 2023, a total of 54 participants have completed the NAVSAT survey. NAVSAT responses were 5-point Likert scales on either a satisfaction scale ranging from “very dissatisfied” to “very satisfied” or an agreement scale from “small degree” to “very large degree.” Results are summarized by collapsing “satisfied” and “very satisfied” as satisfaction and “large degree” and “very large degree” as agreement. Most (81.5%) of respondents reported being satisfied (n = 44) with navigators’ intake procedures at first contact to initiate their child(ren)’s services. Additionally, 79.6% (n = 43) of respondents were satisfied with their frequency of contact with the navigator and 75.9% (n = 41) agreed that their navigator understood the mental health system.

Maintenance

Maintenance measures on the NAVSAT showed high participant satisfaction with navigation services. Most (81.5%) of respondents reported being satisfied with overall pediatric navigation services, and 85.2% of respondents report that the navigator was “helpful” or “very helpful” in connecting them/their child(ren) with services. Majority of respondents were satisfied with the navigator’s ability to listen and understand caregiver concerns (81.5%), understand the impact of their family’s situation (75.9%), and provide appropriate information for potential treatment options (79.6%) for the child and/or family members. Overall, 92.6% of caregivers agreed that navigators recommended the most appropriate resources for their child or family member.

Discussion

The purpose of this study is to evaluate PSS along the RE-AIM framework to inform program impact and integration within primary care settings. The study’s primary findings informed impact of navigation services on referral connection rate and gaps in follow-up connection based on patient demographics that will be integrated into navigator protocols to address equitable impact and reach of the navigation system. Specifically, results of this study will be evaluated for their impact on three levels: the patient, provider, and system.

Patient level

The impact of evaluating program effectiveness and reach across different demographics is vital to understanding the full implementation of a program. Through this study, we found significant differences in three areas: 1) patients referred to PSS compared to those in the available population being serviced at our participating primary care clinics, 2) in connection to services within two-weeks, and 3) in patients who responded to two-week navigator follow-up. The differences found between the referred patients and the available population in this study may demonstrate a filtering of patients during their initial primary care visit to prioritize navigation referrals for patients deemed by their primary care providers as high-risk for referral noncompletion (e.g. patients on Medicaid and non-English speaking patients). The increase in connection rate and response to follow up as age increases may indicate increased severity or increased life disruption of conditions requiring navigation support. For example, a patient experiencing attention-deficit hyperactivity disorder at a younger age and not yet in school may not feel as motivated to seek early intervention as a patient finding it difficult to engage in the classroom. Lastly, racial differences in connection rate and response to follow-up, like the initial filtering at the primary care provider level, may indicate increased navigation support to patients deemed highest risk for referral noncompletion. Consistent with similar studies, these findings suggest a need for standardization of navigator and PNP processes to assess referral and inherent bias impact on PNP effectiveness.

These findings echo recent studies emphasizing the role of linking navigation, health technology, and patients’ electronic health records to improve access, accuracy, and understanding of patients’ comprehensive health and ongoing quality of care [34,35,36]. Based on findings from this study, we are currently piloting full integration of navigation documentation within electronic medical records, to support a closed-loop referral process from provider through navigation services.

Additionally, patient responses to the NAVSAT showed high levels of satisfaction with the navigators and PSS but could only be completed with patients who responded to follow-up, responded to the survey, and who had connected or were pending connection to services. The findings may overestimate patients’ caregiver satisfaction and emphasize the need to address satisfaction and participation for a larger, more diverse sample regarding demographics and service connection. Future research may benefit from smaller studies targeting participants lost to follow up or not connected to services to explore responses to navigation systems, navigators, and health system navigation self-efficacy.

Provider level

Providers’ systemwide demonstrate referral adequacy, with 100% clinic-level adherence to universal screening guidelines and increases in referrals to PSS annually until 2022. The observed increase in referrals from 2020 to 2021 may have been influenced by COVID-19, as children and their families experienced increased instability and may have required additional community resources [37,38,39]. However, barriers to service connection did not significantly include COVID-19 as a reason. Within this health system, changes in referrals in 2022 may have been potentially impacted by systemwide paused well-child checks for those not requiring vaccinations during the “tripledemic” of RSV, Influenza and COVID or decreased clinical needs following the onset of COVID-19 [40]; however, this has not been confirmed.

Still, this study found the primary barriers to service connection were waitlists and service capacity issues. Unfortunately, this is not a novel issue, as previous literature has identified long wait lists, inadequate referrals, limited bilingual providers, insurance gaps, and lack of closed loop referrals as barriers to service connection [41]. PSS seeks to adapt and automate navigation processes to improve provider capacity and reduce navigator burden, through identification of services with limited waitlists and personalized matching generation. Limited research currently exists surrounding the automation of primary care screening and navigation services among pediatrics, particularly within a network of primary care clinics [42, 43]. Such adaptations would provide a novel approach to pediatric navigation in primary care.

System level

The benefits of universal screening for early detection of developmental concerns among pediatrics are well-documented [44,45,46,47]. PSS employs a contemporary, two-generation approach through inclusion of social determinants of health screening (i.e., SWYC) and connection to community-based resources beyond the health system [32, 47]. PNPs connection to community-based support enhances the capacity of health systems by utilizing external resources and promoting the health of children and their families. The findings of this study also demonstrate the successful implementation of PNPs alongside system-level changes to support cohesion of provider training, support for influxes in identified patient needs, and to provide real-time program feedback loops and quality improvement. PNPs may increase sustainability within a health system by not relying on primary care providers to refer, connect, and follow-up with patients and their families, given their caseloads and vulnerability to physician burnout [48]. Navigators are key players in service connection among children and their families. System-level data provides ongoing support to adapt and maintain PSS as an intervention to enhance service and resource connection among children and their families, promoting carryover of care from primary care to community. This data supports the expansion of PSS across counties within the health system and emphasizes reach among special populations, such as those in rural areas or those in special populations who demonstrate relatively higher healthcare service utilization.

Limitations

The current data is reliant upon partial manual data entry at both patient caregiver contact and in medical chart review. Connection rate may not accurately estimate service connection due to patients lost to follow up and by patients who are unable to connect before the two-week follow-up. Future studies will follow-up at later dates to assess connection rate for pending patients. This study is also lacking a provider assessment to better understand ease of use of referrals and its impact on program sustainability. However, an author within this research team (KS) led the development and implementation of PSS and works as a pediatrician within this health system. Her insight to its development and connections with providers across primary clinics has played an integral role in the adoption and sustainability of this program, up to this point. Still, provider assessments may be considered in future research and evaluation efforts within this PNP. Lastly, for representativeness, insurance status was not available for patients specific to the available geographic area and were not included in this analysis, however, this sample over represents that national average of families receiving Medicaid benefits and the percentage of families covered by Medicaid and the Children’s Health Insurance Program (CHIP) in South Carolina.

Conclusion

The findings in this study suggest significant differences in PNP effectiveness and implementation across patients, potentially related to provider and navigator assessment of risk. However, without standardized protocols throughout the navigation process, the impact of PNPs is difficult to assess. As part of our ongoing implementation, we are working in collaboration with navigators and providers to support protocol standardization and tiered levels of support based on observed risk of referral noncompletion. Study findings will be used to inform delegation of navigation resources to patients at highest risk for lower service connection and loss to follow-up to ensure equitable distribution of program resources. Future research may benefit from exploring integration of closed loop referral systems to support completeness in referral connection data to inform gaps and high-risk groups requiring more support.