Medical service performance evaluation of tertiary general hospitals in Sichuan Province in China based on diagnosis-related groups

Abstract

Objective

This study aims to assess the performance of medical services offered by tertiary general hospitals in Sichuan Province, China, by applying two comprehensive evaluation models. The findings may provide insights for policy-making and future scientific research.

Methods

Based on the diagnosis-related groups (DRGs) data, the Principal Component Analysis Modified Rank Sum Ratio (PMRSR) and Principal Component Analysis and Entropy Combined Weighted Technique for Order Preference by Similarity to an Ideal Solution Modified Rank Sum Ratio (CWTMRSR) models were developed to separately assess the medical service performance in 130 tertiary general hospitals in Sichuan Province, China. The results of the two evaluation models were compared with the head-to-tail consistency rate.

Results

The medical service performance of the 130 tertiary general hospitals was categorized into four groups using the PMRSR and CWTMRSR models. Among them, 86.92% were classified as “plain” and “medium”, 6.15% as “poor”, and 6.92% as “excellent”. The number of hospitals in each group generated by both models was consistent, with 8 “poor” hospitals, 57 “plain” hospitals, 56 “medium” hospitals, and 9 “excellent” hospitals. The “excellent” hospitals identified by the two models were all Grade A tertiary general hospitals. Furthermore, the head-to-tail consistency rate of the two evaluation models was 94.23%, indicating a strong consistency between the two models. Except for the cost efficiency index (CEI) indicator, the “excellent” hospitals demonstrated superior performance on indicators such as the case-mix index (CMI), number of DRGs (ND), total weight (TW), time efficiency index (TEI), mortality of middle and low-risk groups cases (MMLRG), and standardized cases fatality rate (SCFR) compared to the “poor” hospitals.

Conclusions

There are disparities in the performance of medical services offered by tertiary general hospitals in Sichuan Province, China. For hospitals categorized as “poor”, there is potential for them to strengthen their management capabilities and medical techniques to retain more suitable patients while increasing their ND, TW, and CMI indicators. Furthermore, it is crucial for them to significantly elevate their quality of care to effectively reduce the MMLRG and SCFR indicators to narrow the gap with the “excellent” hospitals. The combined application of the PMRSR and CWTMRSR models can improve the reliability and stability of the medical service performance evaluation.

Introduction

Aagja and Garg [1] defined hospital service performance as the discrepancy between patient’s or their attendants’ perceptions of the services offered by a particular hospital and their expectations regarding such services. In recent years, various dimensions of healthcare service performance have been utilized and discussed in global literature [2]. The scientific evaluation of medical service performance is essential for hospital administration and represents a crucial element in promoting hospital sustainability. However, due to the diversity and complexity of medical services [3], the variety of medical needs, and the information asymmetry in the healthcare system [4, 5], numerous indicators reflect medical service performance. For instance, traditional healthcare performance evaluation systems assess hospital inpatient services performance based on indicators such as average cost, length of stay (LOS), mortality [6], work efficiency, and workload [3].

However, practices have indicated that utilizing isolated indicators to evaluate hospital medical service performance is inadequate and inappropriate. These indicators are unable to reflect the connotation of medical services and cannot offer a comprehensive representation of different medical service performances, thereby raising concerns regarding comparability, comprehensiveness, and representativeness [6, 7]. Previous research has demonstrated that to ensure the reliability of evaluating medical service performance, it is necessary to adjust the system risk in its evaluation [8, 9]. One widely employed method to address this issue is to adjust the risk between cases through case-mix [10]. The Diagnosis-Related Groups (DRGs), developed by Fetter Robert of Yale University in 1967 [11], were designed to tackle these issues. DRGs are a globally recognized advanced hospital management tool. As a pragmatic management tool, DRGs comprehensively consider the different demographic characteristics of patients and divide a class of patients with similar clinical causes and treatments into the same DRG group for management purposes [11]. This augments data comparability in both horizontal and vertical comparisons across different hospitals or departments. DRGs have been employed extensively in various areas such as pricing and payment systems, budget allocation, performance evaluation, and comparison [12].

DRGs in China have been extensively researched for an extended period. Currently, there exist three main national-level versions. The initial version implemented was China National Diagnosis Related Groups (CN-DRG), which was utilized for hospital performance evaluation and administration in 2013. Subsequently, the second version implemented was China Diagnosis Related Groups (C-DRG), which was employed for pricing and payment in 2017. Finally, the third version implemented was China Healthcare Security Diagnosis Related Groups (CHS-DRG), which was utilized for health insurance payment in 2019 [13]. In 2013, the then-Chinese National Health Commission’s Bureau of Medical Administration introduced the CN-DRG for hospital performance management and evaluation. After years of practical implementation, the Chinese government mandated the initiation of DRGs piloting at the national level in 2017 [13], indicating the commencement of DRGs management and evaluation at the national level. At the regional level, provinces and municipalities like Shanghai, Zhejiang, and Sichuan have formulated their specific policies and regulations related to DRGs. They have also developed region-specific versions of DRGs, such as Medicare-Severity DRGs (MS-DRG) based on national DRGs versions, in consideration of their unique population characteristics, disease structures, and levels of economic development [13]. Following governmental administrative requirements, hospitals of all levels and types have been encouraged to adopt DRGs for management, thereby erecting the basis for conducting a comprehensive medical services performance evaluation.

In China, the literature concerning performance evaluation of medical services has dramatically increased since 1980 [14]. At that time, “Key Performance Indicators (KPI)” and “Balance Score Card (BSC)” methods were ubiquitously utilized for hospital medical service performance evaluation, focusing on indicators such as medical cost, LOS, and medical quality [15,16,17]. However, the reliability of these results was questionable due to the lack of risk adjustment. Recognizing the importance of risk adjustment in performance evaluation and its advantages in terms of objectivity, reliability, impartiality, inclusiveness [18], and being able to combine with other evaluation methods [19], researchers at Peking University developed the DRGs system as a performance evaluation tool in 2005 [20,21,22]. Subsequently, numerous studies have attempted to develop models based on DRGs for evaluating the performance or quality of medical services.

Our review of the literature highlighted that despite the rapid advances in medical service performance evaluation using DRGs tools, there is a scarcity of combined models incorporating methods of principal component analysis (PCA), Combined Weighted Technique for Order Preference by Similarity to an Ideal Solution (CWTOPSIS), PCA-Modified rank sum ratio (PMRSR) and CWTOPSIS-Modified rank sum ratio (CWTMRSR) for evaluating medical service performance based on DRGs data. Previous studies conducted the medical service performance evaluation by adopting various isolated methods and were limited to a specific region, hospital, department, or case [14]. For example, XK Liu (2021) [23] evaluated the performance of medical services for breast cancer patients using 6 inherent indicators of CN-DRG. JJ Lu (2023) [24] assessed the organ transplant department’s performance by relying on the same 6 indicators of DRGs. Furthermore, X Dai (2023) [25] evaluated the public health service effect in a city in the Inner Mongolia Autonomous Region. They utilized entropy-weighted TOPSIS and RSR models, as well as developed an entropy-weighted TOPSIS and RSR fuzzy combination model, to rank 14 primary health care service centers. Their findings suggest that the joint entropy-weighted TOPSIS and RSR model can evaluate the effectiveness of basic public health services more comprehensively and holistically.

To the best of our knowledge, there have been no studies utilizing PCA, CWTOPSIS, PMRSR, and CWTMRSR models for evaluating medical service performance at the provincial level based on DRGs data. Therefore, this study introduces novel comprehensive evaluation models to evaluate the medical service performance of 130 hospitals in Sichuan Province in Southwest China. The comprehensive evaluation procedures are as follows: 1) The PCA method was utilized to rank hospitals; 2) The CWTOPSIS method was used to rank hospitals; 3) The RSR method was employed in combination with the PCA and the CWTOPSIS methods to formulate PMRSR and CWTMRSR models for categorizing hospitals’ medical service performance based on DRGs data, respectively. The findings of this study may provide valuable insights for healthcare administrators in policy-making and future scientific research.

Methods

Data sources

Medical service performance evaluation was conducted in tertiary general hospitals in Sichuan Province, excluding specialized hospitals such as traditional Chinese medicine hospitals and hospitals of traditional Chinese medicine and Western medicine.

In 2016, the former Sichuan Province Health and Family Planning Commission uniformly implemented a system called the “Front Page of Medical Records (FPMR) (2014 edition)” [26], which utilized standardized formats to collect patients’ medical data, such as diagnosis and treatment information across the region. The Sichuan Health Data Analysis and Decision Support Cloud Platform (SHDADSCP) employed the region-specific MS-DRG to analyze the FPMR data provided by hospitals enrolled in SHDADSCP and constructed a comprehensive evaluation matrix consisting of 7 indicators.

The data utilized in this study was obtained from the SHDADSCP of tertiary general hospitals in 2022 [27]. To ensure scientifically sound evaluation results, the study encompassed all tertiary general hospitals enrolled in SHDADSCP, resulting in a total sample size of 130, including 53 Grade A tertiary hospitals and 77 Grade B tertiary hospitals. Grade A represents the highest level and Grade B represents the second highest level within the Chinese hospital classification system. Geographically, the 130 hospitals were distributed across all 21 municipalities and autonomous prefectures of Sichuan Province (Fig. 1).

Fig. 1

Geographical and hospital-level distribution of 130 hospitals

Indicator determination

The hospital performance evaluation section of the SHDADSCP proposed a medical performance evaluation matrix consisting of three primary indicators and seven secondary indicators [27]. By referring to relevant literature [28], we selected three primary indicators of medical service ability, medical service efficiency, and medical service security, denoted as E₁, E₂, and E₃ respectively. Additionally, seven secondary indicators, including case-mix index (CMI), number of DRGs (ND), total weight (TW), cost efficiency index (CEI), time efficiency index (TEI), mortality of middle and low-risk groups cases (MMLRG), and standardized cases fatality rate (SCFR) were selected and labeled as X1, X2, X3, X4, X5, X6, and X7 respectively. Based on previous research, CMI, ND, and TW were defined as positive indicators, while CEI, TEI, MMLRG, and SCFR were defined as negative indicators [25, 29, 30]. Subsequently, the evaluation matrix of the medical service performance in this study was established (Table 1).

Table 1 Medical service performance evaluation matrix based on DRGs

Procedures for the construction of the evaluation models

In the comprehensive evaluation of medical service performance, to make full use of the original data and mitigate the influence of the data itself on the evaluation results [31], the RSR method was combined with the PCA and CWTOPSIS methods in our study to separately develop the PMRSR and CWTMRSR models. The integration of these three methods enhances the reliability and stability of the comprehensive evaluation results compared to using PCA, TOPSIS, or RSR alone, thereby increasing the research’s distinctiveness and measurability [32]. Statistical analysis of the relevant data was performed using Excel 2016 and SPSS 26.0 software. A P value less than 0.05 was considered as statistically significant.

Principal component analysis (PCA) method

PCA is a widely utilized approach for integrating information exacted from original indicators, exploring the significant influences of original indicators, and evaluating performances based on comprehensive evaluation model scores. By reducing multidimensional data into two or three dimensions, the overall features of the dataset can be converted into a visual format that allows for simple interpretation [31]. It can be used to address collinearity among various items [32] such as DRGs indicators, and to reduce them into meaningful components for further analysis. In the medical field, PCA has been extensively adopted to identify patterns for comprehensive evaluation [33,34,35,36,37]. We conducted the PCA evaluation on hospital medical service performance using the following procedures.

• 1. Extraction of data. The original data matrix was extracted and set as X_ij, where i = 1,2,…,m; j = 1,2,…,n; m represents the number of secondary indicators; n represents the number of study hospitals.

• 2. Standardization of data. The data was standardized through a two-step process. Firstly, negative indicators such as CEI, TEI, MMLRG, and SCFR were normalized using Eqs. (1)-(2), then followed by a dimensionless transformation of all indicators using Eq. (3) [38].

$${X^{\prime}}_{\text{ij}} = \frac{1}{X_{\text{ij}}}$$

(1)

$${X^{\prime}}_{ij} = 100 - X_{ij}$$

(2)

$$Z_{\text{ij}} = \frac{X^{\prime}_{ij} - \bar{X}^{\prime}_{ij}}{S_{j}}$$

(3)

• 3. Kaiser–Meyer–Olkin (KMO) and Bartlett’s test of sphericity. Before conducting PCA, it is essential to assess the standardized data for its appropriateness. The commonly used methods for this assessment are the KMO measure and Bartlett’s test of sphericity. The KMO value ranges from 0 to 1, with a higher value indicating stronger correlations between indicators, thus making the data suitable for PCA. Conversely, a lower KMO value suggests weaker correlations and unsuitability for PCA. Bartlett’s test of sphericity examines whether the covariance matrix in a factor analysis model is an identity matrix. When the KMO value exceeds 0.6 or 0.7 and Bartlett’s test yields a significance level below 0.05, the data are regarded as appropriate for PCA.

• 4. Calculate the variance–covariance matrix based on Z_ij and labeled it as the matrix R_ij, where i = 1,2,…,m; j = 1,2,…,n; m represents the number of secondary indicators; n represents the number of study hospitals.

• 5. Principal components (PC_S) were generated by calculating the eigenvector and the eigenvalue based on the matrix R_ij, and were a linear combination of the standardized data Z_ij.

$$C_{i} = \frac{q_{i}}{\sqrt{\lambda_{i}}} , \text{where}\ q_{i}\ \text{is eigenvector},\ \lambda_{i}\ \text{is eigenvalue},\ C_{i}\ \text{is the coefficient of}\ PC_{s}$$

(4)

$$PC_{s} = \sum\limits^{m}_{\text{i} = 1}\ \, C_{i} \times Z_{ij}$$

(5)

• 6. The number of PC_S is determined by the principle of accumulative variance contribution ≥ 80% [39], and complying with practical meanings.

• 7. The PCA comprehensive evaluation model was established based on PC_S and was labeled as EP_C.

$$EP_{c} = \sum\limits^{m}_{i = 1} \frac{\lambda_{i}}{\sum\limits^{m}_{i = 1} \lambda_{i}} \times PC_{s}$$

(6)

The value of EP_C was utilized for ranking the 130 hospitals. A higher EP_C value indicates a better ranking result, whereas a lower EPC value represents a worse ranking result.

• 8. The 130 hospitals were compared and ranked based on the values of EP_C.

PCA-Entropy combined-weighted TOPSIS (CWTOPSIS) method

Weight was widely acknowledged as a crucial factor due to its influence on the calculation results of evaluations. Various models, such as PCA, Entropy, Analytic Hierarchy Process (AHP), and Delphi, could be applied to determine the weights of evaluation indicators [40,41,42,43]. In this study, we combined the PCA and Entropy models to determine weights based on the following equations to control indicator collinearity and ensure objectivity in weight determination.

• 1. Determination of primary indicators’ weights (W_P) based on the PCA model.

$$W_{p} = \frac{\lambda_{i}}{\sum\limits^{m}_{i=1}\lambda_{i}}, \text{where}\ \lambda_{i}\ \text{is eigenvalue}$$

(7)

• 2. Determination of secondary indicators’ weights (W_S) based on the Entropy model.

  • 1) The original data matrix X_ij was normalized using the following equations, where i = 1,2,…,m; j = 1,2,…,n; m represents the number of secondary indicators; n represents the number of study hospitals.

    $$\text{Positive indicators:}\ Y_{ij} = \frac{X_{ij} - \min(X_{ij})}{\max(X_{ij}) - \min(X_{ij})}$$

    (8)
    $$\text{Negative indicators:}\ Y_{ij} = \frac{\max(X_{ij}) - X_{ij}}{\max(X_{ij}) - \min(X_{ij})}$$

    (9)

  • 2) The normalized data matrix was added 0.001 to eliminate the influence of zero value. 

    $${Y^{\prime}}_{ij} = Y_{ij} + 0.001$$

    (10)

  • 3) The value of P_ij was calculated.

    $$P_{\text{ij}} = \frac{Y^{\prime}_{j}}{\sum\limits^{n}_{j=1}Y^{\prime}_{j}}$$

    (11)

  • 4) The value of E_j was calculated.

    $$E_{j} = \frac{1}{Ln(n)}\sum\limits^{m}_{i=j} P_{ij} \times Ln(P_{ij})$$

    (12)

  • 5) The weights of secondary indicators were calculated.

$$W_{s} = \frac{1-E_{j}}{m - \sum\limits^{m}_{j=1} E_{j}}$$

(13)

• 3. Calculation of PCA-Entropy combined weights of evaluation indicators.

$$W_{cw} = W_{p} \times W_{s}$$

(14)

The final weights of evaluation indicators are shown in Table 2.

Table 2 Determination of weights of evaluation indicators

The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) is a multi-criteria decision analysis model utilized to identify solutions from a finite set of alternatives based upon simultaneous minimization of distance from an ideal point and maximization of distance from a nadir point [44]. This method utilizes a normalized raw data matrix to form a space for both positive and negative ideal solutions of priority solutions. The solutions under evaluation are considered as points in this space. The distance between a point and the positive and ideal solutions is obtained. This distance aids in identifying the relative closeness between the solution to be evaluated and the positive ideal solution, providing a basis for evaluating its advantages and disadvantages [45]. The weights of indicators must be predetermined before employing the TOPSIS model, whether through subjective or objective means [46]. We conducted PCA combined with Entropy-weighted TOPSIS using the following steps.

• 1. The normalized data matrix ${Y^{\prime}}_{ij}$ of the Entropy Model was used.

• 2. The PCA and Entropy combined-weighted normalized data matrix was formulated.

  $$Z_{ij} = W_{cw} \times {Y^{\prime}}_{ij}$$

  (15)

• 3. The weighted Euclidean distance ${D^{+}}_{i}$ and ${D^{-}}_{i}$ of each indicator to the optimal scheme and the worst scheme was calculated, then the approximation degree Ci to the optimal scheme was calculated based on ${D^{+}}_{i}$ and ${D^{-}}_{i}$.

$${D^{+}}_{i} = \sqrt{\sum\limits^{n}_{i=1} \left(Z_{ij} - \max Z_{ij}\right)^{2}}$$

(16)

$${D^{-}}_{i} = \sqrt{\sum\limits^{n}_{i=1} \left(Z_{ij} - \min Z_{ij}\right)^{2}}$$

(17)

$$C_{i} = \frac{{D^{-}}_{i}}{{D^{+}}_{i} + {D^{-}}_{i}}$$

(18)

The value of C_i was between 0–1. A C_i value close to 1 indicates a higher likelihood of positive ideal solution (optimal level), whereas a C_i value close to 0 indicates a higher likelihood of negative ideal solution (worst level).

• 4. The 130 hospitals were compared and ranked based on the values of Ci.

The PCA-Modified Rank Sum Ratio (PMRSR) and CWTOPSIS-Modified Rank Sum Ratio (CWTMRSR) models

The RSR comprehensive evaluation method, proposed by Chinese statistician Professor Fengtiao Tian in 1988 [47], involves constructing a matrix of m columns and n rows and using rank conversion to create a dimensionless statistical indicator, is a mature and comprehensive evaluation method. Parametric statistical methods are employed to study the distribution of RSR [47]. The RSR value is utilized for direct ranking of evaluation objects’ level or comparison of confidence intervals of the RSR for each group [47]. In our study, the RSR model was initially developed to rank and classify the medical service performance of 130 tertiary general hospitals. To enhance the reliability and stability of medical service performance evaluation results, we established the PMRSR and CWTMRSR models to separately conduct comprehensive evaluations on these 130 hospitals. The detailed processes are as follows.

• 1. The RSR value was substituted with the values of EP_C and C_i.

• 2. The values of EP_C and C_i were ordered from smallest to largest.

• 3. The downward cumulative frequency P was calculated.

  $$P = \frac{\bar{R}}{n} \times 100\%,\ \text{where}\ \bar{R}\ \text{is average rank.}$$

  (19)

• 4. The Comparison Table of Percentage and Probability Unit [48] was used to determine the P’s corresponding Probit.

• 5. The EP_C and C_i were regarded as dependent variables, with Probit serving as the independent variable for calculating the linear regression equations of medical service performance evaluation. These equations were subsequently utilized to compute the fitted values of EP_C and C_i for each hospital.

$$\hat{E}P_{c} = \text{a} + b \times probit$$

(20)

$$\hat{C}_{i} = c + d \times probit$$

(21)

• 6. The 130 hospitals were ranked from worst to best of medical service performance based on the fitted values of $\hat{E}P_{c}$ and $\hat{C}_{i}$.

• 7. The 130 hospitals were categorized according to the reasonable classification principle proposed by Professor Fengtiao Tian [49]. Probit critical values were utilized in the regression equation to calculate the RSR critical values for classification. All 130 study hospitals were then classified into four categories: excellent, medium, plain, and poor. Variance analysis was employed to compare differences between groups and LSD was used for pairwise comparisons.

The performance comparison of the PMRSR and CWTMRSR models

To assess the degree of inconsistency among different comprehensive evaluation models, Professor Yu LP (2008) introduced the concept of head-to-tail consistency rate [50], which has been widely acknowledged by researchers. The equation,

$$S = \frac{x + y}{0.4n}$$

(22)

where x denotes the number of identical evaluation objects within the top 20%, y denotes the number of identical evaluation objects within the bottom 20%, and n represents the total number of evaluation objects. The value of S ranges from 0–1, with a higher value indicating greater consistency among different comprehensive evaluation models.

Results

PCA ranking results

The KMO and Bartlett’s test of sphericity showed that the original data met the preconditions for PCA, with a KMO value of 0.747 and P = 0.000 < 0.05. Following the principal components extraction principle, three principal components were extracted. PC₁ accounted for 49.52% of the total variance, with an eigenvalue was 3.4667; PC₂ contributed 20.47% of the total variance, with an eigenvalue was 1.4331; and PC₃ contributed 12.79% of the total variance, with an eigenvalue was 0.8956. Together, PC₁, PC₂, and PC₃ explained 82.79% of the accumulative variance contribution (Fig. 2).

Fig. 2

Accumulative variance contribution of principal components

The eigenvector matrix for all evaluation indicators of the three significant PCs was presented and eigenvectors with values greater or equal to 0.5 were considered significant (Table 3).

Table 3 The eigenvector matrix of three significant PCs

The PC₁ possessed higher eigenvectors on CMI, ND, and TW, indicating its representation of the “medical service ability” component. The PC₂ showed greater eigenvectors on MMLRG and SCFR, suggesting its representation of the “medical service security” component. The PC₃ displayed higher values on CEI and TEI, illustrating its representation of the “medical service efficiency” component. We calculated the coefficient of each PC based on both eigenvector and eigenvalue values and obtained the equations of the three significant PCs as follows.

$$\text{PC}_{1} = 0.479 \times \text{Z}_{11} + 0.465 \times \text{Z}_{12} + 0.479 \times \text{Z}_{13} - 0.361 \times \text{Z}_{14} + 0.323 \times \text{Z}_{15} + 0.231 \times \text{Z}_{16} + 0.191 \times \text{Z}_{17}$$

(23)

$$\text{PC}_2 = -0.181 \times \text{Z}_{21} - 0.059 \times \text{Z}_{22} - 0.144 \times \text{Z}_{23} + 0.172 \times \text{Z}_{24} - 0.116 \times \text{Z}_{25} + 0.654 \times \text{Z}_{26} + 0.688 \times \text{Z}_{27}$$

(24)

$$\text{PC}_3 = -0.150 \times \text{Z}_{31} + 0.050 \times \text{Z}_{32} + 0.075 \times \text{Z}_{33} + 0.609 \times \text{Z}_{34} + 0.763 \times \text{Z}_{35} - 0.111 \times \text{Z}_{36} + 0.062 \times \text{Z}_{37}$$

(25)

Based on PC₁, PC₂, and PC₃, along with their corresponding weights, we have successfully developed the comprehensive ranking equation of PCA and labeled it as EP_C.

$$\text{EP}_{\text{C}} = 0.598 \times \text{PC}_1 + 0.247 \times \text{PC}_{2} + 0.155 \times \text{PC}_{3}$$

(26)

Through the calculation of EP_C, we were able to derive scores of every study hospital and subsequently rank them based on the EP_C values. The ranking results showed that all of the top 10 hospitals were grade A tertiary hospitals, with Chengdu municipality occupying the largest number of hospitals, totaling four in the top 10 (Table 4).

Table 4 The ranking results of the PCA method

PCA-Entropy-Combined-Weighted TOPSIS (CWTOPSIS) ranking results

Based on the PCA and Entropy methods, we obtained the combined weights of evaluation indicators. These weights were then integrated with the normalized data matrix to construct a weighted normalized data matrix for TOPSIS evaluation. Subsequently, the 130 hospitals were ranked based on their C_i values. The ranking results showed that all of the top 10 hospitals were also Grade A tertiary hospitals, with Chengdu municipality occupying the largest number of hospitals, totaling four in the top 10 (Table 5).

Table 5 The ranking results of the CWTOPSIS method

PMRSR and CWTMRSR comprehensive evaluation results

Treating EP_C and C_i as the dependent variables and Probit as the independent variable, we obtained the regression equations of $\hat{E}P_{c}$ and $\hat{C}_{i}$ as follows.

$$\hat{E}P_{c} = 1.15 \times \text{Pr}\,obit - 5.774$$

(27)

$$\hat{C}_{i} = 0.064 \times \text{Pr}\,obit - 0.01$$

(28)

The results of the variance analysis and adjusted coefficient of determination showed that the two regression equations were statistically significant (F = 6368.695, P < 0.001; F = 556.936, P < 0.001) and presented good fitness (adjusted R² = 0.980; adjusted R² = 0.812). The Probit critical values of 3.5,5 and 6.5 were substituted into the two regression equations to calculate the RSR as the basis for classification purposes, followed by calculating the fitted values of $\hat{E}P_{c}$ and $\hat{C}_{i}$ for all hospitals. The final results are presented in supplementary Table 6.

Performance test of PMRSR and CWTMRSR models

The head-to-tail consistency rate was calculated utilizing the aforementioned equation. The high head-to-tail consistency rate of 0.94 for both PMRSR and CWTMRSR models indicates a strong level of agreement in the comprehensive evaluation results. This finding enhances the reliability and stability of our evaluation and strengthens the applicability of the methods utilized in this study.

Discussion

In 2018, the former Sichuan Province Health and Family Planning Commission conducted comprehensive supervision over medical institutions, personnel, and behaviors, which was known as the “three medical supervisions” [51]. As part of this supervision system, the use of DRGs indicators was incorporated as a monitoring strategy to establish an objective evaluation system for medical service performance. Compared to traditional evaluation indicators, DRGs indicators are interrelated and mutually constrained [52]. For instance, to simultaneously increase the value of CMI while decreasing CEI and TEI, hospitals are required to enhance their medical techniques to treat more severe patients while also making significant efforts to reduce inpatients’ LOS, lower patients’ medical expenses, and ensure their medical security. The efficiency of evaluation will be enhanced due to the properties of DRGs indicators [52]. Additionally, the application of DRGs indicators can overcome the limitations associated with single index methods and the challenges in horizontal comparisons. However, when utilizing DRGs indicators for comprehensive evaluations, it is essential to consider the issues of multicollinearity and weighting. Potential solutions to address these concerns include employing methods such as PCA and Entropy [32].

This study utilized the PCA and Entropy methods to tackle the issues of multicollinearity and weighting in DRGs indicators. Multicollinearity, which refers to the presence of collinearity among multiple indicators, can lead to incorrect statistical inferences [32]. Through the application of the PCA method, the original seven indicators were represented by three significant PCs, thereby diminishing the interaction effects among the original indicators and improving the evaluation accuracy. Additionally, a combined weight method using PCA and Entropy was employed to calculate the weights for each indicator, enhancing their ability to accurately reflect the genuine medical service performance level of hospitals [53]. When developing an evaluation model with selected evaluation indicators, it is important to consider the impact of each indicator on the results, and the determination of indicator weights plays a decisive role [54]. Currently, there are numerous subjective and objective methods for determining indicator weights, each possessing its own advantages and limitations. In this study, we utilized the variance contributions of PCs to determine the primary indicators’ weights, followed by the application of the Entropy method to determine the secondary indicators’ weights. This weight-determination process represents an objective method based on mathematical theories, ensuring the generation of reliable and applicable evaluation results that can be widely applied to comprehensive evaluations of medical services [55]. Building upon these foundations, we have developed novel PMRSR and CWTMRSR models for comprehensive evaluations. This combined evaluation model may possess more adaptable attributes than traditional techniques as it considers the strengths and weaknesses of each method and complements them to achieve more precise and reliable evaluation results [56].

This study has developed the PMRSR and CWTMRSR models to assess the performance of medical services in 130 hospitals. In 2008, Professor Yu LP raised the issue that despite aiming to evaluate the same objects using the same indicators and data, different models may generate inconsistent results. This issue has been acknowledged by numerous researchers and warrants further investigation [50]. In contrast to prior research studies [57,58,59], we also employed the head-to-tail consistency rate to examine the consistency between the results generated by both models. The findings suggest a high level of consistency in the comprehensive evaluation results, with a consistency rate of 94.23% in both models. The categorization results of the PMRSR and CWTMRSR models are consistent, revealing 8 (6.15%) poor hospitals, 57 (43.85%) plain hospitals, 56 (43.08%) medium hospitals, and 9 (6.92%) excellent hospitals. This serves as evidence that we have obtained reliable evaluation results using both models. However, some disparities between the results of the two models should also be acknowledged. For instance, the 9 excellent hospitals exhibit a higher consistency rate (88.89%) compared to the 8 poor hospitals (62.50%). This could be attributed to two possible reasons: firstly, all 9 excellent hospitals in Sichuan province are Grade A tertiary hospitals with relatively balanced development and quite similar data from different indicators; Secondly, the 8 poor hospitals are predominantly Grade B tertiary hospitals with diverse developmental characteristics, leading to significant variations in data from different indicators. This may result in greater disparities between various evaluation models [50]. Despite minor disparities between the two models, their combined implementation still yields stable and reliable evaluation results, as demonstrated by Zhao HJ’s (2019) research on the comprehensive evaluation of the "Basic Standards of Hospital Traditional Chinese Medicine Pharmacy" implementation status in Sichuan [60].

According to the categorization results generated by the PMRSR and CWTMRSR models, there are significant professional implications that warrant discussion. Firstly, disparities in the performance of medical services among 130 hospitals in Sichuan Province are observed. The majority (93.08%) of hospitals are categorized as “plain”, “medium” and “poor”, with only a small proportion (6.92%) categorized as “excellent”. Notably, all “excellent” hospitals are Grade A tertiary hospitals, whereas all “poor” hospitals are predominantly Grade B tertiary hospitals. In 2021, the Health Commission of Sichuan Province promulgated a government document titled “Implementation Rules for the Evaluation Criteria of Tertiary Hospitals in Sichuan Province (2021 edition)” to guide tertiary hospitals in enhancing their daily management and constantly improving medical qualities [61]. The findings of our study indicate that although all tertiary hospitals in Sichuan Province are mandated to adhere to the same construction standards, Grade A tertiary hospitals demonstrate superior performance in medical services compared to Grade B hospitals. Additionally, when comparing “excellent” and “poor” hospitals using seven evaluation indicators, we observed that except for the CEI indicator, the “excellent” hospitals outperformed the “poor” hospitals in all other indicators. The average values of CMI, ND, and TW for the “excellent” hospitals exhibit an increase of 55.99%, 47.05%, and 1097.72% respectively compared to those of the “poor” hospitals. Additionally, the average values of TEI, MMLRG, and SCFR indicators for the “excellent” hospitals demonstrate a decrease of 47.84%, 523.39%, and 167.70% respectively compared to those of the “poor” hospitals. The responsibilities of tertiary hospitals in China are defined as admitting and treating severe, complicated, and acute patients [62]. However, the comprehensive evaluation results provide compelling evidence that there is still room for improvement in the medical service performance of “poor” hospitals to align with their functional positioning through enhancements in various dimensions such as medical ability, efficiency, and security. For example, they may further strengthen their management skills and medical techniques to retain more suitable patients while increasing their ND, TW and CMI indicators. More importantly, it is crucial for them to significantly improve their medical qualities, thereby substantially reducing MMLRG and SCFR indicators to narrow the gap with “excellent” hospitals. Secondly, we explore the variations among indicators within the categorization of “excellent” hospitals. To assess indicator variation, we calculate the coefficient of variance (CV) and identify TW (CV = 58.18%) and MMLRG (CV = 62.11%) as having the highest degree of variation. The top 1 hospital (hospital 1) identified by both models has a TW value of 338,473.86, while the top 9 hospitals (hospital 38 identified by the PMRSR model, hospital 18 identified by the CWTMRSR model) have TW values of 83,169.8 and 88,231.39, respectively. The TW value of Hospital 1 is 3.07 and 2.84 times higher than those of Hospital 38 and Hospital 18, respectively. Hospital 1 serves as the medical center for China’s western region and is the leading hospital located in the capital city of Chengdu in Sichuan Province. Due to its advanced medical equipment, highly skilled medical personnel, and exceptional medical techniques, patients from the province and surrounding areas prefer to seek treatment at this hospital. As a result, there is a substantial increase in the number of cases for each DRG, leading to a corresponding improvement in the TW value. Conversely, despite being ranked first among the 130 hospitals, hospital 1 has the highest MMLRG value (0.17) compared to those of hospital 38 (0.09) and hospital 18 (0.00). This discrepancy may be influenced by the combined weight of evaluation indicators. Among the seven evaluation indicators, TW holds the highest combined weight (39.95%), followed by CMI (13.80%) and MMLRG (12.47%). Additionally, data indicates that both TW and CMI values of hospital 1 are significantly higher than those of hospitals 18 and 38, which may account for this evaluation result. Future research is anticipated to further investigate the mechanisms of how different weights affect comprehensive evaluation results.

Limitations and future research

There are several limitations of this study that need to be acknowledged. Firstly, the quality of the FPMR is crucial for obtaining accurate DRGs indicators, which are essential for a comprehensive evaluation of medical services. Despite the establishment of standard formats and strict quality control by the health authorities and medical record management departments, errors in the FPMR data provided by hospitals may still occur due to inappropriate coding of diseases and misunderstanding of the criteria, which could potentially undermine the accuracy of the evaluation results. Secondly, given the numerous DRGs versions currently utilized in China, including the CN-DRG, C-DRG, and CHS-DRG, it is important to note that the performance evaluation in this study is based on the MS-DRG. Therefore, the results may not be applicable to regions using different DRGs versions. Thirdly, despite the rapid development of comprehensive evaluation of medical services, most results still remain at a theoretical level. Large-scale practical applications are anticipated in the future. Lastly, this study has observed potential influencing effects from different DRGs indicators’ weights on the comprehensive evaluation results. Future research is needed to thoroughly discuss these underlying mechanisms.

Conclusions

Various methodologies can be employed to develop models for evaluating the performance of medical services. DRGs, in particular, are a widely recognized risk adjustment tool that can be adopted to objectively evaluate the performance of medical services across different medical institutions and regions. The combination of the RSR method with PCA, Entropy, and TOPSIS methods enables the construction of models for assessing the performance of medical services, thereby increasing the likelihood of generating consistent and reliable evaluation results. Differences in the performance of medical services among hospitals in Sichuan Province have been observed using the PMRSR and CWTMRSR models developed within this study. For hospitals categorized as “poor”, there is potential for them to strengthen their management capabilities and medical techniques to retain more suitable patients and simultaneously boost their ND, TW, and CMI indicators. Furthermore, it is crucial for them to significantly elevate their standards of care to effectively reduce MMLRG and SCFR indicators to narrow the gap with “excellent” hospitals.