Reliability assessment of a hospital quality measure based on rates of adverse outcomes on nursing units.

Document Type


Publication Date



DOI: 10.1177/0962280215618688


The purpose of this study was to develop methods for assessing the reliability of scores on a widely disseminated hospital quality measure based on nursing unit fall rates. Poisson regression interactive multilevel modeling was adapted to account for clustering of units within hospitals. Three signal-noise reliability measures were computed. Squared correlations between the hospital score and true hospital fall rate averaged 0.52 ± 0.18 for total falls (0.68 ± 0.18 for injurious falls). Reliabilities on the other two measures averaged at least 0.70 but varied widely across hospitals. Parametric bootstrap data reflecting within-unit noise in falls were generated to evaluate percentile-ranked hospital scores as estimators of true hospital fall rate ranks. Spearman correlations between bootstrap hospital scores and true fall rates averaged 0.81 ± 0.01 (0.79 ± 0.01). Bias was negligible, but ranked hospital scores were imprecise, varying across bootstrap samples with average SD 11.8 (14.9) percentiles. Across bootstrap samples, hospital-measure scores fell in the same decile as the true fall rate in about 30% of cases. Findings underscore the importance of thoroughly assessing reliability of quality measurements before deciding how they will be used. Both the hospital measure and the reliability methods described can be adapted to other contexts involving clustered rates of adverse patient outcomes.

Journal Title

Statistical methods in medical research





First Page


Last Page


MeSH Keywords

Accidental Falls; Biostatistics; Health Personnel; Hospitals; Humans; Inpatients; Models, Statistical; Nursing Care; Patient Safety; Quality Assurance, Health Care; Quality of Health Care; Reproducibility of Results; Signal-To-Noise Ratio


Health care quality; quality measurement; reliability

Library Record