Reliability of Physician-Level Measures of Patient Experience in Primary Care

Background Patient experience measures are widely used to compare performance at the individual physician level. Objective To assess the impact of unmeasured patient characteristics on visit-level patient experience measures and the sample sizes required to reliably measure patient experience at the primary care physician (PCP) level. Design Repeated cross-sectional design. Setting Academic family medicine practice in California. Participants One thousand one hundred forty-one adult patients attending 1319 visits with 56 PCPs (including 45 resident and 11 faculty physicians). Measurements Post-visit patient experience surveys including patient measures used for standard adjustment as recommend by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Consortium and additional patient characteristics used for expanded adjustment (including attitudes toward healthcare, global life satisfaction, patient personality, current symptom bother, and marital status). Results The amount of variance in patient experience explained doubled with expanded adjustment for patient characteristics compared with standard adjustment (R 2 = 20.0% vs. 9.6%, respectively). With expanded adjustment, the amount of variance attributable to the PCP dropped from 6.1% to 3.4% and the required sample size to achieve a reliability of 0.90 in the physician-level patient experience measure increased from 138 to 255 patients per physician. After ranking of the 56 PCPs by average patient experience, 8 were reclassified into or out of the top or bottom quartiles of average experience with expanded as compared to standard adjustment [14.3% (95% CI: 7.0–25.2%)]. Conclusions Widely used methods for measuring PCP-level patient experience may not account sufficiently for influential patient characteristics. If methods were adapted to account for these characteristics, patient sample sizes for reliable between-physician comparisons may be too large for most practices to obtain.