A new approach to analyzing the Achilles' heel of multisource feedback programs: Can we really trust ratings of leaders at the group level of analysis?

This research addresses the Achilles’ heel of all multi-source leadership feedback programs regardless of whether they use 360°, 270°, or 180° ratings. Namely, should all feedback be universally aggregated by feedback groups and reported to respondents? A sample of raters from the Center for Creative Leadership (N=15,500 including subordinates, peers, and others) was grouped to align with each raters’ matching focal subject (J=1,550). Two scales from CCL’s Benchmarks instrument (Building and Mending Relationships and Career Management) were used to: (1) predict a derailment factor (Problems with Interpersonal Relationships) and (2) determine the operative level of analysis and its aggregation characteristics using Within and Between Analysis (WABA). In many cases, the variables under consideration did not aggregate optimally. Using a technique based upon each group’s coefficient of variation as a measure of internal consensus, between 3% and 25% of groups should not have their averages reported. Alternative approaches are discussed.