Scoring System Improvements to Three Leadership Predictors

This project sought to examine and improve the reliability of the scoring systems for three instruments which have been used in previous Army leadership research. Review of existing literature and interviews with project staff participating in prior research provided initial information concerning the strengths and weaknesses of the scoring systems for the three instruments. This information was used to recommend modifications to the original scoring systems. Six individuals were trained to use the modified scoring systems. The modified scoring systems were evaluated by rescoring responses randomly selected from the sample which had been scored according to the scoring systems originally developed for the leadership research program. Reliability estimates for the three modified scoring systems were consistently strong and showed improvements over those obtained through the original scoring systems. Interrater agreement indices were significant for nearly all ratings. Validity estimates provided evidence that each modified instrument was moderately to highly correlated with conceptually similar scores generated through the original scoring scheme. The report recommends use of the revised rating systems in future research to improve the quality of measurement from the three predictors. Prepared in cooperation with George Mason Univ., VA