Reliability and Validity of Proposed Risk Stratification Methods for Laboratory Developed Tests

To determine whether different laboratory developed test (LDT) risk stratification proposals would assign differing levels of risk to selected LDTs as a measure of the validity of those proposals, and whether there would be differing interrater agreement rates as a measure of the reliability of those proposals. A total of 4 reviewers applied 6 proposals for risk stratification of 4 LDTs. Interrater agreement was calculated as a measure of the reliability of the proposals. Also, a consensus risk categorization and concordance rate for each LDT was developed as a measure of the validity of the proposals. Interrater agreement rates (reliability) ranged from 38% to 100%, and concordance rates (validity) ranged from 20% to 100%. A spectrum of reliability and validity was observed depending on the policy used and the LDT categorized. Before implementation or legislation of risk-stratification methods, large evaluations of reliability and validity should be conducted on any proposed method.