Outcomes of Endovascular Treatments of Aneurysms: Observer Variability and Implications for Interpreting Case Series and Planning Randomized Trials

Angiographic results are commonly used as a surrogate marker of success of coiling of intracranial aneurysms. Inter- and intraobserver agreement in judging angiographic results remain poorly characterized. Our goal was to offer such an evaluation of a grading scale commonly used to evaluate results of endovascular treatment of aneurysms. A portfolio of 90 angiographic images from 45 patients selected from the core lab data base of a randomized trial was sent to 12 observers on 2 occasions more than 3 months apart. The variability of a 3-value grading scale used to score angiographic results and of a final judgment regarding the presence of a recurrence was studied using κ statistics. Ten participants responded once and 6 responded twice. Agreement was poor to moderate (κ = 0.28-0.5) for senior and junior observers judging angiographic results immediately or 12-18 months after treatment. Agreement reached a reassuring "substantial" (κ = 0.62) level, with a dichotomous presence-absence of a major recurrence, and intraobserver agreement was better in experienced core lab assessors. There is an important variability in the assessment of angiographic outcomes of endovascular treatments, rendering comparisons between publications risky, if not invalid. A simple dichotomous judgment can be used as a surrogate outcome in randomized trials designed to assess the value of new endovascular devices.