Psychometric evaluation of a direct observation of procedural skills assessment tool for ultrasound‐guided regional anaesthesia

Summary Assessment tools must be investigated for reliability, validity and feasibility before being implemented. In 2013, the Australian and New Zealand College of Anaesthetists introduced workplace‐based assessments, including a direct observation of a procedural skills assessment tool. The objective of this study was to evaluate the psychometric properties of this assessment tool for ultrasound‐guided regional anaesthesia. Six experts assessed 30 video‐recorded trainee performances of ultrasound‐guided regional anaesthesia. Inter‐rater reliability, assessed using absolute agreement intraclass correlation coefficients, varied from 0.10 to 0.49 for the nine individual nine‐point scale items, and was 0.25 for a ‘total score’ of all items. Internal consistency was measured by correlation between ‘total score’ and ‘overall performance’ scale item (r = 0.68, p