Effects of differential time and difficulty weighting on the improvement of OSCE quality metrics: Vpliv časovnih in težavnostnih diferencialnih uteži na izboljšanje kazalcev kakovosti OSKI - Mihevc, Matic; Masnik, Klara; Petreski, Tadej; Pulko, Nejc; Bevc, Sebastjan (Acta Medico-Biotechnica, 2022)
Purpose: Objective structured clinical examination (OSCE) has become a leading method for assessing clinical skills. A common problem observed in OSCE is the discrepancy between checklist scores and global scale scores. To reduce this discrepancy, checklists have been unsuccessfully weighted. However, differential weighting based on a combination of time and difficulty weighting had not been investigated. The goals of our study were to: (a) develop new formulas for time and difficulty weights, (b) design an experimental checklist considering the calculated weights, and (c) evaluate the effectiveness of the method in improving OSCE quality metrics.Methods: First, we created formulas for time and difficulty weights for each checklist item. Second, we formulated an experimental checklist consideringthe calculated weights of each item. Simultaneously, a control checklist was independently weighted by OSCE board members. Third, we compared the experimental and control checklists during the OSCE, wherein students (n=55) were graded by ten assessors using both checklists and the holistic global rating form. Finally, we performed statistical analysis with paired samples t-test, simple linear regression, and scale reliability statistics.Results: A higher correlation coefficient between the checklist and the holistic scoring form (r=0.622, p
Keywords: OSCE, clinical skills, peer assessment, quality metrics, weighting, outliers