Reliability of Virtual Objective Structured Clinical Examination in Pharmacy Practice Experience Education Using Generalizability Theory

Abstract

Background and Objectives: The virtual Objective Structured Clinical Examination (OSCE) was established using the MyDispense database in various pharmacy schools worldwide. However, rigorous reliability estimation is lacking, which could affect student preparedness for the real-world pharmacy practice experience. Hence, this study aims to estimate the reliability of virtual OSCE using a robust reliability estimation method that has already been successfully established in medical schools. Materials and Methods: The reliability and components of variation were estimated using the generalized theory (G-theory) statistical framework. 4 facets were used to assess the components of variation: students, stations, items and years. The reliability of different stations (from 2-5) in virtual OSCE was estimated, with 182 students registered for pharmacy practice experience in various academic years between 2020 and 2023. The case scenarios were studied for content validity through expert academicians in pharmacy practice from universities other than the study site. Results: The G-coefficient with various virtual OSCE stations was close to the benchmark reliability of 0.70; however, it is still a low-stakes assessment. This study’s low-stakes assessment might be due to the estimation of component variance in more significant (22.60%) regarding the student’s performance varies mainly in different stations across the academic years. The low component variance (1.41%) indicated the negligible difference between the levels of students in various academic years. Conclusion: The number of stations in virtual OSCE strongly influences reliability. Pharmacy curriculum academicians can consider G-theory to estimate the reliability of virtual OSCE in preparing students for a better real-world pharmacy practice experience.