Training and Assessment in Flexible Sigmoidoscopy: using a Novel Direct Observation of Procedural Skills (DOPS) Assessment Tool
Keith Siau1,2,3, James Crossley4, Paul Dunckley1,5, Gavin Johnson1,6, Adam Haycock7, John T. Anderson1,5, Marietta Iacucci3,8, Mark Feeney1,9, on behalf of the Joint Advisory Group on Gastrointestinal Endoscopy (JAG)
1) Joint Advisory Group on Gastrointestinal Endoscopy, Royal College of Physicians, London;
2) Endoscopy Department, Dudley Group Hospitals NHS Foundation Trust, Dudley;
3) Institute of Immunology and Immunotherapy, NIHR Birmingham Biomedical Research Centre, University of Birmingham, Birmingham;
4) Academic Unit of Medical Education, University of Sheffield, Sheffield;
5) Department of Gastroenterology, Gloucestershire Hospitals NHSFT, Gloucestershire;
6) Department of Gastroenterology, University College London Hospitals NHSFT, London;
7) Wolfson Unit, St Mark’s Hospital, London;
8) Institute of Translational Medicine, University Hospitals Birmingham NHS Foundation Trust, Birmingham;
9) Department of Gastroenterology, Torbay and South Devon NHS Foundation Trust, Torquay, United Kingdom
Background & Aims: Data supporting milestone development during flexible sigmoidoscopy (FS) training are lacking. We aimed to present validity evidence for our formative direct observation of procedural skills (DOPS) assessment in FS, and use DOPS to establish competency benchmarks and define learning curves for a national training cohort.
Methods: This prospective UK-wide (211 centres) study included all FS formative DOPS assessments submitted to the national e-portfolio. Reliability was estimated from generalisability theory analysis. Item and global DOPS scores were correlated with lifetime procedure count to study learning curves, with competency benchmarks defined using contrasting groups analysis. Multivariable binary logistic regression was performed
to identify independent predictors of DOPS competence.
Results: This analysis included 3,616 DOPS submitted for 468 trainees. From generalisability analysis, sources of overall competency score variance included: trainee ability (27%), assessor stringency (15%), assessor subjectivity attributable to the trainee (18%) and case-to-case variation (40%), which enabled the modelling of reliability estimates. The competency benchmark (mean DOPS score: 3.84) was achieved after 150-174 procedures. Across the cohort, competency development occurred in the order of: pre-procedural (50-74), non-technical (75-149), technical (125-174) and post-procedural (175-199) skills. Lifetime procedural count (p<0.001), case difficulty (p<0.001), and lifetime formative DOPS count (p=0.001) were independently associated with DOPS competence, but not trainee or assessor specialty.
Conclusion: Sigmoidoscopy DOPS can provide valid and reliable assessments of competency during training and can be used to chart competency development. Contrary to earlier studies, based on destination-orientated endpoints, overall competency in sigmoidoscopy was attained after 150 lifetime procedures.
Key words: competence − flexible sigmoidoscopy − formative assessment.
Abbreviations: DOPS: direct observation of procedural skills; DOPyS: the direct observation of polypectomy skills; EFA: exploratory factor analysis; ENTS: endoscopic non-technical skills; FS: flexible sigmoidoscopy; JAG: Joint Advisory Group on Gastrointestinal Endoscopy; JETS: JAG Endoscopy Training System; GP: general practitioner; NME: non-medical endoscopist.