Document Type : Original Article
Author
Department of Emergency Medicine, Faculty of Medicine, King Abulaziz University, Jeddah, Saudi Arabia, Clinical Skills and Simulation Center, King Abulaziz University, Jeddah, Saudi Arabia
Abstract
Using simulation in high‑stakes assessments has been evolving as a method to improve the
assessment process. There is a concurrent need to address challenges and establish best practices
to ensure the best quality when implementing high‑stakes evaluations. The aim of this study is to
provide an insight for stakeholders about using multiple modalities of simulation in high‑stakes
evaluations by presenting challenges, best practices, and future directions described in the relevant
literature.A scoping review of original studies (from the year 1994–2021) including the use of common
modalities (standardized patients, high‑fidelity mannequins, part‑task trainers, virtual simulation, and
hybrid simulation) was conducted. The search covered the common databases: PubMed, Education
Resource Information Center, Cumulative Index to Nursing and Allied Health Literature, and the
Cochrane library.Initial screening in the databases resulted in 111,253 articles. After the application
of a refining search strategy, 47 articles were included for comprehensive evaluation. Most articles
were about credentialing/licensure purposes of assessment. Many articles were specialty‑specific,
especially focusing on anesthesia. Most challenges described were related to the validity of the
assessment that should be considered in the planning phase. Best practices were mostly related
to planning for measures to assure the validity of the assessment tools and process.The use of
multimodality simulation for high‑stakes assessment is growing despite challenges; this growth is
associated with the accumulation of experience shared in literature. This growth will help to improve
planning, practices, and goals achievement from such an application.
Keywords
1. Calhoun AW, Bhanji F, Sherbino J, Hatala R. Simulation for
high‑stakes assessment in pediatric emergency medicine. Clin
Pediatr Emerg Med 2016;17:212.
2. Rizzolo MA, Kardong‑Edgren S, Oermann MH, Jeffries PR.
The national league for nursing project to explore the use of
simulation for high‑stakes assessment: Process, outcomes, and
recommendations. Nurs Educ Perspect 2015;36:299‑303.
3. Kordi M, Fakari FR, Mazloum SR, Khadivzadeh T, Akhlaghi F,
Tara M. Comparison of the effect of web‑based, simulation‑based,
and conventional training on the accuracy of visual estimation
of postpartum hemorrhage volume on midwifery students:
A randomized clinical trial. J Educ Health Promot 2016;5:22.
4. Kordi M, Erfanian F, Fakari FR, Dastfan F, Nejad KS. The
comparison the effect of training by means of simulation and oral
method on midwives’ skill in management of shoulder dystocia.
J Educ Health Promot 2017;6:50.
5. O’Leary F. Simulation as a high stakes assessment tool in
emergency medicine. Emerg Med Australas 2015;27:173‑5.
6. Chambers KA, Boulet JR, Gary NE. The management of patient
encounter time in a high‑stakes assessment using standardized
patients. Med Educ 2000;34:813‑7.
7. Ziv A, Rubin O, Sidi A, Berkenstadt H. Credentialing and
certifying with simulation. Anesthesiol Clin 2007;25:261‑9.
8. Hatala R, Kassen BO, Nishikawa J, Cole G, Issenberg SB.
Incorporating simulation technology in a canadian internal
medicine specialty examination: A descriptive report. Acad Med
2005;80:554‑6.
9. Kardong‑Edgren S, Hanberg AD, Keenan C, Ackerman AA,
Chambers K. A discussion of high‑stakes testing: An extension of a
2009 INACSL conference roundtable. Clin Simul Nurs 2011;7:e19.
10. Leblanc VR. Review article: Simulation in anesthesia: State of the
science and looking forward. Can J Anaesth 2012;59:193‑202.
11. Sucharew H, Macaluso M. Progress notes: Methods for research
evidence synthesis: The scoping review approach. J Hosp Med
2019;14:416‑8.
12. Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A,
McEwen SA. A scoping review of scoping reviews: Advancing
the approach and enhancing the consistency. Res Synth Methods
2014;5:371‑85.
13. Furman GE, Smee S, Wilson C. Quality assurance best practices
for simulation‑based examinations. Simul Healthc 2010;5:226‑31.
14. Boulet JR. Summative assessment in medicine: The promise
of simulation for high‑stakes evaluation. Acad Emerg Med
2008;15:1017‑24.
15. Petrusa ER. Current challenges and future opportunities for
simulation in high‑stakes assessment. Simul Healthc 2009;4:3‑5.
16. Bensfield LA, Olech MJ, Horsley TL. Simulation for high‑stakes
evaluation in nursing. Nurse Educ 2012;37:71‑4.
17. Adamo G. Simulated and standardized patients in OSCEs:
Achievements and challenges 1992‑2003. Med Teach
2003;25:262‑70.
18. Weinger MB, Banerjee A, Burden AR, McIvor WR, Boulet J,
CooperJB, et al. Simulation‑based assessment of the management
of critical events by board‑certified Anesthesiologists.
Anesthesiology 2017;127:475‑89.
19. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R,
et al. Criteria for good assessment: Consensus statement and
recommendations from the Ottawa 2010 conference. Med Teach
2011;33:206‑14.
20. Cannon K, Hartsell Z, Ivanov I, Charles J, Joshi H, Blair J, et al.
Perceptions of internal medicine residency program candidates
on the use of simulation in the selection process. J Grad Med Educ
2014;6:338‑40.
21. Collins JP, Harden RM. AMEE medical education guide No. 13:
Real patients, simulated patients and simulators in clinical
examinations. Med Teach 1998;20:508.
22. BouletJR, Murray D. Review article: Assessment in anesthesiology
education. Can J Anaesth 2012;59:182‑92.
23. Tavakol M, Mohagheghi MA, Dennick R. Assessing the skills of
surgical residents using simulation. J Surg Educ 2008;65:77‑83.
24. Swing SR. Assessing the ACGME general competencies: General
considerations and assessment methods. Acad Emerg Med
2002;9:1278‑88.
25. Srinivasan M, Hwang JC, West D, Yellowlees PM. Assessment
of clinical skills using simulator technologies. Acad Psychiatry
2006;30:505‑15.
26. Boulet JR, De Champlain AF, McKinley DW. Setting defensible
performance standards on OSCEs and standardized patient
examinations. Med Teach 2003;25:245‑9.
27. Glavin RJ, Gaba DM. Challenges and opportunities in simulation
and assessment. Simul Healthc 2008;3:69‑71.
28. Sadeghi M, Taghva A, Mirsepassi G, Hassanzadeh M. How
do examiners and examinees think about role‑playing of
standardized patients in an OSCE setting? Acad Psychiatry
2007;31:358‑62.
29. Leo N, Foot C, Venkatesh B, Corke C, Manoj S, Lucey M, et al.
“High‑stakes assessment of the non‑technical skills of critical care
trainees using simulation: Feasibility, acceptability and reliability.
Crit Care Resusc 2014;16:6‑12.
30. Harvey P, Radomski N. Performance pressure: Simulated patients
and high‑stakes examinations in a regional clinical school. Aust J
Rural Health 2011;19:284‑9.
31. Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating
simulation‑based objective structured clinical examination into
the Israeli National Board Examination in Anesthesiology. Anesth
Analg 2006;102:853‑8.
32. Turner JL, Dankoski ME. Objective structured clinical exams:A critical review. Fam Med 2008;40:574‑8.
33. McNaughton N, Ravitz P, Wadell A, Hodges BD. Psychiatric
education and simulation: A review of the literature. Can J
Psychiatry 2008;53:85‑93.
34. Feldman M, Lazzara EH, Vanderbilt AA, DiazGranados D. Rater
training to support high‑stakes simulation‑based assessments.
J Contin Educ Health Prof 2012;32:279‑86.
35. Levine AI, Schwartz AD, Bryson EO, Demaria S Jr. Role of
simulation in U.S. physician licensure and certification. Mt Sinai
J Med 2012;79:140‑53.
36. Schuwirth LW, van der Vleuten CP. The use of clinical simulations
in assessment. Med Educ 2003;37 Suppl 1:65‑71.
37. Yudkowsky R. Should we use standardized patients instead of
real patients for high‑stakes exams in psychiatry? Acad Psychiatry
2003;26:187‑92.
38. Oermann MH, Kardong‑Edgren S, Rizzolo MA. Summative
simulated‑based assessment in nursing programs. J Nurs Educ
2016;55:323‑8.
39. Goldberg A, Samuelson S, Levine A, DeMaria S. High‑stakes
simulation‑based assessment for retraining and returning
physicians to practice. Int Anesthesiol Clin 2015;53:70‑80.
40. Holmboe E, Rizzolo MA, Sachdeva AK, Rosenberg M, Ziv A.
Simulation‑based assessment and the regulation of healthcare
professionals. Simul Healthc 2011;6 Suppl:S58‑62.
41. Banerjee A. Using simulation for primary certification. Int
Anesthesiol Clin 2015;53:42‑59.
42. DeMaria S Jr., Samuelson ST, Schwartz AD, Sim AJ,
Levine AI. Simulation‑based assessment and retraining for the
anesthesiologist seeking reentry to clinical practice: A case series.
Anesthesiology 2013;119:206‑17.
43. Orledge J, Phillips WJ, Murray WB, Lerant A. The use of
simulation in healthcare: From systems issues, to team building,
to task training, to education and high stakes examinations. Curr
Opin Crit Care 2012;18:326‑32.
44. McGrath JL, Taekman JM, Dev P, Danforth DR, Moheb D,
Kman N, et al. Using virtual reality simulation environments to
assess competence for emergency medicine learners. Acad Emerg
Med 2018;25:186‑95.
45. Warner DO, Isaak RS, Peterson‑Layhe C, Lien CA, Sun H,
Menzies AO, et al. Development of an objective structured clinical
examination as a component of assessment for initial board
certification in anesthesiology. Anesth Analg 2020;130:258‑64.
46. Bauer D, Lahner FM, Schmitz FM, Guttormsen S, Huwendiek S.
An overview of and approach to selecting appropriate patient
representations in teaching and summative assessment in medical
education. Swiss Med Wkly 2020;150:w20382.
47. Nadir NA, Hart D, Cassara M, Noelker J, Moadel T, Kulkarni M,
et al. Simulation‑based remediation in emergency medicine
residency training: A consensus study. West J Emerg Med
2019;20:145‑56.
48. Monti M, Klöckner‑Cronauer C, Hautz SC, Schnabel KP,
Breckwoldt J, Junod‑Perron N, et al. Improving the assessment
of communication competencies in a national licensing OSCE:
Lessons learned from an experts’ symposium. BMC Med Educ
2020;20:171.
49. Arrogante O, González‑Romero GM, López‑Torre EM,
Carrión‑García L, Polo A. Comparing formative and summative
simulation‑based assessment in undergraduate nursing students:
Nursing competency acquisition and clinical simulation
satisfaction. BMC Nurs 2021;20:92.
50. Isaak R, Stiegler M, Hobbs G, Martinelli SM, Zvara D, Arora H,
et al. Comparing real‑time versus delayed video assessments for
evaluating ACGME sub‑competency milestones in simulated
patient care environments. Cureus 2018;10:e2267.
51. Sabzwari SR, Afzal A, Nanji K. Mimicking rashes: Use of
moulage technique in undergraduate assessment at the Aga Khan
University, Karachi. Educ Health (Abingdon) 2017;30:60‑3.
high‑stakes assessment in pediatric emergency medicine. Clin
Pediatr Emerg Med 2016;17:212.
2. Rizzolo MA, Kardong‑Edgren S, Oermann MH, Jeffries PR.
The national league for nursing project to explore the use of
simulation for high‑stakes assessment: Process, outcomes, and
recommendations. Nurs Educ Perspect 2015;36:299‑303.
3. Kordi M, Fakari FR, Mazloum SR, Khadivzadeh T, Akhlaghi F,
Tara M. Comparison of the effect of web‑based, simulation‑based,
and conventional training on the accuracy of visual estimation
of postpartum hemorrhage volume on midwifery students:
A randomized clinical trial. J Educ Health Promot 2016;5:22.
4. Kordi M, Erfanian F, Fakari FR, Dastfan F, Nejad KS. The
comparison the effect of training by means of simulation and oral
method on midwives’ skill in management of shoulder dystocia.
J Educ Health Promot 2017;6:50.
5. O’Leary F. Simulation as a high stakes assessment tool in
emergency medicine. Emerg Med Australas 2015;27:173‑5.
6. Chambers KA, Boulet JR, Gary NE. The management of patient
encounter time in a high‑stakes assessment using standardized
patients. Med Educ 2000;34:813‑7.
7. Ziv A, Rubin O, Sidi A, Berkenstadt H. Credentialing and
certifying with simulation. Anesthesiol Clin 2007;25:261‑9.
8. Hatala R, Kassen BO, Nishikawa J, Cole G, Issenberg SB.
Incorporating simulation technology in a canadian internal
medicine specialty examination: A descriptive report. Acad Med
2005;80:554‑6.
9. Kardong‑Edgren S, Hanberg AD, Keenan C, Ackerman AA,
Chambers K. A discussion of high‑stakes testing: An extension of a
2009 INACSL conference roundtable. Clin Simul Nurs 2011;7:e19.
10. Leblanc VR. Review article: Simulation in anesthesia: State of the
science and looking forward. Can J Anaesth 2012;59:193‑202.
11. Sucharew H, Macaluso M. Progress notes: Methods for research
evidence synthesis: The scoping review approach. J Hosp Med
2019;14:416‑8.
12. Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A,
McEwen SA. A scoping review of scoping reviews: Advancing
the approach and enhancing the consistency. Res Synth Methods
2014;5:371‑85.
13. Furman GE, Smee S, Wilson C. Quality assurance best practices
for simulation‑based examinations. Simul Healthc 2010;5:226‑31.
14. Boulet JR. Summative assessment in medicine: The promise
of simulation for high‑stakes evaluation. Acad Emerg Med
2008;15:1017‑24.
15. Petrusa ER. Current challenges and future opportunities for
simulation in high‑stakes assessment. Simul Healthc 2009;4:3‑5.
16. Bensfield LA, Olech MJ, Horsley TL. Simulation for high‑stakes
evaluation in nursing. Nurse Educ 2012;37:71‑4.
17. Adamo G. Simulated and standardized patients in OSCEs:
Achievements and challenges 1992‑2003. Med Teach
2003;25:262‑70.
18. Weinger MB, Banerjee A, Burden AR, McIvor WR, Boulet J,
CooperJB, et al. Simulation‑based assessment of the management
of critical events by board‑certified Anesthesiologists.
Anesthesiology 2017;127:475‑89.
19. Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R,
et al. Criteria for good assessment: Consensus statement and
recommendations from the Ottawa 2010 conference. Med Teach
2011;33:206‑14.
20. Cannon K, Hartsell Z, Ivanov I, Charles J, Joshi H, Blair J, et al.
Perceptions of internal medicine residency program candidates
on the use of simulation in the selection process. J Grad Med Educ
2014;6:338‑40.
21. Collins JP, Harden RM. AMEE medical education guide No. 13:
Real patients, simulated patients and simulators in clinical
examinations. Med Teach 1998;20:508.
22. BouletJR, Murray D. Review article: Assessment in anesthesiology
education. Can J Anaesth 2012;59:182‑92.
23. Tavakol M, Mohagheghi MA, Dennick R. Assessing the skills of
surgical residents using simulation. J Surg Educ 2008;65:77‑83.
24. Swing SR. Assessing the ACGME general competencies: General
considerations and assessment methods. Acad Emerg Med
2002;9:1278‑88.
25. Srinivasan M, Hwang JC, West D, Yellowlees PM. Assessment
of clinical skills using simulator technologies. Acad Psychiatry
2006;30:505‑15.
26. Boulet JR, De Champlain AF, McKinley DW. Setting defensible
performance standards on OSCEs and standardized patient
examinations. Med Teach 2003;25:245‑9.
27. Glavin RJ, Gaba DM. Challenges and opportunities in simulation
and assessment. Simul Healthc 2008;3:69‑71.
28. Sadeghi M, Taghva A, Mirsepassi G, Hassanzadeh M. How
do examiners and examinees think about role‑playing of
standardized patients in an OSCE setting? Acad Psychiatry
2007;31:358‑62.
29. Leo N, Foot C, Venkatesh B, Corke C, Manoj S, Lucey M, et al.
“High‑stakes assessment of the non‑technical skills of critical care
trainees using simulation: Feasibility, acceptability and reliability.
Crit Care Resusc 2014;16:6‑12.
30. Harvey P, Radomski N. Performance pressure: Simulated patients
and high‑stakes examinations in a regional clinical school. Aust J
Rural Health 2011;19:284‑9.
31. Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating
simulation‑based objective structured clinical examination into
the Israeli National Board Examination in Anesthesiology. Anesth
Analg 2006;102:853‑8.
32. Turner JL, Dankoski ME. Objective structured clinical exams:A critical review. Fam Med 2008;40:574‑8.
33. McNaughton N, Ravitz P, Wadell A, Hodges BD. Psychiatric
education and simulation: A review of the literature. Can J
Psychiatry 2008;53:85‑93.
34. Feldman M, Lazzara EH, Vanderbilt AA, DiazGranados D. Rater
training to support high‑stakes simulation‑based assessments.
J Contin Educ Health Prof 2012;32:279‑86.
35. Levine AI, Schwartz AD, Bryson EO, Demaria S Jr. Role of
simulation in U.S. physician licensure and certification. Mt Sinai
J Med 2012;79:140‑53.
36. Schuwirth LW, van der Vleuten CP. The use of clinical simulations
in assessment. Med Educ 2003;37 Suppl 1:65‑71.
37. Yudkowsky R. Should we use standardized patients instead of
real patients for high‑stakes exams in psychiatry? Acad Psychiatry
2003;26:187‑92.
38. Oermann MH, Kardong‑Edgren S, Rizzolo MA. Summative
simulated‑based assessment in nursing programs. J Nurs Educ
2016;55:323‑8.
39. Goldberg A, Samuelson S, Levine A, DeMaria S. High‑stakes
simulation‑based assessment for retraining and returning
physicians to practice. Int Anesthesiol Clin 2015;53:70‑80.
40. Holmboe E, Rizzolo MA, Sachdeva AK, Rosenberg M, Ziv A.
Simulation‑based assessment and the regulation of healthcare
professionals. Simul Healthc 2011;6 Suppl:S58‑62.
41. Banerjee A. Using simulation for primary certification. Int
Anesthesiol Clin 2015;53:42‑59.
42. DeMaria S Jr., Samuelson ST, Schwartz AD, Sim AJ,
Levine AI. Simulation‑based assessment and retraining for the
anesthesiologist seeking reentry to clinical practice: A case series.
Anesthesiology 2013;119:206‑17.
43. Orledge J, Phillips WJ, Murray WB, Lerant A. The use of
simulation in healthcare: From systems issues, to team building,
to task training, to education and high stakes examinations. Curr
Opin Crit Care 2012;18:326‑32.
44. McGrath JL, Taekman JM, Dev P, Danforth DR, Moheb D,
Kman N, et al. Using virtual reality simulation environments to
assess competence for emergency medicine learners. Acad Emerg
Med 2018;25:186‑95.
45. Warner DO, Isaak RS, Peterson‑Layhe C, Lien CA, Sun H,
Menzies AO, et al. Development of an objective structured clinical
examination as a component of assessment for initial board
certification in anesthesiology. Anesth Analg 2020;130:258‑64.
46. Bauer D, Lahner FM, Schmitz FM, Guttormsen S, Huwendiek S.
An overview of and approach to selecting appropriate patient
representations in teaching and summative assessment in medical
education. Swiss Med Wkly 2020;150:w20382.
47. Nadir NA, Hart D, Cassara M, Noelker J, Moadel T, Kulkarni M,
et al. Simulation‑based remediation in emergency medicine
residency training: A consensus study. West J Emerg Med
2019;20:145‑56.
48. Monti M, Klöckner‑Cronauer C, Hautz SC, Schnabel KP,
Breckwoldt J, Junod‑Perron N, et al. Improving the assessment
of communication competencies in a national licensing OSCE:
Lessons learned from an experts’ symposium. BMC Med Educ
2020;20:171.
49. Arrogante O, González‑Romero GM, López‑Torre EM,
Carrión‑García L, Polo A. Comparing formative and summative
simulation‑based assessment in undergraduate nursing students:
Nursing competency acquisition and clinical simulation
satisfaction. BMC Nurs 2021;20:92.
50. Isaak R, Stiegler M, Hobbs G, Martinelli SM, Zvara D, Arora H,
et al. Comparing real‑time versus delayed video assessments for
evaluating ACGME sub‑competency milestones in simulated
patient care environments. Cureus 2018;10:e2267.
51. Sabzwari SR, Afzal A, Nanji K. Mimicking rashes: Use of
moulage technique in undergraduate assessment at the Aga Khan
University, Karachi. Educ Health (Abingdon) 2017;30:60‑3.
)