Publications

BACKGROUND: Quantification of mastery is the first step in using objective metrics for teaching. We hypothesized that during orotracheal intubation, top tier performers have less idle time compared to lower tier performers.

METHODS: At the Anesthesiology 2018 Annual Meeting, 82 participants intubated a normal airway simulator and a burnt airway simulator. The movements of the participant’s laryngoscope were quantified using electromagnetic motion sensors. Top tier performers were defined as participants who intubated both simulators successfully in less than the median time for each simulator. Idle Time was defined as the duration of time when the laryngoscope was not moving.

RESULTS: Top performers showed less Idle Time when intubating the normal airway compared to lower tier performers (14.5 ± 9.8 seconds vs 34.0 ± 52.0 seconds, respectively P < .01). Likewise, top performers showed less Idle Time when intubating the burnt airway compared to lower tier performers (18.6 ± 15.2 seconds vs 63.4 ± 59.11 seconds; P < .01). Comparing performance on the burnt airway to the normal airway, there was a difference for lower tier performers (63.4 ± 59.1 seconds vs 34.0 ± 52.0 seconds; P < .01) but not for top tier performers (18.6 ± 15.2 seconds vs 14.5 ± 9.8 seconds; P = .07).

CONCLUSION: Similar to our previous findings with other procedures, Idle Time was shown to have known group validity evidence when comparing top performers with lower tier performers. Further, Idle Time was correlated with procedure difficulty in our prior work. We observed statistically significant differences in Idle Times for lower tier performers when comparing the normal airway to the burnt airway but not for top tier performers. Our findings support the continued exploration of Idle Time for development of objective assessment and curricula.

BACKGROUND: Time away from surgical practice can lead to skills decay. Research residents are thought to be prone to skills decay, given their limited experience and reduced exposure to clinical activities during their research training years. This study takes a cross-sectional approach to assess differences in residents’ skills at the beginning and end of their research years using virtual reality. We hypothesized that research residents will have measurable decay in psychomotor skills when evaluated using virtual reality.

METHODS: Surgical residents (n = 28) were divided into two groups; the first group was just beginning their research time (clinical residents: n = 19) and the second group (research residents: n = 9) had just finished at least 2 y of research. All participants were asked to perform a target-tracking task using a haptic device, and their performance was compared using Welch's t-test.

RESULTS: Research residents showed a higher level of “tracking error” (1.69 ± 0.44 cm versus 1.40 ± 0.19 cm; P = 0.04) and a similar level of “path length” (62.5 ± 10.5 cm versus 62.1 ± 5.2 cm; P = 0.92) when compared with clinical residents.

CONCLUSIONS: The increased “tracking error” among residents at the end of their research time suggests fine psychomotor skills decay in residents who spend time away from clinical duties during laboratory time. This decay demonstrates the need for research residents to regularly participate in clinical activities, simulation, or assessments to minimize and monitor skills decay while away from clinical practice. Additional longitudinal studies may help better map learning and decay curves for residents who spend time away from clinical practice.

BACKGROUND: Time away from surgical practice can lead to skills decay. Research residents are thought to be prone to skills decay, given their limited experience and reduced exposure to clinical activities during their research training years. This study takes a cross-sectional approach to assess differences in residents’ skills at the beginning and end of their research years using virtual reality. We hypothesized that research residents will have measurable decay in psychomotor skills when evaluated using virtual reality.

METHODS: Surgical residents (n = 28) were divided into two groups; the first group was just beginning their research time (clinical residents: n = 19) and the second group (research residents: n = 9) had just finished at least 2 y of research. All participants were asked to perform a target-tracking task using a haptic device, and their performance was compared using Welch's t-test.

RESULTS: Research residents showed a higher level of “tracking error” (1.69 ± 0.44 cm versus 1.40 ± 0.19 cm; P = 0.04) and a similar level of “path length” (62.5 ± 10.5 cm versus 62.1 ± 5.2 cm; P = 0.92) when compared with clinical residents.

CONCLUSIONS: The increased “tracking error” among residents at the end of their research time suggests fine psychomotor skills decay in residents who spend time away from clinical duties during laboratory time. This decay demonstrates the need for research residents to regularly participate in clinical activities, simulation, or assessments to minimize and monitor skills decay while away from clinical practice. Additional longitudinal studies may help better map learning and decay curves for residents who spend time away from clinical practice.

BACKGROUND: This study investigated the possibility of using virtual reality perceptual-motor tasks as a screening tool for laparoscopic ability. We hypothesized that perceptual-motor skills assessed using virtual reality will correlate with the quality of simulated laparoscopic ventral hernia repair.

MATERIALS AND METHODS: Surgical residents (N = 37), performed 2 virtual reality perceptual-motor tasks: (1) force matching and (2) target tracking. Participants also performed a laparoscopic ventral hernia repair on a simulator and final product quality score, and endoscopic visualization errors were calculated. Correlational analysis was performed to assess the relationship between performance on virtual reality tasks and laparoscopic ventral hernia repair.

RESULTS: Residents with poor performance on force matching in virtual reality-"peak deflection" (r = -0.34, P < .05) and "summation distance" (r = -0.36, P < .05)-had lower final product quality scores. Likewise, poor performance in virtual reality-based target tracking-"path length" (r = -0.49, P < .05) and "maximum distance" (r = -0.37, P < .05)-correlated with a lower final product quality score.

CONCLUSION: Our findings support the notion that virtual reality could be used as a screening tool for perceptual-motor skill. Trainees identified as having poor perceptual-motor skill can benefit from focused curricula, allowing them to hone personal areas of weakness and maximize technical skill.

OBJECTIVE: This paper provides a literature review and detailed discussion of surgical skill terminology. Culminating in a novel model that proposes a set of unique definitions, this review is designed to facilitate shared understanding to study and develop metrics quantifying surgical skill.

DESIGN: Objective surgical skill analysis depends on consistent definitions and shared understanding of terms like performance, expertise, experience, aptitude, ability, competency, and proficiency.

STRUCTURE: Each term is discussed in turn, drawing from existing literature and colloquial uses.

IMPLICATIONS: A new model of definitions is proposed to cement a common and consistent lexicon for future skills analysis, and to quantitatively describe a surgeon's performance throughout their career.

BACKGROUND: Interest is growing in simulation-based continuing medical education courses for practicing surgeons. However, little research has explored the instruction employed during these courses. This study examines instruction practices used during an annual simulation-based continuing medical education course.

METHODS: Audio-video data were collected from surgeon instructors (n= 12) who taught a simulated laparoscopic hernia repair continuing medical education course across 2 years. Surgeon learners (n= 58) were grouped by their self-reported laparoscopic and hernia repair experience. Instructors' transcribed dialogue was automatically coded for 5 types of responses to the following questions: anecdotes, confirming, correcting, guidance, and what not to do. Differences in these responses were measured against the progress of the simulations and across learners with different experience levels. Postcourse interviews with instructors were conducted for additional qualitative validation.

RESULTS: Performing t tests of instructor responses revealed that they were significantly more likely to answer in forms coded as anecdotes when responding to relative experts and in forms coded as what not to do when responding to novices. Linear regressions of each code against normalized progressions of each simulation revealed a significant relationship between progression through a simulation and frequency of the what not to do code for less-experienced learners. Postcourse interviews revealed that instructors continuously assess participants throughout a session and modify their teaching strategies.

CONCLUSION: Instructors significantly modified the focus of their teaching as a function both of their learners' self-reported experience levels, their assessment of learner needs, and learner progression through the training sessions

ABSTRACT: The current patient care workflow for physicians and healthcare providers includes multi-level communication structures (i.e., doctor-patient communication; doctor-nurse communication, etc.) and a wide variety of data management systems (1). When seeking to understand how to advocate for system changes that might improve workflow, a primary requirement is to first establish a definition and standard description of workflow components. It is also important to review the history behind current workflow structures and acknowledge current system successes as new implementations can disrupt what is already working. A functional and historical review can help to frame current needs and strategies for developing and implementing system changes to improve workflow and patient care.

BACKGROUND: This project involved the development and evaluation of a new visual bleeding feedback (VBF) system for tourniquet training. We hypothesized that dynamic VBF during junctional tourniquet training would be helpful and well received by trainees.

MATERIALS AND METHODS: We designed the VBF to simulate femoral bleeding. Medical students (n=15) and emergency medical service (EMS) members (n=4) were randomized in a single-blind, crossover study to the VBF or without feedback groups. Poststudy surveys assessing VBF usefulness and recommendations were conducted along with participants' reported confidence using a 7-point Likert scale. Data from the different groups were compared using Wilcoxon signed-rank and rank-sum tests.

RESULTS: Participants rated the helpfulness of the VBF highly (6.53/7.00) and indicated they were very likely to recommend the VBF simulator to others (6.80/7.00). Pre- and post-VBF confidence were not statistically different (P=0.59). Likewise, tourniquet application times for VBF and without feedback before crossover were not statistically different (P=0.63). Although participant confidence did not change significantly from beginning to end of the study (P=0.46), application time was significantly reduced (P=0.001).

CONCLUSIONS: New tourniquet learners liked our VBF prototype and found it useful. Although confidence did not change over the course of the study for any group, application times improved. Future studies using outcomes of this study will allow us to continue VBF development as well as incorporate other quantitative measures of task performance to elucidate VBF's true benefit and help trainees achieve mastery in junctional tourniquet skills.

INTRODUCTION: The objective of this study was to determine whether decision-based procedural mapping demonstrates differences in attendings versus residents.

METHODS: Attendings and residents were interviewed about operative decision-making in laparoscopic cholecystectomy (LC) using a cognitive task analysis framework. Interviews were converted into procedural maps. Operative steps, patient factors, and surgeon factors noted by attendings and residents were compared. Two scoring methods were used to compare map structures of attendings versus residents.

RESULTS: Six attendings and six residents were interviewed. There were no significant differences in the number of patient or surgeon factors identified. Attendings had significantly more operative steps (29.67 ± 1.9 vs. 23.3 ± 1.9, p = 0.04) and crosslinks (3.2 ± 0.5 vs. 1 ± 0.4, p = 0.005) in their maps and a higher total score (90.2 ± 8.4 vs. 63.2 ± 3.8, p = 0.015) than residents.

CONCLUSION: LC procedural map scoring for attendings and residents demonstrated significant differences in structural complexity and may provide a useful framework for assessing decision making.

BACKGROUND: The aim was to validate the potential use of a single, early procedure, operative task as a predictive metric for overall performance. The authors hypothesized that a shortcut psychomotor assessment would be as informative as a total procedural psychomotor assessment when evaluating laparoscopic ventral hernia repair performance on a simulator.

METHODS: Using electromagnetic sensors, hand motion data were collected from 38 surgery residents during a simulated laparoscopic ventral hernia repair procedure. Three time-based phases of the procedure were defined: Early Phase (start time through completion of first anchoring suture), Mid Phase (start time through completion of second anchoring suture), and Total Operative Time. Correlations were calculated comparing time and motion metrics for each phase with the final laparoscopic ventral hernia repair score.

RESULTS: Analyses revealed that execution time and motion, for the first anchoring suture, predicted procedural outcomes. Greater execution times and path lengths correlated to lesser laparoscopic ventral hernia repair scores (r = -0.56, P = .0008 and r = -0.51, P = .0025, respectively). Greater bimanual dexterity measures correlated to Greater LVH repair scores (r = + 0.47, P = .0058).

CONCLUSIONS: This study provides validity evidence for use of a single, early operative task as a shortcut assessment to predict resident performance during a simulated laparoscopic ventral hernia repair procedure. With the continued development and decreasing costs of motion technology, faculty should be well-versed in the use of motion metrics for performance measurements. The results strongly support the use of dexterity and economy of motion (path length + execution time) metrics as early predictors of operative performance.

BACKGROUND: Given their variegated backgrounds, surgeons taking continuing medical education (CME) courses possess different learning needs. This study examines the relationship between surgeons' levels of experience and the questions they asked in a simulation-based CME course.

METHODS: We analyzed transcribed audio-video data collected from surgeons participating in a simulated laparoscopic hernia repair CME course and identified four types of questions learners posed to their instructors. Linear regressions compared how often these questions were asked versus self-reported operative experience.

RESULTS: Both Requesting Guidance and Requesting Confirmation were inversely proportional to experience, whereas Asking About a Specific Case was directly proportional to experience. Requesting Instructor Preference exhibited no significant correlation with experience.

CONCLUSION: Practicing surgeons with relatively less experience tend to ask for confirmation and guidance, whereas those with greater experience tend to focus on specific hypothetical scenarios. This data can be used to tailor instruction based on learners' self-reported experience level.

BACKGROUND: Residents engaging in dedicated research experiences may return to clinical training with less surgical skill. The study aims were 1) to evaluate faculty perceptions of residents skills decay during dedicated research fellowships, and 2) to compare faculty and resident perceptions of residents skills decay.

METHODS: Faculty and residents were surveyed on resident research practices and perceptions of resident skills decay.

RESULTS: Faculty thought residents returning from research demonstrate less technical skill (Median = 4; 5-point Likert scale,1 = Strongly disagree, 5 = Strongly agree), demonstrate less confidence (Median = 4), and require more instruction (Median = 4). Both faculty and residents perceived the largest skill reduction in complex procedures, technical surgical skills, and knowledge of procedure steps (p < 0.05).

CONCLUSION: While dedicated research experiences provide valuable academic experience, there is a cost to clinical skills retention and confidence specifically in the areas of complex operative procedures and technical surgical skills.

INTRODUCTION: Laparoscopic inguinal hernia repair has been shown to have significant benefits when compared to open inguinal hernia repair, yet remains underutilized in the United States. The traditional model of short, hands-on, cognitive courses to enhance the adoption of new techniques fails to lead to significant levels of practice implementation for most surgeons. We hypothesized that a comprehensive program would facilitate the adoption of laparoscopic inguinal hernia repair (TEP) for practicing surgeons.

METHODS: A team of experts in simulation, coaching, and hernia care created a comprehensive training program to facilitate the adoption of TEP. Three surgeons who routinely performed open inguinal hernia repair with greater than 50 cases annually were recruited to participate in the program. Coaches were selected based on their procedural expertise and underwent formal training in surgical coaching. Participants were required to evaluate all aspects of the educational program and were surveyed out to one year following completion of the program to assess for sustained adoption of TEP.

RESULTS: All three participants successfully completed the first three steps of the seven-step program. Two participants completed the full course, while the third dropped out of the program due to time constraints and low case volume. Participant surgeons rated Orientation (4.7/5), GlovesOn training (5/5), and Preceptored Cases (5/5) as highly important training activities that contributed to advancing their knowledge and technical performance of the TEP procedure. At one year, both participants were performing TEPs for “most of their cases” and were confident in their ability to perform the procedure. The total cost of the program including all travel, personal coaching, and simulation was $8638.60 per participant.

DISCUSSION: Our comprehensive educational program led to full and sustained adoption of TEP for those who completed the course. Time constraints, travel costs, and case volume are major considerations for successful completion; however, the program is feasible, acceptable, and affordable.

BACKGROUND: The aim of this study was to assess performance measurement validity of our newly developed robotic surgery task trainer. We hypothesized that residents would exhibit wide variations in their intercohort performance as well as a measurable difference compared to surgeons in fellowship training.

MATERIALS AND METHODS: Our laboratory synthesized a model of a pelvic tumor that simulates unexpected bleeding. Surgical residents and fellows of varying specialties completed a demographic survey and were allowed 20 minutes to resect the tumor using the da Vinci robot and achieve hemostasis. At a standardized event in the simulation, venous bleeding began, and participants attempted hemostasis using suture ligation. A motion tracking system, using electromagnetic sensors, recorded participants' hand movements. A postparticipation Likert scale survey evaluated participants' assessment of the model's realism and usefulness.

RESULTS: Three of the seven residents (postgraduate year 2-5), and the fellow successfully resected the tumor in the allotted time. Residents showed high variability in performance and blood loss (125-700 mL) both within their cohort and compared to the fellow (150 mL blood). All participants rated the model as having high realism and utility for trainees.

CONCLUSIONS: The results support that our bleeding pelvic tumor simulator has the ability to discriminate resident performance in robotic surgery. The combination of motion, decision-making, and blood loss metrics offers a multilevel performance assessment, analyzing both technical and decision-making abilities.

Abstract: Palpation plays a critical role in medical physical exams. Despite the wide range of exams, there are several reproducible and subconscious sets of maneuvers that are common to examination by palpation. Previous studies by our group demonstrated the use of manikins and pressure sensors for measuring and quantifying how physicians palpate during different physical exams. In this study we develop mathematical models that describe some of these common maneuvers. Dynamic pressure data was measured using a simplified testbed and different autoregressive models were used to describe the motion of interest. The frequency, direction and type of motion used were identified from the models. We believe these models can a provide better understanding of how humans explore objects in general and more specifically give insights to understand medical physical exams.

Abstract: Operating room teams consist of team members with diverse training backgrounds. In addition to differences in training, each team member has unique and complex decision making paths. As such, team members may function in the same environment largely unaware of their team members’ perspectives. The goal of our work was to use a theory-based approach to better understand the complexity of knowledge-based intra-operative decision making. Cognitive task analysis methods were used to extract the knowledge, thought processes, goal structures and critical decisions that provide the foundation for surgical task performance. A triangulated and iterative approach is presented.

Objective: Define, extract and evaluate potential performance indicators from computer-generated data collected during simulated clinical female pelvic examinations.

Design: Qualitative and quantitative study analyzing computer generated simulator data and written clinical assessments collected from medical students who performed physical examinations on three clinically different pelvic simulators.

Setting: Introduction to patient care course at a major United States medical school.

Participants: Seventy-three pre-clinical medical students performed 219 simulated pelvic examinations and generated 219 written clinical assessments.

Measurements: Cronbach's alpha for the newly defined performance indicators, Pearson's correlation of performance indicators with scored written clinical assessments of simulator findings.

Results: Four novel performance indicators were defined: time to perform a complete examination, number of critical areas touched during the exam, the maximum pressure used, and the frequency at which these areas were touched. The reliability coefficients (alpha) were time = 0.7240, critical areas = 0.6329, maximum pressure = 0.7701, and frequency = 0.5011. Of the four indicators, three correlated positively and significantly with the written clinical assessment scores: critical areas, p < 0.01; frequency, p < 0.05; and maximum pressure, p < 0.05.

Conclusion: This study demonstrates a novel method of analyzing raw numerical data generated from a newly developed patient simulator; deriving performance indicators from computer generated simulator data; and assessing validity of those indicators by comparing them with written assessment scores. Results show the new assessment measures provide an objective, reliable, and valid method of assessing students' physical examination techniques on the pelvic exam simulator.

Publisher: IOS PRESS, NIEUWE HEMWEG 6B, 1013 BG AMSTERDAM, NETHERLANDS

Research Areas: Computer Science; Engineering

Web of Science Categories: Computer Science, Artificial Intelligence; Computer Science, Cybernetics; Computer Science, Interdisciplinary Applications; Engineering, Biomedical