audio1038336609
===

[00:00:00]

Hello, behind the knife listeners. I'm am Anole, a general surgery PGY four at Duke Hospital. A few weeks ago, our team released the beta version of an oral board simulator, in which an AI agent takes users through a scenario in real time and provides extensive feedback about performance. First of all, thank you to all of our listeners.

We've had an immense amount of support and feedback, and because of that, we've made many changes and are so excited about the future. Today we wanted to take some time to talk about those changes in our plans. And to do so, I'm joined with Dr. Patrick Geoff, co-director of Behind the Knife and a Trauma and Critical Care Surgeon at Duke, as well as Dr.

Matthew Swenson, a general surgeon at Valley Health System and Air Force Veteran, who is the creator of the Oral Board simulator. After the discussion, we will play an example scenario so you can get a better sense of how it works. We'll also go through the feedback from that scenario.

Thanks Simon. We are super excited to talk about the simulator today. We've had a really positive response and it's literally getting better and better by the [00:01:00] day. This is the way of the future. , This is gonna be a very important part to how we prepare for the boards and so much more. So again, we're really excited to talk about today and share some of the improvements we have, uh, coming in the near future.

Well, thank you Ayman. Thank you Dr. George off. I have been so impressed with the behind the knife community. We've gotten so much usage outta the simulator and so much actionable feedback. Uh, your audience demands high quality, but they're also kind. So the feedback we have gotten has been both relevant and complimentary.

We, we have done our best to respond to the feedback quickly and have thus made many improvements, as you mentioned, sometimes daily. Uh, early feedback was for issues with the website itself, which were 100% related to the fact that this is beta and I am not a professional programmer. Uh, as we worked out those bugs.

However, the feedback is focused more on how the simulator itself performs and how we can improve the feedback and metrics that users see after they complete their simulation. [00:02:00]

And tha thank you so much for all the work that you've done on this Dr. Swensen. And, uh, to start with how the simulator performs.

I think one question we get quite frequently is about the variety of topics. So often users will ask us, is this really testable? So for, for example, if you're taking a breast scenario, do you really need to know about the reconstruction options after a mastectomy?

Absolutely. Honestly, this even surprises me sometimes as the scope of the test is quite vast.

I have had users ask questions like that, and my own first instinct is to agree with them and look into why that question was asked. But then when I look at the score curriculum documents on the a BS website, I almost always find that the topic is testable. Some examples when you look at that sheet, um, of topics that are testable include hand tendon injuries, vena cava, filter insertion on block, abdominal organ retrieval for transplant, cystostomy hysterectomy, and singal ectomy, and a parotidectomy.

Now this [00:03:00] doesn't necessarily mean that the a BS expects us to understand plastic reconstruction, the depth of a plastic surgeon, but they do expect us to be aware of the, the array of options that exist, especially since it may change our operative management and plan. Since we as general surgeons interact with many different specialties in part of our board exam is to make sure we understand how to function in multidisciplinary fashion, the a BS demands that we have a very wide breadth of knowledge and thus multiple testable topics on the test.

Yeah. Thank you. I, I think definitely when it impacts your surgical plan, it's very important that we know at least some of the basics now, when I first started taking the oral bo tests and I, I used to get a lot more odd responses from the simulator, but lately I've noticed that it's been happening much, much less.

So, Dr. Swensen, what's going on here? And can you tell us a little bit about what's happening behind the scenes? Is this something that I imagined or truly are we working on this simulator?

We are working on this. [00:04:00] Absolutely. And this is why the user feedback has been, uh, and the usage itself has been so crucial in these early days.

Our examiner, I like to think of it as an actor who's following a detailed script, and I mean detailed. So in the early days of the project, I was trying to tell the examiner how to act, and it was a few pages long. As I have worked with it, that has expanded to over 20 pages of instructions for how the examiner should act in different scenarios and situations as the instructions got longer.

However, the LLM has had difficulty following all those instructions, and so this would lead to odd answers or behaviors. Our solution to this problem is through a process called fine tuning. This is the equivalent of taking our actor and sending them to medical school and surgical residency, which means we need to give it fewer instructions and it's better at following the instructions.

We do give it. So every time we get feedback or every time a user takes a test, we're able to incorporate those into our fine tuning [00:05:00] methodology, which is what we are working on and continuing to do. Um, ultimately this will enable us to really mimic the flow of a true oral board scenario. And again, the input from all of the users is so important for that process.

The beautiful thing of course, is that as time goes on, this will only get better and better.

Yeah, definitely. As we get more data, it's, it's evident that it's just improving and improving. And speaking of the performance, let's talk about feedback. So it seems like a lot of changes has happened. My readiness went from pretty decent, you know, I was feeling a little maybe overconfident about these oral boards and all of a sudden , my, readiness is all the way back down to 1%.

So can you explain a little bit about what those changes have been, why we've made them, and. , And what you

see in the future for that. So again, thank you to the users for pointing out how important this marker of progress is to them. We have updated the score to be much more reflective of both the scope of the exam as well as how well you're doing on the [00:06:00] exam. So the reality is, for instance, for trauma, there are 47 different conditions and procedures that are testable on the exam.

We want the readiness score to not only show how well you're doing on trauma tests, but how many of those trauma tests you've actually taken. The total number of test conditions and procedures in the score curriculum is whopping 362. So, Iman, in your case, you may have taken a lot of exams and felt ready, but in reality, you've probably only taken 10 to 20 tests of the 362 available.

The readiness score now reflects this, uh, emphasizing that not only does is it important to do well on a test, but it's important to take multiple tests in order to truly be ready for test day.

Yeah, I think that now it's much more accurate to how ready I probably am for this exam. Sure. And, uh, another, another thing that users may notice is that we've done a complete makeover on the examiners.

What is that

all about? Yes, absolutely. This has actually been kind of fun. Uh, as we worked with the tech companies that host our [00:07:00] simulators, we were able to improve functionality and introduce some additional features. Part of the makeover was to improve the speed of transcription and our ability to analyze the test after simulation is over.

Uh, one of my new favorites though, is the multiple voices. So we now have 20 different voices that a simulator uses to administer your test. Some talk a bit fast, others a bit slow, some have an accent, et cetera. Uh, this really helps us to create a real world experience and prepare you for the real examiners you'll meet on test day.

Yeah, def definitely very important and I, I love it. Even just the voice itself gives it a different personality and it, it does add a lot to the experience. Now, one thing is users have commented about correct answers. Sometimes being flagged is incorrect, and sometimes this happens with more modern type answers.

Can you comment on

this? Of course we've noticed examples both in testing ourselves and through user feedback that, um, a user may give a correct [00:08:00] answer, but the LLM will flag that answer is incorrect. Now, most of the time what is happening is that users giving a good answer, but it's not a standard board answer.

A good example from just yesterday as a user, uh, reached out to us and stated that she would use FFP in a burn resuscitation instead of albumin, and the LLM flagged this as incorrect. Now the reality for us is that we are doing our very best to keep the LLM as up to date as possible, but there are so many nuances and different ways of managing patients that we can't simply code in every possibility in every scenario.

What's good about this, however, is that this is likely to happen in your actual exam. There's a very high chance that one of your examiners may not be familiar with a procedure or a management TE technique you discuss, especially if it's a newer technique or an institution that's doing something new, it's okay to give that answer, um, but just be ready to defend it.

And I think our LLM, um, is demonstrating that, uh, for you.

Yeah, remember this, [00:09:00] the test is all about being a safe surgeon and someone who's thoughtful and can think through the scenarios presented, uh, to them. And so, you know, in general, you're going to want to err on the side of. The standard of care or very accepted treatments.

However, uh, in the burn example is a perfect one. FFP is an accepted treatment. And so if you're not totally outside the box here, uh, talking about FFP and how you would use it, especially if that's your practice is gonna be an excellent way to approach the exam. I think the only caution would be how far outside of that practice you're getting, uh, you know how new.

Uh, very, very new perhaps is the, the treatment plan that you're talking about. You might wanna be careful, uh, in those scenarios. So we're also really excited, uh, to share that the simulator is gonna be undergoing a big makeover. We're gonna get it fully integrated into the behind the knife platform, including apps, uh, and relatively short order.

And this is gonna [00:10:00] improve the user experience significantly. It's gonna be beautiful, very smooth. And in addition to the platform update, we're also gonna be adding. Some new features. So, Dr. Swensen, what's, what's in store when we talk about the features?

Sure. Some features themselves that we're adding include some of the favorites of my own residents and medical students that I work with in our institution, uh, when I give it to them and let them test them out.

Uh, there's three specific features, uh, named Coach Wingman and op Guide. Each, they have their own unique purpose built for the surgical resident. Coach was actually part of my original idea for the project Coach is gonna analyze the transcript. After it's created by performance, then it will act like a real coach.

So it will identify weaknesses, make suggestions, walk you through scenarios, et cetera. Early on, I actually had users comment. It would be nice to know what the right answer was in a certain scenario. Well, this is exactly what coach is designed to do. Our examiners, when you take a test are purposely designed to not be [00:11:00] helpful.

They just administer a test. Coach is the exact opposite. You ask it a question, it will give you an answer. So it's a great resource for both senior and junior residents. On guide's a fun feature because its sole purpose is to help you practice board appropriate procedural steps. If you ask it how to perform a Whipple, it will provide a concise description of the procedure that was written by Dr.

George off and his other colleagues at Behind the Knife. Uh, wingman is an awesome feature for studying unfamiliar topics. It is set up just like the self-paced examiner. It presents you with a scenario. Patient STEM asks you what you would do if you don't know what to do. Then you click on the wingman button and Wingman will answer it for you in the first person as if it was you.

And then it will go to the next question and you can answer that question or you can tell Wingman to answer it for you. These answers are designed to both show an ideal answer and teach more about the topics. So it's a great way to study for that possible parotidectomy scenario, for example, that uh, you may feel [00:12:00] unprepared to handle.

Ugh, makes me shutter

thinking about having to describe a parotid make. I can hardly even say it. So these are Optum features. It's gonna make the user, uh, experience much better, the simulator much more useful, especially for residents at different stages of their training. And we talked about this before, but the boards are the most legitimate test you're gonna take in training, I think probably ever, because it focuses on real clinical challenges.

, In addition to the core knowledge, it also focuses on your ability to think and be safe. As a surgeon. So that's why reviewing board review material and using the simulator throughout training is a good idea. Not only will it make you better prepared for boards, but more importantly, you'll be better prepared for your next case.

Consult and clinic patient. , I think we should also take a moment to share plans to roll out the simulator in different specialties, including vascular. Colorectal surgical oncology, CT surgery, pediatric surgery, and even OB GYN with plans for more. And [00:13:00] finally, , we need, , your help. As we mentioned, all this feedback has been incredible.

The simulator is the worst it's ever gonna be right now. , The more feedback we get from you, the better. , If you go to the site or the app, you'll see in the bottom right hand corner, there's a, a button you can click on to provide feedback after you've done an exam.

, There's also a link,, in the show notes for this episode. If you've used it, please take just a moment to share your thoughts. We look at every single one, , of these responses and integrate them into what we do.

Well, as a final note before we wrap up, you mentioned a d different specialties. Uh, we have had several program directors and other surgical educators reach out to us to compliment us on this project and to ask how they can help, and we are so grateful to every one of them.

, For any of you who are listening, , right now our biggest ask is to share this resource with your residents and with other educators who work with residents In the future, as we expand other specialties, we will need [00:14:00] additional help in validating the content of our tests. Right now, what helps us the most is volume.

So please share this with everyone.

So one more time.

Thank you everybody for all of your help and all of your time to test this out and use this simulator again. We really appreciate it. To wrap up, we're gonna have one full scenario that we're gonna run through. This is taken by Dr. George off, and we can all see the vast improvements in the simulator and really hope that you do as well.

So thank you for listening. Dominate the day.

Are you ready to begin your test? I am

a 28-year-old male, is brought to the trauma bay after a high speed motor vehicle collision. He is intubated in the field. On arrival, his blood pressure is 90 60 and heart rate is one 30. He has decreased breath sounds on the right and extensive subcutaneous emphysema over the right chest and neck. A right-sided chest tube is placed with a large air leak and minimal blood [00:15:00] output.

What would you like to do now? Sure I would assemble my team, ensure we have the appropriate equipment and blood available. Make sure everyone Dons PPE and then perform a full primary survey with adjuncts including chest x-ray, pelvic x-ray, and a fast exam.

On primary survey you note the following airway is secured with an endotracheal tube breathing, decreased breath sounds on the right, persistent large air leak from the right chest tube and extensive subcutaneous emphysema Circulation. BP 90 60 HR one 30, no external bleeding. Disability GCs three T Exposure.

No other obvious injuries. Chest x-ray shows a persistent right pneumothorax despite the chest tube with significant subcutaneous emphysema. What is your differential diagnosis and what is your next step? I'm concerned about a major injury to the airway or the bronchus on the right side with persistent air leak.

For that reason, I'm [00:16:00] ready the operating room with plans to proceed with a thoracotomy. In the meantime, I'd like to ensure the patient has adequate IV access. Start whole blood resuscitation and send a full set of trauma labs. The patient is rapidly prepared for the operating room. He remains hypotensive and hypoxic despite the chest tube and ongoing resuscitation.

You proceed to the operating room for an emergent right thoracotomy. Upon entering the chest, you find a large amount of air escaping from the hilum and a disrupted right main stem bronchus. Describe your operative management. Sure. First, I would ensure that there's no major vascular injuries, especially at the hilum, and would assess the right lung to ensure that all portions of it are viable.

There are no major vascular injuries. The right main stem bronchus is completely transacted near the carina, the lung appears viable. How would you proceed [00:17:00] with repair of the bronchial injury? I'd like to perform an end-to-end anastomosis using running 3.0 Prolene suture. I would take care to orient the lung appropriately so it's not twisted.

Make sure that the anastomosis is tension free. I'll perform bronchoscopy upon completion. And perform an air leak test as well to ensure an airtight. The bronchial anastomosis is completed with a running three and prolene suture. The lung is expanded and a bronchoscopy confirms a patent airtight anastomosis.

There is no significant air leak. The patient is stabilized and transferred to the ICU On postoperative day two, the patient develops increasing subcutaneous emphysema and hypoxia. The chest tube is bubbling continuously. What is your differential diagnosis and how would you proceed? I'm concerned that there's an anastomotic complication, specifically a leak.

I would start by performing a bronchoscopy at the bedside in the ICU to evaluate the [00:18:00] anastomosis and the remainder of the airway.

Bedside bronchoscopy reveals a partial dehiscence of the right main stem bronchial anastomosis with ongoing air leak. The patient remains hypoxic and is developing worsening subcutaneous emphysema. What is your next step in management? Yeah, I take the patient back to the operating room for reopening of the thoracotomy, evaluation of the anastomosis, and uh, if I'm able to do revision of the anastomosis.

The patient is taken back to the operating room. On re exploration, you find a partial dehiscence of the right mainstem bronchial anastomosis. The surrounding tissue appears emus but viable. Describe your operative approach to revision of the anastomosis. I'd like to perform a complete revision. I would also evaluate or look for local tissue, the intercostal muscle flap to wrap.

The anastomosis and [00:19:00] buttress. It could also consider using a topical agents like Fibrin Glue. To help reinforce the new anastomotic suture line,

you perform a complete revision of the anastomosis and buttress it with a vascularized intercostal muscle flap. The repair is completed and the patient is returned to the ICU. Your oral board exam is now complete. Your transcript will now be sent to Behind the Knife For performance Analysis, please click the end test button to complete this session.

All right, so I've received my feedback for this exam and it says that I have passed and my overall score is 91. Competency was 91, clarity was 90, and I have the seven different American Board of Surgery essential attributes of a certifiable surgeon listed out. And with individual scores [00:20:00] for each, these includes essential organization, decision making, problem solving, error avoidance, recognition of limitation, surgical judgment, and adaptability.

I also have a full write out here for my score interpretation. It includes knowledge gaps, professionalism, study plan, and more detailed commentary. Pretty minimal knowledge gaps here. I did try my hardest at this one, and it gave me a few things like I did not mention intraoperative bronchoscopy to confirm repair after revision, for example, and they said that.

Use of topical agents like fibrin glue for airway anastomosis. Reinforcement is not standard and lacks strong evidence. Primary focus should be on a well vascularized tissue buttress. Otherwise, management was strong, is what it said. Professionalism, I was all good there for a study plan. It recommended, uh, reviewing the standard steps and adjuncts for bronchial anastomosis revision.

They recommended I listened to Behind the Knife Oral Board Scenario number 71, which is a scenario we wrote on esophageal and tracheal trauma with Empaa as the major complication. And finally, I have a full [00:21:00] commentary here, and this is, I think. Where the simulator really shines, it really breaks it down.

It talks about the scenario itself and some of the key management tips, things that you might want to consider when it comes to test taking skill sets and what the examiner might be looking for. Refers back to the A BS and score topics. Again, really nice, so. Happy with this one. Again, I, I gave it my all here and fortunately passed.

It's worth noting that there are a number of other considerations for this case. First, you could and, and might want to actually recommend consultation to your thoracic surgery colleagues in hopes that they're immediately available. Certainly the examiner. We'll say they're not available, but it's probably worth mentioning that also that are consulting or calling a senior partner for additional hands.

That's always wise in a complicated and rare case like this. There's also different ways to perform the bronchial anastomosis. I mentioned continuous suture. You can use multiple running suture techniques or even interrupted [00:22:00] suture, and it'd probably be wise to buttress of the repair at the initial operation.

That may actually be why it ended up having the complication that we had in this case.