That was probably the quietest I've seen a room get after an announcement that I've ever experienced. Good job, everybody. It's good to be with you all here today. I hope that all of you did a much better job than I did in packing for this event and brought coats with you with this lovely weather we're now experiencing. You'd think living in Michigan I'd know better, but apparently I didn't. Hopefully the rest of you have done better than that. My name is David Westman, and I'm the Executive Director of the American Society of Colon and Rectal Surgeons, otherwise known as ASCARS, a pretty apropos acronym, I would say. I'm pleased to have with me my current president, Dr. Sonia Ramamurthy, as my co-moderator. Today we are going to be talking about the topic of society-sponsored education, including some new directions that we've been experiencing and that we hope that you will come away with just some good takeaways from learning from the experiences of three organizations. We have a lot of ground to cover today, so I am going to right away go and introduce our first speaker of the day, who is my colleague at ASCARS, Erica Flynn. Erica is our Associate Executive Director for Membership, Education, and Meetings, and she's going to describe some of the things that we're doing in the last couple of years in our journey towards educational excellence. And then Dr. Ramamurthy will introduce our other esteemed colleagues from ACS and ACP, and will also then moderate our question and answers. So, Erica, take it away. Thank you, David. So a little bit about ASCARS. You may have seen or heard our acronym thrown around this meeting a good bit over the last few days. We are a specialty society for colon and rectal surgeons. We have about 4,000 total members, 2,500 of which are physician members. So that's just a little bit of a, you know, setting the stage as we get into the education and content that we're going to discuss. So why are we talking about the need for society education? There's been a lot of changes that's been occurring since post-pandemic. The evolving technological needs of the 21st century, AI advancements, they're reshaping the industry and employment landscape altogether. In addition to that, there's the generational shift, how the younger members are learning, how it's, you know, completely different with the Gen Z than millennials, than baby boomers, than Gen X. In addition to that, you know, just this idea of lifelong learning. People are changing careers now at a more rapid rate, and there's this need for skill upgrades, especially, you know, with new tools in technology. You might be doing the same job for 20 years. Now there's all these new things you have to learn. So for you to stay current in your current job, you need to learn new things. So that is why society education is so important, because societies can give their members the education that meets all of these needs. We can ensure that all of our education is equitable and accessible for our members, so they can stay competitive in these rapidly changing landscapes. We can ensure that there's digital literacy, basically, you know, delineate from the misinformation. Our members know that what they're getting from us is real. That's the content that we need to be providing them, and that's why they come to the societies for their education. Additionally, we need to and we can engage with our younger society members by giving them a voice in the society, so they can continue to learn throughout their entire career continuum and network with their peers. We've been hearing this concept throughout the last few days or yesterday and today related to year-long engagement, and that's going to be something I'm going to touch on a little bit today. So what did we do? You know, I came to Askers in 2021, and we did some big things starting in 2022. We had our inaugural Surgical Leadership Institute. It was a one-and-a-half-day in-person event, about 100 participants, three virtual sessions, non-CME. There was a wealth of topics, some short presentation. It focused on the ten competencies of leadership. We had industry partner engagement during this event, but the feedback we received after, you know, members wanted more interaction. They wanted a deeper dive into targeted topics. They wanted to share what they were learning too and not just be, you know, told all of these things. Same time, we also launched our educational portal, Askers U. It was free to members. We bundled it with our membership. It's an education portal. All of our content is in one place. It's aligned with our needs gap analysis blueprint. We have six pillars of colorectal surgery that are clinical and one non-clinical pillar, and there's percentages that are matched to each of those pillars, which demonstrates, you know, the content that we need to be producing across the specialty. In our learning portal, it's our textbook of colon and rectal surgery, self-assessment, practice questions for trainees, access to CME, practice guidelines, video education, and our annual meeting content in addition to a plethora of other things. In the first year, we had 15,000 views and 1,800 unique users. The average engagement time was about 18 minutes per segment. So how did we grow in the last two years? We had another surgical leadership institute. This time, we kept it with CME. So, you know, that was something we wanted to leverage to our society members. We did four topic deep dives, including substantial interactive activities about knowing yourself, knowing your worth, negotiations, conflict resolution. These were the four topics that really resonated with our members on these leadership skills. We increased corporate partner engagement with the program and actually had our corporate partners a part of the program. They were sitting at the tables with the colorectal surgeons and participating, so leveling the playing field across everyone that was there. The feedback we got from this, members all shared positive feedback. Primary takeaways is to have maybe more targeted audiences. We did not target the audience. It was open to any of our membership. So, you know, the feedback we got, different career levels might need different things. So how are we going to go about that next? With AskResU, we continually curate our content, update it normally. There's accredited and non-accredited content on our learning portal. We incorporated a corporate partner education. There's a place on our portal for corporate partners to have their content for our viewers to see. We make all of our digital publishing updates live, so we could, you know, update the textbook or any of our clinical practice guidelines much more easily. We have 94,000 active users in the U.S., 110 worldwide, 3,500 unique users, and the average engagement time is about 13 minutes. So evolution, we're continuing to change faster than we know what to do with. What do we do? How do we get there? So we think that the key to the next steps are basically engaging the younger generations of our surgeons with a longitudinal and multimodal curriculum. Those are the two key words. People want to be engaged over time. That year-round engagement, how are we doing that? And multimodal, we heard it earlier in a session about, you know, breaking through the screen. How do we get people to stay focused on our content after the annual meeting or after the Surgical Leadership Institute? What can we do to get them to keep engaging with the society? So we need to, you know, connect them at a young age, give them their network of peers, have in-person learning and social media outlets, give them access to mentorship and leadership when they want it. You know, if they want access to someone to talk to, a leader within the society, we can provide that for them, introduce them to the concept of collaborating with our corporate partners. And, you know, more importantly, especially with the Gen Z, you know, give them an opportunity to get involved, provide and create what they feel strongly about. And then as a society, continue to develop this content so they can adapt and modernize their journey. So our future, you know, we decided instead of having these one-off Surgical Leadership Institutes, to try to do this in more of a longitudinal fashion where we have varying touch points along the colorectal continuum. So at a colorectal fellow phase, maybe they meet twice, maybe there's virtual education, maybe there's social media follow-up, engaging with them at varying levels. Then, you know, when there are new members, they're still continue to meet, you know, give them access to our annual meetings, show them the importance of what the society can provide. As they grow in their career, you know, mentorship, leadership, maybe get them engaged with gamification, you know, increase all the things that they are interested in included in our curricula. You know, as they grow in their career, they may or may not need as much engagement opportunities because, you know, they already are on their career trajectory path. But there are some that pivot careers. There are some that want to go into leadership, program directors, you know, chief medical officers, whatever path they take. So provide them that professional development as well. And then there's our lifelong learners, you know, how do you prepare for retirement, what's the succession plan look like. That's something that we don't really put a lot of education out there on, but there is a need for that. So kind of doing a gap analysis of those learners to see where to go with our content. And mostly with our educational platform, you know, adapt to incorporate AI, what's next along that pathway. So just in closing before I turn it over, you know, I put a little pathway across the bottom in that circular Olympic graph, but I just wanted to share that society education, it's meant to build resilience and empower our members at all levels to take part in the solutions to ensure that our future generations are equipped to tackle the challenges that they have and to foster the innovation for science, sustainability, and technology. So with that, thank you. Isn't she great? You can't have her. She's ours. Okay. Well, if that was a window into what we're doing as a specialty organization, a small to midsize organization, we're going to blow it up for you and look at what are the big houses of surgery and the big houses of medicine doing to provide society-based education. So it's my honor to introduce, if you think about education as a surgeon, the name that comes to mind is Dr. Sachdeva. So it's my honor to introduce Dr. Ajit Sachdeva, who's the founding director of the American College of Surgeons Division of Education and the founding member of the Academy of Master Surgeons. Under his leadership, several education programs have been launched, and these are highly sought after throughout all surgery, ortho, ENT, colorectal, across this country. And most of them have their origins with Dr. Sachdeva. So he is an incredible asset to the House of Surgery, and we look forward to your talk. Thank you. Thank you very much, Dr. Ramamurthy. It really is an honor and a privilege to be here in front of you to share with you some of the excitement that we feel every day in regard to the education activities at the American College of Surgeons. I must start by mentioning that our goal is much like what you heard from the colorectal surgery standpoint, is to look at the whole continuum. And I'm going to touch upon those and share with you how we focus on the specific points where we feel that the return on investment, the value that education generates, quite often it does not generate money, but it generates value, which is critical to us, our members, the patients we serve, and society at large. How we can work with this and what a unique role, especially societies, play in this space. I will start by mentioning that I have no conflicts with any commercial entity to disclose. And I thought it might be worthwhile to start by framing some of the issues that all of us know about and many of us are trying to get our hands around for the benefit of our members and the profession and patients as a whole. Obviously, the monumental changes in healthcare and rapidly evolving healthcare systems are all around us. We are part of them and we have to keep up with those changes. In addition to that, there's a plethora of external regulations and mandates that keep coming at us from all sides and frustrate many of our members. And one solution to that frustration is certainly education where we can get people excited about the joy of learning, about learning something new each day, and then providing the best care to patients. Science and technology continue to develop. We've just heard a little bit about AI. I'll mention that again in a few moments. But simulation is another huge area where I know the colorectal surgeons and I also know the American College of Physicians has been very involved with in the area of the whole spectrum of simulation. In addition to that, we as professional organizations need to look at the nexus between cutting-edge education and the best patient care. And that is, I believe, our sweet spot because we've got quality programs, we have databases, we have educational programs, and if we marry the two in a nice, cohesive, and effective way, only good things can happen. And clearly, there are new imperatives that are upon us. Society keeps changing. Generations keep looking for different things. And of course, technology, as we all know, is growing in leaps and bounds and we are still recovering from the impact of the pandemic, which has impacted many of our learners in terms of wellness and some of the other issues that we need to tackle as well. So let me start with this framework, which I alluded to earlier. Oftentimes, the medical school years, the residency and fellowship training, and lifelong practice are looked upon as separate boxes or separate areas of progression. And they have been siloed because nationally, our accreditation systems in all these three domains are different. Many times, our faculty within our spheres of influence within our institutions are different, our staff are different, and indeed, the curricula are different, although there is overlap certainly along this continuum. We have looked upon this as one seamless continuum, and that is the beauty of a professional society where we can really occupy this whole space, get our hands around it, and make effective change. The vulnerabilities you see at the bottom are the transitions across this continuum. Clearly, we could spend many lifetimes trying to create educational programs across this continuum, but where is the maximum value that we can generate where the need is the greatest in terms of professional development and the risks are the greatest and the rewards are going to be wonderful as well. So those are from medical school to residency, certainly during the five, seven, eight years of training, and then the huge span that all of us in the professional society domain are serving our members in, and that is 20, 30, 40, sometimes more than 40 years of practice, but everything changes. Our practices change, technology changes, new procedures come up, and it is imperative that we provide the leadership to the members and the profession to address these transitions, and that would not only ensure a smooth transition in the careers of people, but also the best patient care. This is another conceptual framework I thought I would share with you. We oftentimes in education and surgery education is no different talk about competence, but none of us, if we needed an operation or our family members needed an operation, would look around for a competent surgeon. We wouldn't. We would look for the expert, if not the master. So we need to look at these levels, and this is a modification of the Dreyfus and Dreyfus model that has been described in the medical education literature. Proficiency is where we want our trainees to go forward with, and that means we are creating and training safe surgeons with good judgment, but they're not experts by any means. Expertise comes from deliberate practice, being in different situations. We've written articles ourselves on the expertise space, and, of course, we have relied quite a bit on the work of Dr. Anders Erikson who talked about repeated practice, deliberate practice, and looked at ballerinas, musicians, and all, and translated their experiences into the medical and health professions world, and clearly mastery is where we all aspire to be if we can continue to grow all through our lifetime, and it's our job as professional societies to support that aspiration. Now, very quickly, I'm not going to go through all this with you in detail, but just to give you a glimpse, we have tried to address the transitions in a very positive planned way. These are the programs that we have, the big programs that we have for practicing surgeons. We have our annual clinical congress, which you see in the top left. We have CSAP, our self-assessment program that is taken to the next level through each edition. It's very different now than what it was when I started at the college, and then we have the surgeons as educators course, fully oversubscribed year after year. The surgeons as leaders course also exists, which is in the middle just below the clinical congress. We have a simulation summit. We have programs on teamwork. You see that space where you have the OR team working together, and that talks about how we work together in teams. We have simulation courses. On the left bottom is a book on surgical education that we published in 2021 and then we have a review courses like comprehensive general surgery review course that you see in the top row to the right. For surgery trainees and medical students we have a whole continuum from the time when they enter residency which are the first few curricula you see on the left side of the screen and then going on to when they enter residency there are simulation based curricula and I'd certainly be happy to chat with you a little bit about them if there is interest during the discussion or beyond. We have programs for patients because our patients in the surgical world need a different kind of training and their caregivers need a different kind of training if they're going to take care of all the tubes drains and things we send them home with and send them home quickly after a surgical procedure so we have a whole bunch of patient education programs that have been translated into many different languages. We also have a simulation center accreditation program that was launched in 2005. We launched it to standardize simulation based training across the country. Right now we have 95 of these centers 83 are in the U.S. rest of them abroad and we have 20 ACS accredited simulation fellowship programs grafted onto these comprehensive centers. We also have an Academy of Master Surgeon Educators that Dr. Ramamurti mentioned and the mission is to play a key leadership role in advancing the science and practice of education across all surgical specialties the House of Surgery promoting the highest achievements in the lifetimes of surgeons. We have three categories of membership right now we have a membership of 400 it's a very prestigious group and the group gets together and we are looking now at mentoring the upcoming and the people who are at in mid careers as surgeon educators. We also have a focus on credentialing and privileging which is huge especially with the growth of robotic surgery and where technology fits in. AI has its opportunities but it also has its challenges. We all have heard about the problems with chat GPT and the hallucinations which do not lead to great information exchange. It's our job as a profession to figure out what is a trusted content in this age and perhaps create a universe which is controlled so that the generative AI is not open-ended it doesn't get things from all over the world some good some not so good we can control some of that as professional societies. So I would like to conclude by mentioning that innovative education and training programs that address specific needs especially during transitions and the needs of individuals and inter professional teams are absolutely essential and no one can do this better than professional societies. Number two we have to focus on leadership and systems of care. We heard about the great leadership Institute of the colorectal surgery community. We have also the surgeons as leaders course as I mentioned earlier and that is morphing into a leadership Institute. We get people together during our Congress. Our course is held in spring. They get together at Congress to share their experiences and as is true with all adult education they learn more from each other than they learn from any one of us but it's a very successful oversubtribed course and that is a huge focus of ours and certainly our specialty societies have a very unique role and it is a role that is expanding and needs to expand and I believe we need to collaborate even more and work together to enhance surgical education so that we can provide the best education and training to our trainees starting from medical students up and to our practicing surgeons and our surgical teams until the last day of their practice. It's been a great opportunity for me to come here share some of this excitement with you and a great privilege to address you this afternoon. I'd be happy to answer any questions during the discussion period. Thank you very much. Thank you Dr. Sachdeva. That was great. We are lucky to have Dr. Sachdeva's counterpart on the medicine side. Dr. Devorin Chick is an internal medicine physician and chief learning officer and senior vice president for medical education at the American College of Physicians. Dr. Chick has the enormous responsibility of ACP's medical education strategy, products, services, and advocacy and she's going to talk to us about confirmation of relevant education for maintenance of knowledge, which I love. Different from maintenance of certification. It's very purposeful and intentional. I love it. Hello everyone. I share in the privilege of getting to speak to all of you. I suspect that you're already fully convinced that education is important for specialty societies, which is why you're sitting in these seats. I don't know about you, but we run a survey each year of our members and ask why they're members. What do they value about us? The top thing they tend to value is access to high quality education. They interpret that in various different ways. I think our premier journal, the Annals of Internal Medicine, is considered education for many of those folks or our clinical guidelines and so on. Of course, they highly value our advocacy work, but they really look to their professional society to set standards of quality and to provide them trusted, reliable, evidence-based information. ACP has had a service known as MixApp, a medical knowledge self-assessment program. A number of your organizations also have SAPs of various sorts. I'm going to let you know a little bit about some specifics of that. We've heard about the generalities of various education. Certainly, we provide the usuals, the simulation, the annual meetings, coding and payment guidance, our clinical guidelines, our online learning center. We've got AI-supported communication skills training and so on. It goes on and on and on. We want to really provide you with some very specific examples of ways that we have been updating and continuing to future-proof our medical knowledge self-assessment program, MixApp, to meet the specific new learning needs that we've heard about repeatedly through this conference. That our members, particularly our future members, our current Millennials and Gen Zers, they're looking for longitudinal engagement with the highest quality digital experience. They're looking for content integration, for exploration in ways that they can explore and learn and customize and engage with content on their own time. So we'll talk about a little bit about that. So yeah, I work for ACP. No other conflicts. We're going to talk about the enhancement of that adult learning engagement with online education. I was planning to talk about the modified NGOF approach. I may abbreviate that a bit, but I can certainly speak with you more personally about that regarding assessment standards if we run out of time. And we'll talk about some very specific evalued assessments. If you're interested in learning more about the specifics of MixApp, it's available to acponline.org.mixapp to learn about a little bit. It's a comprehensive learning system. There is a syllabus that spans the discipline in terms of the basic approaches. It's the equivalent of like 14 books in online format that also includes study and self-assessment tools and residency training program support. And it's curated. We will be launching the new modern version of MixApp in February. It'll be called ACP MixApp. Our prior versions of MixApp were all numbered. They were additions that had start and end dates. We will now be converting to a continually updated longitudinal system for the future. And we'll be enhancing it with what we call the core confirmation of relevant education. So a year in the digital life of our expiring platform MixApp 19, 22,000 unique visitors every week. More than half of those are physicians in practice. So these are our full members and slightly less than half are residents. They engage with us content. They find it engaging. Nearly 30 million questions engaged with over the course of a single calendar year. Almost over four million text pages and nearly two hours of learning per access. So the future is ACP MixApp, our new longitudinal version. We're continually evolving this. We're constantly looking for enhancements that will meet our learners needs. And that is how we are able to keep the entire internal medicine community excited about and engaged with MixApp. We've integrated within our online platform not only our full syllabus but also a digest like you know the Cliff Notes guide to internal medicine. So you can switch back and forth depending on your learning needs. Embedded values are interactive. Multimedia, lots of multimedia. I'll show you that. Online text highlighting in multiple colors. People can play with it the way they want. Gold standard question bank design that includes lots of bells and whistles. The content is continually updated and refreshed every quarter. There are new information updates, new question items, new elements related to clinical guidelines. And then the core of our platform is now a personalized learning plan that people can really centralize, add content or topics to. And that allows us to integrate the rest of our society's offerings within a learning plan. So I'll show you that. Also an ITE score upload. So the syllabus itself is up to almost 30,000 syllabus pages of content. So this is not something you're going to develop overnight. But it is worth growing because it needs to be sufficiently comprehensive for our physicians in practice to feel that it really meets their needs. We have extraordinary amount of editing stages and that is necessary to meet the high quality expectations. Contributor reviews, peer reviews, professional medical editor reviews, clinical physician editor reviews at multiple stages, both generalists and subspecialists. So this isn't just something that just, you know, gets popped out based on a writer and an editor. The questions themselves undergo about eight and a quarter hours per question for writing and multiple rounds of editing. So that's almost 13,000 hours of work to generate the questions necessary for ACP MixApp. And the critiques are robust. You can really learn. That's one of the keys to how people value our program. Within the critiques of the questions, they get significant amount of learning. And going forward, people are so excited about that, that we're actually going to start embedding figures and tables within our critiques as well. The tables themselves continue to expand. One thing that we've noted with the transition from print to digital is that folks really want, when you're reading online, you need it to be more tables, more bulleted lists, a little bit less free text. It is just a natural conversion in terms of the learning and reading space. Images and multimedia are essential. We've really bumped up our percentage. We've got 23% more images in ACP MixApp than was in our most recent image. This requires a significant image database element in terms of crowdsourcing fresh images as well. Key points are real important with our digital team as well. We get lots of feedback that folks, yeah, they love the content, but they also want key points that are provided right at the top of our content areas. We used to provide key points, you know, give them a section and then a key point, they would follow it as a reinforcer. Now we've collected all those key points. They're still there so that you can see key points within the embedded and within the text, but we also collect them all, put them at the top, and allow an expansion for folks who just want to really read through all the key points. And plenty of, as we said, visual updates. This is the board basic supplement. This is that CliffsNotes version of medicine. As you can see, it's very simplified, but sometimes that's what you really need. Just remind me, what was it again? What was the diagnosis again? What was the test again? And that's particularly helpful for folks who are studying for exams. They want responsiveness, so it's got to be handheld responsive and tablet responsive and personalized and customizable. Folks want to be able to engage with content, so all of our content is tagged, can be sorted, you can pull out questions. For instance, if you just want to answer questions that have to do with urgent care issues or hospital issues or end-of-life care issues or whatever, and track your progress. Interactions that simulate what you really need to do in life, particularly passing a board exam, are important. So being able to cross out answer choices, that's a big plus for our learners. They're able to cross out answers when they do the board exams, and so they want to be able to do that in their learning platform as well. And as much gamification as we can get, we're experimenting with ideas about, you know, keeping track of a streak for your learning and so on, with real-time feedback as well. We also provide plenty of customization. You can build things your own way, also easy buttons. Internal medicine has something called the longitudinal knowledge assessment, which is one of the options provided by the American Board of Internal Medicine for a maintenance of certification exam. So we offer a pseudo LKA experience within the Mixap program as well. And again, give them personal control. Allow anything that allows folks to take the reins and adjust things their way. This is a snapshot of the learning plan. You can add topics to them. Once you've added a topic to your learning plan, it collects information from across the entire Mixap platform. So the text, all the questions that are related to that, any images, issues, any notes that you've made, anything you've highlighted, it's all in your learning plan. And importantly, we also add learning links on that topic. Those learning links allow people to access other materials offered by ACP or Annals of Internal Medicine that are relevant to that topic. So they can link to an Annals of Internal Medicine article, a clinical guideline, and so on. It's all within there. Educators also love this. It's not just a learning plan, it's also a teaching plan. We also now offer an in-training exam feedback feature where our residents, once they've gotten their in-training exam score reports back, they can actually download directly into their learning plan all the topics that relate to questions they got wrong on the in-training exam. So our residency program directors are all signing up their residents for Mixap because they want to be able to allow them to support this sort of easy button learning plan. Board exam topic guide also facilitates a strategic learning plan. It allows people to access high yield questions, earn as you go CME, and so on. Now the confirmation of relevant education is our newest feature within the ACP Mixap. The confirmation core was built to provide further gamification and also to confirm engagement with the content, offer yet more questions, because that's the one thing everybody wants from Mixap, give me more. So this was a way to layer on yet another layer of content and interaction, and to provide a pilot service that's a true high stakes, not high stakes, but high quality assessment of their knowledge. So with core, you have extra questions within each area that remain locked until you have engaged with enough of the Mixap content questions and performed at a sufficient proficiency of those questions to meet the threshold to unlock that topic area. So this learner has not yet unlocked endocrinology or general medicine, they're the grayed out core areas in the bottom, but up top they have unlocked by doing enough work within gastroenterology, hepatology, and general until medicine too. They've unlocked them but have not yet engaged with the core questions or passed that assessment. The very top one, cardiovascular medicine, they not only unlocked but they engaged with the core quiz and they passed it. So they've earned a digital badge. Once they've engaged with the content and have managed to unlock a content area, they get celebration and they love this. Confetti comes across the screen, they get an announcement, you can, you know, congratulations, you've unlocked a core quiz. They might not even know they were doing this, but it's just like this, wow, it's this little, you know, what we call an Easter egg surprise within the system. It is so much fun. Once they engage with the quiz, we have various ways for them to get further feedback. Right now there's an LKA timing feedback feature. The content is standardized. I'm just going to say that the core quizzes are fully pretest and there is a highly valid passing threshold that's set through a modified ANGOF method, which I'll skim over, but we can get to it in the future. And there's yet more celebration, yet more confetti if you win. The brief version of the modified ANGOF standard is that a subject matter panel is convened. The subject matter experts try to, on their own, decide for themselves for any particular question, what percent of minimally qualified physicians are likely to get that question correct? And that kind of gives you a sense of like, what's that gray zone area? What percentage, not, not your, you know, the, your experts and your proficiency, which we'd like, but of the minimum, what's that percentage that they are likely to get it correct? And then you go across, there's some, there's some rounds of agreement, there's some averaging, there's some reviews, and you use this sort of general approach and at the end of it all, you've got a passing threshold that's set. So that's the, that's the crib notes version of it all. In addition, the items are also pre-tested, which is a standard pre-testing if you're involved in a, in training exam and so on, in terms of quality assessments and, and the performance characteristics of those questions. The results that you get from your core quiz link directly back into the learning platform. So after you've completed your core quiz, whether you got a passing score or not, you get information about the object, educational objectives that you engaged with, and a very easy link right into related text. So it really stimulates continued learning of that area. Once you've earned your digital badge, you can share it. They're valid for, at this point, five years. Once you've collected ten badges at any point in time, you earn a certificate which confirms that you are actively participating in the ACP Mixed App Core program. So this is a valid standardized certificate. People love this. In the pilot service, it should replace our board certification. It's given me motivation toward, toward completion, key learning points, very encouraging learning experience. They like that sort of carrot-on-a-stick kind of experience. We're already hearing about applicants for residency using this in their portfolio, wanting, especially the international grads who have already done some training and want to demonstrate that they're, their high level of skill, they're using it to demonstrate in their portfolio. Residency programs have been assigning badges for milestones demonstration and for rotation support. We're hearing a lot about alternative licensure physicians. This, we expect, will be another certificate that they can use to demonstrate engagement with quality learning. So we're working through that. I'm going to skip the research information. We don't have sufficient time to discuss that, but if you'd like to learn more about CORE, that's the link, and I'm available. Thank you. So I would invite anyone with any questions to come to the microphones and feel free to ask, but maybe I'll, I'll kick this off a little bit. You know, I think as healthcare is changing, and who knows where it will evolve to, but stakeholders are very interested in the business of medicine and quality outcomes, and we know when you practice, quality comes from the education and learning best practices. But how do you – I think that the physician population, the healthcare provider population is going to become more and more dependent on our societies to provide this education as it becomes less important to the other side of where they work. And so how are you planning for the future for this, the increased need or demand of your content for your members that's not so much happening as much in other realms that there used to be, and how are you planning for it in a multi-generational way? So for example, at Askars, it's very easy to get our young people online to look at these things, but it's much harder to get people after a certain age to engage with that. And so maybe talk also about in-person or other forms of engagement through which we can approach this multi-generational group in healthcare today. Big question. I will share with you one or two perspectives in regard to the great questions you've asked. First of all, I think we need to have the longitudinal programs that we have talked about briefly this afternoon, and that has been shown through good educational research that if you have a one-time intervention and there's no reinforcement and applicability, it gets forgotten. In fact, all of us have been subjects of that. We go to a CME course, we listen to some great presentations, and next week half of it has just gone right through our brains and is past history. So that is something that we need to do, and there has to be constant reinforcement. In regard to online learning and so on, I think that there is a clear place for online learning. We've learned a lot during the pandemic that is bearing us in good stead. We've seen that not everything requires in-person dialogues and in-person interaction. Having said that, everything that we need to teach and everything people need to learn cannot be done all online. In fact, surgery, for example, we have to be there in the trenches with our trainees, with our young people, with our colleagues doing what we do best and demonstrating what we do and how we do it. So there's a balance, and that's something that all of us are struggling with where that happy medium is. We're also struggling with the generational issue that you mentioned, my final comment. I find it troubling sometimes when I have residents talk to me about really accessing things in bits and using it in bits. I think that is very useful for certain areas of reinforcement, but you have to have a certain foundational knowledge and skill set on which you can build. You cannot train a surgeon to be an expert surgeon by looking at snippets that are three minutes long online. It cannot happen. We have to have an in-depth understanding of issues, and there are two things in expertise. There's general expertise and adaptive expertise. We wrote an article on this. The adaptive expertise is when something goes wrong, when you find an unexpected finding in the OR and you have to make a decision. That cannot be done in snippets. You have to be there. You have to learn in context, and likewise, as simulation complements our real experiences, we have to have a hybrid model between online learning and also in-person education. Those are some of my comments. Great comments. Dr. Chik? Yes. I think fundamentally, this is all about approaching what we offer from an instructional design perspective. If we have a tool, maybe we bought into an LMS, we've got an AI something or other, we're trying to create stuff with it because we want to get some value out of it. Unfortunately, that locks us into you've got to hammer, everything's a nail. Indeed, online education, Mixed App is just one part of our portfolio. Instructional design is about putting the learner in the middle. It's about figuring out what are your objectives? What kind of skills are you trying to train? What kind of knowledge are you trying to date? What's the best way to make that happen using the feasible tools at your disposal in a way that's going to be engaging? What's going to be the evaluation method and so on? Really thinking, sort of stepping back as educators and not forgetting the basics of instructional design. Sometimes it's a webinar that makes most sense. There needs to be conversation, but it doesn't have to be in person. Sometimes you need a two-day long conference. Sometimes it's a two-minute push or a YouTube video. You do have to stop and think about what exactly you're trying to accomplish. Okay. Eric? I just will add something briefly related to the bites of learning. They do need to lead into something larger. With every five-minute video here, that's in a clinical practice guideline here. That's in a textbook here. That's in a webinar here. Having that variety of resources for our members that are easily accessible, where they know where to go, when they need whatever type they desire, I think just having that whole breadth available is vital. All right. I think we have a question. Good afternoon, everyone. My name is Christopher Cross. I work for ASCO. My question is regarding, you know, you've shared a lot about the content that you all develop and how amazing it is and even adding on certificates, but I think one of you also mentioned one of the issues is how do you sort of measure the impact of the content that you have developed and shared? And I know, like, measuring impact will take on sort of a unique spin based on what you want to measure, but if you have any sort of general frameworks or advice on how do you actually do that for the folks you're training, especially through these virtual educational models, that would be ideal. Ah, excellent question. I think in general, educators tend to underperform in terms of assessing ourselves and our education. We put too little effort into the outcomes assessment of what we're doing. So there's an entire research plan that we put together when we put together a new educational intervention. We include an assessment of outcomes as part of that plan. You can kind of imagine it as if you're writing a grant proposal for anything and just ask yourself, what am I, if you've got learning objectives, again, it's part of that instructional design, if you've got learning objectives and goals, how are you going to evaluate those? How do you know if you've achieved your goals? Sometimes the best you can do is, you know, learner enjoyment, you know, stage zero, do they show up? That's actually important information in many ways. People value and show up for things that they feel that you're offering value to them, you're meeting a learning need, but, you know, we do retrospective pre-posts for online modules quite a bit. That's a simple tip I can give you. So a retrospective pre-post is if your specific question was if your online module or whatever that you've given, rather than doing a pre-test or a pre-assessment and then a post, people before they've done the module, they don't, it's the old thing, they don't know what they don't know, right? So if you ask them a pre-test or a pre, you know, how much do you know about this, they're going to say average, you know, maybe whatever. But at the end, if you ask them a dual question, let us know, you know, now that you've completed this, how would you rate your knowledge about this information before you started? And how would you rate your knowledge after you've completed it? And you'll get, retrospectively, what was your pre-knowledge and your post-knowledge. And you'll get a much more rich description of whether they felt they believed they learned something. So that's short-term knowledge change. And then it goes up from there. We do something similar, actually, where we, you know, send out post questionnaires asking, you know, what knowledge they learned. If it's changed patient outcomes. And we follow up, again, six months later, you know, requesting that information. So yeah. Yeah. Yeah, if you look at the Kirkpatrick and Kirkpatrick model for evaluation of the impact of education, the reaction is at the bottom. And that's what we measure, usually, through questionnaires that we send out. Then you go further up with the outcomes being the final step. The problem with outcomes is, quite often, it's very hard to establish causality. For example, someone goes through an educational program. And three months later, that person handles a patient well. Is that because of that course? Or are there other confounders that impact that person's work? And we, as educators, need to decide what is amenable to that patient outcome. And in my mind, if it's a simple procedure, for example, you train somebody to put a central line in. And you look at the impact on the patient in terms of reduction in complications, bloodstream infections, pneumothoraces, or whatever your yardsticks are. In a short period of time, you can probably create a dotted line between the two. But if you're talking about a major procedure that someone has learned, there's so many confounders. There are systems issues. There are team issues. There, I think, we, as educators, need to know where to look at performance rather than patient outcome. So that's a debate that we always have to have among ourselves and within ourselves to figure out what is what along that hierarchy. Male Speaker 1 Thank you so much. Female Speaker 1 Excellent question. Oh, the boss is at the mic. I had to run out for a minute, but I confirmed that nobody asked my question, so I'm going to ask it. I was really struck. Thank you. This was just a great panel. Davern, a question for you. I was struck by your slip of the tongue when you said, high stakes, I mean high quality. And in reality, what you're presenting is core. I'm not sure it's very different than, in fact, the LKA. So I guess a question sort of for all of you as you think about what the societies are doing in terms of education versus the sort of the role around the certification of the boards. You know, are there more opportunities to, in fact, bring the society-developed resources forward as a way to, in fact, get at at least the maintenance of certification? I'm not talking about initial, but the maintenance of certification. Thanks. Davern Dixon Yeah, what a great question. And I'm sure a few of you are aware of some of the controversies, especially in the internal medicine community in this space. So you know, the boards were designed for assessment of individual physicians and to give credit where credit is due. They're assessing not only in knowledge, but they have some other elements of primary source verification and response to professionalism complaints and so on that they do as their scope of work. They're not educators. They never have been, and their systems are not designed to be educational. So with the onset of the longitudinal knowledge assessment, while it does provide question-based biopsies of knowledge, it's not a learning platform, and it doesn't replace learning. One of the biggest concerns we've had as a professional society is the concern of unintended message that if you're engaged in something like that and you're passing it, then you don't really need to keep learning. And that's really not what our profession needs. Our profession needs to stand up for the highest standards, as we've said, not just the minimum competency, but really stimulating folks to enjoy learning to, at least in their knowledge elements, if that's the tool, it's a knowledge-focused tool to get to a mastery of knowledge and proficiency. And so that's why something like ACP Mixed App Core is, frankly, better. It's not equivalent. I'm just going to say it. It's better than the LKA. It's just as standardized. It's just as valid in terms of the way the standard's been set, and it's more engaged for learning. It's more stimulating. People enjoy it. They feel it's higher-quality content. So we can do that, but we're not doing all the other stuff that they deal with. And so we also have to recognize that while we're doing that better, because our skill is assessment, is learning, we need to partner with them in terms of the other things that have to do with certification. The hope is that we can, you know, that ABIM can recognize what we're better at and maybe acknowledge it and, you know, give credit to people who are engaged in something like that or at least feed into the platform by providing, you know, you can imagine a world where your ABIM exam results get downloaded into your learning plan just the way the in-training exam results do. We haven't been able to get to that space. But you know, there's a world there where learners and physicians are fully supported by a learning platform that they love with feedback from a certification board that they trust. Great. Dr. Sachdeva? Yeah. It's a great question, and I would like to share my perspective in regard to how the world has changed during the last couple of years and how our societies are much better placed now to support the endeavors of the board, and there can be much, much, much more effective collaboration. In the past, as we all know, in that continuing education space of 30, 40 years that I alluded to earlier, there was the every 10-year exam that all of us took. So our role as a professional society was to buff up the knowledge of our members so they could pass that exam, which was pretty tough. It was high stakes. It was in exam centers, et cetera. Now the boards have moved more into the formative space and more frequent lower stakes exams like the American Board of Surgery has a two-year cycle now of an open book exam. And I think that provides our specialty societies an even greater opportunity. We are just not training people to sit for an every 10-year exam. We can provide that wholesome experience that the boards cannot do because we have the expertise and the depth and the resources as professional societies that we can partner with the boards and really make a big difference. Thanks. All right. Thank you. Yeah, I would just like to echo what our colleagues up here have said in that increased collaboration with the boards are now more important than ever, too, because, you know, they have the data that says, oh, these are the missed topics, these are the missed questions, and the society can use that data to create our content, develop our education so when we're doing our gap analysis annually, we're incorporating that to create the education that we can then share with our members and our learners so they know that, you know, we're working together as, you know, one specialty. All right. Eric, I know you have a flight to catch, so feel free to take off. We have one last question here, please. Thank you. Well, first of all, Mark Del Monte from the American Academy of Pediatrics. Thank you for this incredible and dynamic session. I almost think that we need a myths and facts sheet to come out of this session because I think we might all be hearing a lot of the same thing, that because everyone works for a large health system now, there's no role of medical societies in education. They're going to get their education somewhere else or they're going to get it online. Or because young people are not interested in in-person learning, it's all going to be online anyway, so there's no role for in-person learning anymore. Or nobody wants paper or anything anymore, and so we should just get off of that and it's all going to be on the phone. There is such an opportunity for innovation in this space. There's such a dynamic role for societies to be able to meet the needs of our members in ways that are designed perfectly for them and no one else in ways that nobody else can. These persistent notions continue to come at us from all different directions. I think you've, not in that framework that I've just laid out, but you've busted myths just by the way you've been describing the work that you're doing. I wonder if you just take a second to address the most persistent myth that you hear and what you say in response to that, and if we could figure out how to continue to do some myth busting after this session is over. I'm hoping this is busting a myth, because I go back and forth. This is what keeps me up at night. There is a perception that folks just won't engage with learning anymore or engage with society learning because of changes with the boards and because they can click off their CME, they don't really have to learn some things, they can do a longitudinal assessment, they get some passing grades and they're done. The reality is, as we said, the majority of ACP's members, what they value about ACP is high quality education. When you ask people why they're buying MixApp, the number one answer is not to pass a board exam. The number one answer is to advance my knowledge. Physicians went into med school, we went through all this stuff for all those years because we liked it and we find it interesting. If we as societies continue to provide learning in ways that are enjoyable, that invigorate our attachment to our profession, that make us feel competent, because it's overwhelming how much we have to know and it's continually changing. If we can be partners to our physicians out there and be their friends in learning, I think there's a lot of space there. Dr. Satchin. I think we need to also bust the myth that people are just chasing the regulations. Yes, people have to meet the regulations, but it's the joy of learning that you were mentioning earlier that we need to focus on. I'll give you one very quick example. Our CSAP program, which is like MixApp, there we got our regents to approve credits of excellence. Now we weren't sure whether one person would subscribe or 10 would subscribe out of the thousands that take that course and that program every three years. Lo and behold, that has been the real bright light of that CSAP program. People subscribe to that excellence model, and in the excellence model, they have to achieve 100% the first time around. They read the material, then they have to get 100%. There's no second try. That's the level of excellence that we are having people strive towards. What they enjoy is seeing their performance as compared and benchmarked with their peers. Even when we look at the future, and I agree, some of the education is going to get picked up by healthcare systems, but the healthcare systems cannot benchmark individual performance across the country. The professional societies can, and that's a unique opportunity to get people excited and really go in this direction working towards excellence. Great. Thank you so much to our speakers and to our questioners. Thank you. Really great session.

ASCARS

medical education

Surgical Leadership Institute

ASCARS U

lifelong learning

multigenerational audiences

technology in education

professional development

American College of Surgeons

American College of Physicians