Privilege, thank you. I'm having word finding difficulties and get started because this is going to be, I think, a very stimulating session. My name is Miguel Paniagua. I'm the Vice President for Medical Education at the American College of Physicians. I am pleased to have three presenters today of two different topics on innovations in medical education related to the use of AI. I will briefly mention, if you have a chance to look at the New England Journal of Medicine perspective piece from August, 2023, by Dr. Adam Rodman, who's a frequent collaborator of ACP, actually. He's at Harvard Medical School. He mentioned in this article, very specifically, that professional societies could lead the way in preparing members for innovations such as AI and medical education. Consider that a call to action. So, first up, we have Leah Binder, who had a distinguished career in procuring grants and funding and doing education, had previously worked at the American College of Cardiology and currently is the Chief Learning Officer for the American Society of Addiction Medicine. And following Leah will be Susanna Alexander, who is the Director of Educational Publishing at the American College of Emergency Physicians. And joining her will be Bozy Jovicevic, who will be presenting as well. He is the CEO and co-founder of EverMed. And EverMed is a tech company, if you don't know about it, that essentially helps organizations with content enhancement in their learning platforms. So, pleased to have Leah Binder join me here at the podium and we'll get started. We'll have plenty of time for questions, hopefully, at the end of both presentations. Okay, good morning, everyone. So, I'm here to talk about artificial intelligence in medical education, specifically with instructional design. So, for those of you that aren't familiar with ASAM, here's just a little background, but we're really dedicated to improving access and quality of care for patients with substance use disorders. So, today our objectives are to talk about some specific examples of AI tools that we've been using, talk about how we're using them and implementing them into our practice, and then some of the challenges and important considerations as we continue to embrace AI. So, just to level set a little bit, because I feel like we are using artificial intelligence more than I think we realize. Like, we've been using Waze to avoid traffic, we've been using Alexa, how many people have Echoes in their homes? And then Netflix, which video we should be watching next. All of these are artificial intelligence powered by sophisticated natural language processing or complex algorithms based on analysis of large data sets. So, AI is already here. What's changed is that it started advancing at a much faster rate. So, now we are moving away from those everyday conveniences into much more transformative tasks. And so, we're just starting to really tap into the potential of what we can use AI for in medical education in particular. So, today we're sort of focusing on the role of AI in healthcare being around these four areas. One is as an enhancement to the quality of their education, whether or not they're small audio or visual editing tools or the ability to develop more complex simulated case scenarios, it's really elevating the quality of the education that we're developing. Also for streamlining. So, some things like generating a custom graphic is really time-consuming. You need a graphic designer and specialized software. And now with AI, that does the heavy lifting. We can very easily enter some text prompts to be able to generate a completely custom image to whatever meets our needs. One of the biggest benefits is in terms of the time saving from automating repetitive tasks, such as transcriptions. I remember just years ago after our annual conference, we'd have to hire temps to go through all of our sessions to provide those transcripts for accessibility. And now we can just use one of the apps. And then last is that more personalized experience with the ability to analyze all the data quickly, we're able to come up with some more personalized learning experiences for our learners and our members. So talking about some specific tools now. So video and imaging and all of multimedia is a very important part of the education that we develop. I think especially in addiction, we're reaching a lot of non-members. So they're sort of not really seeking our education and don't recognize their knowledge gaps. So we really make a concerted effort to make the education really engaging. Sometimes it's really about trying to make a complex topic a little bit easier to understand or a dense, not as interesting topic, a little bit easier to digest. But regardless, it has a very important role. And the three tools that we've really been using are Descript, CapCut, and Topaz. And I'll talk a little bit about how we use them. So the first example is eye contact. And this is a big issue that we have with our recordings. A lot of our faculty are constantly, when instead of looking directly at the camera and having eye contact, their gaze is constantly shifting because they've got their notes, they might even have a script. So it's really distracting when you're watching this video and it's hard to really engage with the learner because they're not looking at the camera. So that just kind of takes away from that whole learning experience. So Descript has this tool and you can see on the slide, it's a very simple, like clicked, and it's able to redirect the presenter's gaze to be directly looking at the camera. So it improves the overall look and feel and the ability for the learner to connect with that presenter, but also just elevates the overall professionalism of that video too. So here's just a quick example. At the top, you can see that his gaze is not looking very direct. And the bottom image, it's a big improvement. And I've just been kind of like, it's such a simple thing that makes such a big difference. Okay, so now getting into image quality. We receive a lot of images from either our faculty or external partners and we're like, we have to use this image. And rather than having to spend a lot of time trying to find a replacement image, we've been able to use tools like Topaz and Canva to enhance the quality of them. So Topaz has this upscaling enhancement. So even if it's a low quality, low resolution image, it can apply these features to really sharpen that image and make it much more of a usable, high quality image. So it saves us a lot of time from trying to replace it. So here's an example of that. So at the top, it's pretty blurry. It's not a great picture, but at the bottom using Topaz, it's greatly enhanced the features and sharpened that image to make it a high quality image for the presentation or whatever course it is. This is an example of using Canva for a generative fill. So as you can see in the top picture, part of her head is chopped off, part of her arm is incomplete. So using Canva, it intelligently fills those missing components to make it a usable, high quality image. So virtual backgrounds. Before COVID, we used to be able to have our faculty flown out to a video studio to have very high quality videos, very professional, well done. Now everything's over Zoom. Even if we have gotten back to traveling, doctors no longer wanna have to take the time off to travel, spend a whole day just to do a one hour recording. So everything's over Zoom. Problem with that, not all the video quality is great, some of their backgrounds. Some people have these really impressive like book cases behind them and it looks great, others much less so. And then when you have multiple different videos of different faculty within one course, it's kind of distracting because you're kind of switching through and like, oh, this one looks great, ooh, this one less so. So it really takes away from the quality of the education. So we've been using CapCut and you can see the top image, it's really dark, it's kind of distracting, it's not impressive. And then the bottom image using CapCut, we've completely replaced it. It is now a professional background, it's lighter and it's no longer subtracting from the content that the speaker is talking about because it just overall, you have this great look and feel. So talking a little bit about audio enhancement, there's two tools that I'm gonna talk about. Eleven Labs is mostly used for voice cloning. Descript is a really versatile tool, that's what we use for the eye contact correction. So with Eleven Labs, one of the ways that we use it, and we actually, so there's two ways. The ability to clone voices is two functions because when we have all these audio, like the production time or sorry, post-production, it's really time consuming. You realize that, oh, this person kind of mispronounced a word or, oh, we wanted to say this thing instead and having to re-record that is really time consuming or trying to cut things out, like to really splice it exactly so that it's a seamless transition takes so much time. So being able to use the voice cloning with Eleven Labs allows us to, if we're just trying to fill in like one or two words or even a sentence, we can replicate the voice to correct that part of the audio and to demonstrate how natural sense, oh, actually, you know what, let me go back. So I wanna talk about the other way, which is the newer way. It's a little bit more substantial and this is getting into those ethical challenges. When you have really limited availability for your faculty and in one case, we really wanted four hours of recording time with one of our experts and they were like, no way. Maybe in three months, but that's not happening. They're like, we can give you an hour and we really needed this one particular expert. No one else had the credibility and was going to be able to replace them. So we decided to kind of take the plunge and explore using full voice cloning to develop the course with his audio throughout. So we did about 40 minutes of recording time and had enough of his voice that we were able to apply that using Eleven Labs to a pre-approved script to generate very natural sounding audio throughout the course. Okay, so I'm gonna, to demonstrate that, I wanna play three different clips and one of which is the human and the others are AI. Let's discuss some consequences of not screening. Who wants to guess which one is the human? So the first one was AI. And so was the second. It was the third one that was the human. So I think this is just a good example and this was a very quick clip, but just to really indicate that the voice cloning, it is effective technology and it really isn't that robotic thing that I think a lot of people would be concerned about when we talk about voice cloning. We can also use Eleven Labs to generate a whole new voice. So it doesn't have to be a specific person we're cloning, just in general, creating an artificial voice that they have that capability to. So more about the specific editing of audio. Again, this is super time-consuming, can be complex, require sophisticated software, but using tools like Descript, we're able to edit very seamlessly, whether or not it's editing out background noise or editing out specific chunks. So since Descript is what we also use for transcriptions, once the audio is now in a Word document, you can cut out the specific word sections that you want to remove and it does it automatically. It's able to pair the audio edits to the text edits. And anyone that sat there and listened and kept trying to splice to that exact nanosecond, it takes forever and sometimes it's just repetitive, oh, that's not good enough, let's try a different spot. So this is really speeds up that post-production of audio editing. So I've got two examples on this. So the initial steps of assessing pain in patients is really important. So the initial steps of assessing pain in patients is really important. So I think that kind of shows how it's able to remove some of that background noise and really make the voice a little bit more crisper. Here's the second example. We need to be connecting with our patient, helping to understand their perspective. We need to be connecting with our patient, helping to understand their perspective. So again, it's just much more crisper, better quality audio. All right, moving into more of the generative AI and creating custom visuals, which has been really a more impressive recent technology. So the two tools that we've been using are Dolly, which is a part of ChatGPT and then Canva. Similar to Descript, we use Canva for a lot. So this has really great capabilities. So in a lot of our courses, we really wanna create these custom graphics that connect to the actual specific content we are talking about, especially when we do a lot of patient case scenarios. So being able to develop a custom image that matches that case scenario, especially to be able to present multiple versions of that same character really enhances sort of having the learner kind of buy in and connect with that patient that they're talking about. So Dolly is able to, if you can kind of read in the slide, it's a text prompt. What you're looking for, it creates multiple different versions based on what you've requested. And you can, like other things in ChatGPT, you can continue to refine it. This looks great, but can you remove that? So it's been a really valuable tool because we don't have, we've got wonderful instructional designers on our team. We do not have a full-time graphic designer. It is extremely time-consuming to create a totally custom image. And then if you need changes to that, it delays things even further. So Dolly really provides a great sort of resource for this in particular. So here's an example of one of those in action. So this was a case scenario for one of our courses on alcohol use disorder and co-occurring psychiatric conditions. We needed to have a 24-year-old indigenous male who was being seen at the hospital and with a very wide smile, trying to indicate a manic episode. So we were able to create this particular image to meet all of the different questions within that case scenario. So this one's using Canva. And for this, we really needed an accurate anatomical image to show the safe and unsafe practices of using long-acting injectable buprenorphine and stock images was not going to cut it for this. So Canva was able to fairly simply create exactly what we needed for this particular course purpose. Here's a simpler version of it. It's just, we were trying to create a circular image that was capturing the spirit of motivational interviewing with different components and principles and with custom icons. So it wasn't, it's probably something that we could have found or created a little bit easily, but it saved a ton of time. It was just a simple thing. And we were able to just really accelerate the overall time it took to develop this course and make sure that throughout it, we had a lot of really engaging visuals that really just enhanced everything. Okay, so now moving into some of the challenges and considerations when we're using these different artificial intelligence tools. So the first one is on that sort of privacy and security, specifically with the voice cloning. So this is dealing with sensitive biometric data. And there's just this kind of, some people have this ick factor, like, ooh, that is like my voice out there. What is it going to be replicated for? So wanting to make sure that we had safeguards around it. So we made sure that we had a very explicit agreement with the faculty member that we used voice cloning with to ensure that we were only using his voice for this project that we would only use it applied to pre-approved scripts and that anything that was using voice cloning, he would be able to review before it was used in anything and be able to have any sort of edits or concerns expressed. And that it was really limited to that project and it wasn't going to be like accessible somewhere where staff could start using it in any course they want. So that was sort of how we tried to address the concerns about privacy with this. The second one is on managing bias. So when you're creating a custom image, the data sets that these AI tools are pulling from might have unintended biases. They might have a lot of stereotypical visuals. So whatever they develop might be biased. And that's something that we really make a concerted effort to avoid, especially in addiction. Biases and stigma are so substantial to being barriers to improved care. So what we've done, and we always have levels of review for any of the images we use to make sure that they're appropriate. We've just added to that. So not only is it additional staff that's reviewing any images, we also have additional levels of our member experts and SMEs to ensure that any images don't have particular biases that were unintended. The second is balancing human experience with the AI tools. So these are really great tools to streamline tasks and save time and elevate the quality, but they are not a replacement for human expertise. They don't have the ability to think as creatively. There's not that critical thinking. So what ASAM has been doing is really leveraging these tools as an accelerator and an enhancer, and not as a replacement. And then the fourth challenge is related to accessibility. Somewhat similar to the issue of bias, it's really important that all of our education is accessible to all types of learners, but the AI tools aren't thinking about that either. So whether it's an image that doesn't have alt text to be able to run through a screen reader or if it's lack of captions, we need to make sure that we are addressing any lack of accessibility issues with any of the AI visuals in particular. So similar to bias, we've added extra layers of review from staff and our experts to ensure that we've corrected any potential accessibility issues. So our takeaways from all of this, mainly that we need to be embracing AI. It's a very valuable resource to elevate and enhance the quality of our educational content and it is something that we should be trying to leverage more as it has so much versatility and value. But as we're doing that, we need to be really considerate of some of those challenges, especially the ethical considerations to make sure that we are using it responsibly. We also need to make sure that we are balancing AI with human expertise and not trying to have it replace it. And then lastly, stay agile. It is rapidly evolving. It is continuing to improve and have so many more uses. So we need to be staying abreast of all of those developments, especially as they are different improvements, to make sure that we are using them and incorporating them in our development of medical education. And with that, thank you for your time. As a reminder, we'll save questions for all three of our presenters after the second presentation. I'll invite Susanna to the mic from ASEP. Hello. So I'm Susanna Alexander. I'm the Director of Educational Publishing at the American College of Emergency Physicians. And today we are going to be discussing how we rapidly implemented an AI-enhanced educational multimedia library. So basically, the purpose today is to talk about all of the challenges that we face, you know, with physicians and their learning barriers. How do we pull content into an all-in-one AI-enhanced multimedia library quickly that can basically amplify physician engagement and help to create a monetization pathway for content? So if you have maybe an old user interface, maybe you have a lot of content in a lot of different places, recordings, webinars, podcasts, publications, articles. Maybe you are using a traditional LMS and your engagement is less than optimal. Or maybe you're looking for new ways to monetize your content. That's what we're discussing today and how we put all of that together. So as we know, physicians are busy. They're overwhelmed. There's a mountain of medical education on the internet being produced by organizations. And so it's basically gotten to the point where it's impossible to get through. And then we as educators, you know, we're struggling with the technology that's evolving all the time. Learners' expectations aren't based on what we're providing. It's based on, you know, their experiences with things like Netflix and Spotify. And then we have, like at ASAP, we have a lot of content. But it is dispersed on multiple platforms. We have it in courses, we have it in products, we have it on the website, we have it in meetings. And there's really not a place where it's consolidated. So that makes it very difficult to find and consume for the physicians. You know, traditional LMSs, they can be clunky. They can have user interfaces that are difficult to navigate. Sometimes the search engine is less than optimal, so even if you have the content, it's still difficult to locate. And ultimately what that leads to is underutilization and undermonetization. So we at ASAP have recognized that the trend is towards personalized on-demand microlearning. People want what they want to watch. They want it when they want to watch it. They want it to be in small chunks. And they want it to be whenever they need it. So they want to be able to search for that topic, pull it up, and they expect their interface to be easy to use, and they expect that content to be curated. And in order for it to be, like, useful and engaging throughout the year, it has to come in bursts or dripped learning. So in order to solve this issue, ASAP has partnered with EverMed. And they bring, this is a true partnership. So we bring the trusted content. We bring the built-in audience. They bring the technology solution, the strategic expertise, and how to increase engagement. And we roll that all the way up into a subscription-based digital product. So and we were able to basically take this from idea to launch in roughly six months from the time that we signed the contract. And so in that six months, we spent the time basically consolidating all of our content, designing the user interface, going through, you know, the technology setup for, like, the SSO, testing it, setting up the email engagement engine, and then setting up all of our monetization pathways. And so over the next year, our goal is to basically, in iterations, optimize that for basically all of next year and beyond. So today, we're going to concentrate on how do you get the content, what content do you include, how do you improve the user engagement, and what are the revenue streams that you can create with a product like this. So what content can this platform hold? Basically it can host video, podcast, and PDF formats. What we did is we pinpointed, we did a little bit of an audit of all of the existing content that we had in those formats. So short videos, webinars, podcasts, any publications, handbooks, guidelines, slides, articles, anything that we were producing that could be hosted on this platform, we did an audit of it and basically identified its usefulness. Some of that content, you know, we located and decided that it maybe needed a little bit of work. Maybe it was an hour-long presentation that needed to be split up into parts. Maybe it was on the website, but it needed to be converted to a PDF. But that audit sort of allowed us to figure out what we had and then identify what kind of original content we needed to create in order to fill gaps and make this a complete platform. So we know that 91% of physicians want short-form videos, that 65% want audio and podcast. And that for our PDF content, they want to be able to download it and maybe mark it up or things like that. And for our particular case, we were already getting CME accreditation for the parts that we were putting into the system. That allowed us to basically use our existing CME system in order to claim credit for the content that's in this platform. So what kind of content doesn't really fit into this platform? That would be more like the long-form, linear courses, the SCORM files, maybe live streaming events, and then like the board prep and maintenance of certification type question banks and things like that aren't going to fit into this type of platform. So you know, when you're doing meetings and you're doing conference content, there's a myth that basically everybody attends the meeting and then that content is maybe good for one to two months after the meeting. If they don't go in and watch it directly after the meeting, you know, they're not going to. But the truth of the matter is that if you curate the content in a way that they want to consume it, they will consume it year-round. And my goal is actually to keep that content up-to-date and evergreen on a system so that we can basically accumulate more and more content over time. So a lot of the common challenges with meeting recordings in particular is that they're often long. Maybe they have generic naming conventions. Maybe they aren't tagged properly for their topics. Maybe you're having to spend a lot of time looking up generic stock images in order to put it in your LMS. So you know, it can be a very time-consuming process to actually create, you know, a library of recordings after a meeting. And so, you know, just to reemphasize, you know, basically what we did, especially for the conference content, we audited it, we centralized it into one location. We decided that nothing over 30 minutes was going to go into this system, so we split it up into parts. We organized it. We made sure that it had proper descriptions and tagging, and so that the AI that's using those things for search tools is accurate. We optimized it as far as, like, using visually appealing thumbnails and making sure that the audio is correct. And we also have a system where we can do a bulk upload to publish the content. So that leaves us with basically a Netflix style of content where you have an image that's generated by part of the presentation. The length of the video is clearly marked. You have a very descriptive naming convention. Whether or not it has CME is clearly labeled. And in this case, you can also have descriptions that can be added and that can be AI generated and then reviewed by the content manager. And those descriptions are valuable because the AI actually uses those as a part of the search tool. So like I said, we have started with the content that we have. We've also started generating a little bit of original content. And it can be, the ideal length of a video is roughly 8 to 10 minutes, but we have things in the system that are still 25 minutes long and people are actually watching them through to completion, which is somewhat surprising, but they are. But we also have some new videos that are anywhere from 30 seconds to 3 minutes that are also some of the most highly used content. So you can have a wide variety of links and people are going to actually consume it. One of the things that we did to leverage this process is we took a historical magazine that we've been publishing for the past 38 years. It was stuck in an old format. It's a 32-page PDF and it was just on a website where people would open it in like an Adobe Reader window and basically download it. So by putting it into ASAP Anytime, what we have the ability to do is we have the ability to break out the articles and all of the features into separate PDFs and we're able to curate collections. So for critical decisions in emergency medicine, we have, for example, a feature that's called the critical procedure. And rather than having to remember, you know, that you saw some procedure like three months ago and you can't figure out which issue it was in or something like that, we're able to create a collection of all of the procedures over many years, at least three, into like a procedures collection and then those are completely searchable and findable on demand whenever people need them. So we have a lot of other things. Just as an example, we have primers and handouts and things stuck on our website, maybe on a section page that, you know, is buried two year, it was published two years ago and it's difficult to find. So this allows us to have collections of things that are sections for this one, it's a research section, like they publish an epic primer and they actually have four of these. And so we're able to give them basically a little library of all of their content that they've produced and it's in an easily accessible way. So, I mean, the purpose is to basically optimize it so that we can increase engagement to get us to, you know, the place where we have curated usable collections. In our platform, we have allowed both CME and accredited and non-CME content and, of course, we know that eventually our users want to have a mobile app and so that is on the roadmap for 2025. So, as I mentioned before, we were already previously accrediting our content and so like we were accrediting the meetings content, we were accrediting the magazine content. And so, ultimately, when we added all that together, it's an enormous amount of CME credits but it's more than any one person is ever going to use in a year. So we basically capped it at 250 CME hours. We have enabled a flexible attestation process where basically the program does tell you, it does like tell you how much watch time you have but it's up to the user to go into our system and say, I have watched five hours of talks and I want to print my certificate on that. So, like I said, the goal is to increase year-round use and personalization. The way that AI helps us do that is basically when the user first goes into the platform, they tell us their specific topic preferences so that maybe they'll tell us that they want all of the trauma or pediatrics content. They have the ability to follow any trusted experts so that also tells the AI like who they're interested in following and gives them sort of like a profile to follow. And then, you know, over time as they use the platform, the AI learns what they are clicking on, how similar they are to other users and it feeds them the content that is most like what they're going to want to see. And so, that triggers sort of internal motivation to like keep coming back to the platform because you're basically being served up what you are most interested in seeing. So, I mean, that basically solves the problem of the content being put into like an LMS where it just sits there and it's not watched or used. Instead, you know, we've put as much as possible into a consolidated all-in-one multimedia library where they can come back and they can engage with it and use it on a regular basis and that gives us the opportunity to also generate revenue. So, a lot of people don't know the difference between an LMS, a learning management system and an LXP. A learning experience platform is basically a little bit more personalized, a little bit more informal. You can have a lot on this platform. It doesn't have to be just like clinical content. It can be like for us, it's like the business of emergency medicine. Maybe it's an example of, you know, an interesting career choice, how to maybe be an emergency physician who works for NASA or something like that. You can have videos of all kinds in one place and so it makes it a much more dynamic and social learning experience. So, the goal though is to make it habitual, right? We want people to like, save, share, download the content. We want the emails that we send out to be relevant. So, we're sending out, we're not sending out sales emails, we're sending out content emails. We're saying, look what we have now, check out this video. And because of that, the open rates and the click rates are much higher than normal. And so, we're going to watch a little bit of a demo of the platform, hopefully. To this short ASAP Anytime demo, let's start. We are now on the ASAP Anytime landing page where users can find out more about the platform and purchase a subscription if they don't have one already. Users who have access can simply click on the login button to access the platform. The platform has single sign-on, so logging in is super simple with only one click needed. Now, we are inside the platform and the first thing we see is the featured area with four pieces of content promoted. These content pieces can be either selected manually from the ASAP Anytime admin panel or they can be selected by the AI recommendation algorithm which promotes content based on the user's interests, watch history, user type, and many more factors. The platform has three types of content available, videos, podcasts, and publications. If we scroll down the home page, we will see various content carousels. The first one is the continue watching carousel. The platform remembers where you stopped watching videos so you can just pick up where you left off. The carousels are horizontally scrollable so you can find everything easily. The platform's recommendation engine will suggest the most relevant content to the user. We also have collections which are content playlists usually grouped by topic and there are also events so users can find last year's or this year's content with just a click. Expert profiles are also present and the platform also helps with finding newest or most popular content. There are carousels based on specific content types such as publications which can be used to promote scientific papers, scientific posters, or clinical guidelines. After this, the platform will start recommending content based on the user's specific interests such as tracks or specific topics. If we go back to the top, we have the navigation bar which helps users find and filter content. The content can be filtered in various ways including content type, interests, events, etc. The search is also very quick and powerful and finds the most relevant content that the user searched for. Going to the collections, each collection has its own page which includes the title and the description as well as an auto play feature so that users can watch the entire collection in one go without any hassle. The events show all content from a specific conference or webinar and provide more information about the event itself. The video pages are sleek and have a great video player that rivals any video hub. Users can like or bookmark the video, share it or download the slides and browse through the recommendations down below or through the collection or event playlists on the side. Experts have their own profiles and users can check out their biographies and follow along with all of their videos. The platform also helps users claim credits for the content they watch with a short instructional video and credit watch time tracking. Lastly, in the user profile, users can see their watch history, bookmarks, likes and the experts they follow and personalize their content further. That is it for this quick demo. Thank you for watching and have a great day. Okay, so basically using this platform has increased our engagement substantially. Some examples are that the average engagement time per registered user is roughly 168 minutes. That's an increase from 90 minutes from our LMS. The number of videos watched, the average number per activated user is roughly 16.7. In the LMS, it was 11. As I said, the completion rate in our LMS, basically people were watching videos for as long as it took to get credit, whereas they're actually watching the full videos in this platform. And, you know, month over month, we are seeing an increase, a cumulative increase in learner engagement. There's an average of 46 minutes of watch time per physician, and they're logging in about two times a month and watching 16 minutes at a time. So that has led to an NPS score of roughly 35, which is considered great. The user feedback is overwhelmingly positive. It's great content, beautiful, easy to use. They really want a mobile app, which is why that's on the roadmap for next year. And because we put content in that wasn't intended for this platform, we just put in first what we had, we know that the content that we put in that's 25 minutes long is actually too long. So I'm now going to pass this off to Bozi. Can you hear me? I don't have this. All right. I've been told I'm too tall. I cannot reach the microphone. So I hope you don't mind. If there is an AI to reduce my accent, I will sign up. That would be great. I hope you understand me. Or to learn how to play basketball. I'm Serbian. I'm tall. No skills, no passion, no focus, no discipline, just height. All right. So I'll redirect to Nikola Jokic. So I'm just here in a couple of minutes. First of all, I'm super grateful to be here and to hear this story and working together in a really partnership-like manner to see the results that you've seen here. And we wanted to share as much as we could. You see the results. You see the numbers. This is how it is. This has been presented to ACIP's committees and education committees in a few days as well. But how do you generate revenue? So now that you consolidated content in all-in-one library and you turned on this engagement engine that has a binge-worthy experience and also a way to bring users back through curated content weekly with counter-recommendations, how do you generate revenue? Serbian wants to generate more revenue. So I'll first talk about how not to generate revenue. If you study Charlie Munger, he always says, invert, invert, invert. First see how not to do something. So first one is sell separately each conference content. Anyone here doing that? So offer separate content packages most? Yeah. So that creates a situation where users feel nickel and dime. It's hard for them to find content they're struggling. Please don't do that. Second one, please don't offer content for free. You're creating a behavior that everything digital should be free. It should not be free. The same folks are paying for Netflix, Spotify, maybe YouTube Premium, HBO, and anything else. Digital has a cost to capture, cost to host, host, host. You see the accent there. Host to stream, cost to stream, cost for API calls, cost for, you know, data processing for personalization. And that's real. And if you want to continue to serve them well, especially younger generations that are digital first, personalized experience only, you will need to invest in that quality. So if you give everything for free, it won't work. And the third one is, this is my favorite one, offer it in a very complex way. I recently have seen a chart designed by engineers about this, one of the associations. So first give it for 30 days to the annual meeting attendees. Then give it to members after 60 days. Then give it to Bojie after 90 days. And then to my grandma after 180 days. That doesn't work. It's complex. It creates so much work for your staff, and it creates so many issues for users. Like, why do that? Number four, promote sporadically. No one locks themselves up in a room at the annual meeting where there's a lot of new content and watches all the content. No one. It's just impossible. Even if you're the conference, there are 10 different things going on at the same time. So why is it then promoted sporadically? Simply because it's promoted around the annual meeting, and annual meeting is at the core, even though these doctors need access to this content for 12 months. And they will consume this content. Content doesn't get old after two months. We show the data. It's the same content, gets watched over and over. The way that they consume it is through a piecemeal manner, where you saw the thing for 17 minutes, 16 minutes per login, and 47 minutes per user per month. We're talking about busy emergency physician specialists. I'm an MD by training. They're very snappy in the way they think. Attention is low. It's 47 minutes per user per month. And the last one, and Susana talked about it, LMSs have been around for 20 years, established category. They are designed for compliant completion and long-form courses and scoring packages. They are not designed for engagement and subscription models. So there's one thing from this slide when we talk about monetization. Right? It is that if you consolidate as much as possible content in one unified library, that will be magical. That is a secret. That is a strategy to unlocking not one, not two, not three, five monetization paths. You don't need to unlock them immediately. You can do it successively. But that one decision to consolidate content unlocks monetization paths. Here is the most beautiful part. You will use the same content with the same staff size, fewer promotional emails, and you will generate way more revenue. So this is not more work for the staff. ASAP didn't hire more staff to deliver this really impactful platform. It's the same staff size. It's a really important consideration. So what are the five ways to monetize? I can talk about for an hour. This will leave some resources at the end. Individual subscriptions, we live in the world of subscriptions. We all subscribe to 15 different things. The next one is institutional subscriptions. Hospital systems, physician groups, pharma companies internationally, they want to bring, they're usually buying bulk for a number of doctors. So it's similar to individual subscription, but it's institutional. The number two and number three, very interesting because bundled with annual meeting, our partnership and other partnerships going really well with the American College of Cardiology, that's what they've done. They bundled access to the all-in-one Netflix of cardiology library with a lot of CME credits into the annual meeting. It's not optional. We call it the forced bundle. They increase the price of the annual meeting. Every user knew that when they are buying annual meeting registration, they will also have access for 12 months. They basically say free 12 months access to the Netflix of cardiology. Everyone wants that. Everyone expects that after the meeting and increase the price a little bit. So they immediately generated revenue from that. But what's really beautiful about that monetization path, you get a lot of users very quickly. And once you get a lot of users very quickly, the monetization path, number five, opens up. I just attended the session an hour ago about attracting industry sponsors year round. So the first question that the industry will ask is how many users are on your platform and how much time do they spend? And that traditionally have been very, very low. So they will not invest. They will go to Medscape to view Medi and spend money there after the annual meeting. But if you do have a lot of users with a lot of engagement, then sponsors are willing to spend money. One example is if you're selling, for example, industry product theaters, sponsored symposiums, so you're probably charging 30 to 50k per slot for the industry. So it's relatively easy to say, well, we also have amplification of this because these users will get access to this platform. They're engaging. Do you want to show product video, non-CME, clearly marked for three months and pay another 30, 40k for that? That's easy because, yeah, I want to be in front of that audience. That's their number one goal. The third one, bundle with membership, that's one of the interesting. There are some associations that have a lot of members, but smaller annual meetings. So this model would work really well. Bundle with membership and increase the price a little bit or keep the price the same. And there are some associations that struggle to deliver member value. Member value has been challenged, which is manifested either by an ability to acquire a lot of new members, especially Gen Z and millennials, we had a talk about it yesterday, or they're carrying churn, like lapsed members. So it's hurting them. And so if they make that membership more valuable and include it as a benefit, that will already have a monetary benefit because you improve acquisition and you decrease churn. And it's also really powerful. Now you can go very creative. You can offer this within certain membership, resident membership, or digital only membership, or create a new tier that has this. You can get very creative. And all of this is possible if you do the first step. Just consolidate, make it easy for end users, make it easy for your association. So those are the five ways that we discovered. And we are, again, we're implementing one by one with our clients, and we are learning constantly. I want to call at least one call a week where I talk to association to understand their situation so I can help. Let's say, what is the first one? What is the second one? What is the third one? Because every association is a little bit different, right? We published a few things on this and table that can help associations if they don't want to talk with us, so they can access this or share QR codes at the end. Just a few comments about the subscription. So we all, we live in the world of subscriptions. It's not just it's an amazing revenue model because it's stable, it's recurring, it drives retention, it has auto pay. Auto pay makes subscription magical. You're not constantly selling them something new, right? I heard American Speech Hearing Association last week, they had, they looked at everything that everyone was buying from a digital catalog, so they looked at their retention. 20% of people were buying again. Once they switch to subscription, they have 60% of people, so it's auto pay and subscription, like we're paying for Netflix and so what if I know everything else? But it's not just that. For end users, it's better. It's easy, it's all-in-one, it's convenient, it's a lot of CME credits. Pricing is a big topic. There are multiple ways to think about it, but if we get the right price value ratio, you can do a lot. I won't talk much about subscriptions. There was a time it was a blockbuster. You buy things one by one. There's Netflix. New York Times recently showed that the green part is now their major revenue driver. Also, there is a beautiful sales page with the benefits that emergency physician can go there and buy access for I think $300 or $500, depending on their member or non-member. Then all the channels, email, social, homepage, are promoting that subscription. Lastly, this is just the first step. We launched this together with ASEP in February. And we are in phase one right now. We centralized and organized content. We set up the engagement engine. Doctors are watching. They're coming back and watching. That works. We set their first monetization pathway, which is working really well. But that's just the beginning. Because, for example, and Susana can tell you about CDEM, is a product called Critical Decision in Emergency Medicine. Because they saw that a lot of users want to be on this platform, they started to add other products that they had before. We are talking about exclusive content. We are expanding monetization pathways. And then in phase three, we're doing everything, basically, engagement, satisfaction, and unlocking all monetization paths. But I would like to make one point. I've been listening to different sessions at this conference. I would argue that an ability to turn your existing, trusted, fresh content into a year-round engagement is the most highest leverage activity you can do for associations to drive new revenue, engage users, make sure that they continue to stay members. Number one reason for stopping membership or not renewing is lack of engagement with the association throughout the year. And also being able to attract generations of Gen Z and millennials, which want everything to be personalized, beautiful, easy to use, and digital-first education. So I would argue that this ability to turn fresh, trusted content into engagement, it solves many, many, many, many challenges that I heard during this conference. I won't talk too much about raw and technical, but there is one key takeaway from what Susanne and I have been talking about, which is this. If you have a content that you feel or know that it's underutilized, conference recordings, webinar recordings, videos, podcasts, then that content can become new revenue stream or multiple streams, source of engagement, digital satisfaction, and continuing to stay relevant to the younger generations of doctors. Thank you. And I will open up for questions. There are some QR codes. Thanks. So this is a QR code to connect on LinkedIn with us if you want to see five ways to monetize or slides from this talk. A lot of people ask about the slides. Please scan it and you'll have it. So thank you. Thank you. We have about five minutes before we transition to lunch where we can continue the conversation. So as people come to the mic, I'd be happy to ask the first question. It's actually for Leah. We also use 11 Labs for voice cloning, and we have a small amount of our med ed faculty that have donated their voices, myself included. And so I had AI Miguel and Real Miguel test for my own family, so my wife and then my son. And my son picked up right away on my inflection and my pausing. My wife couldn't tell the difference between the two, which proves that my wife does not listen to me. So AI helps at home as well. So my real question is this. One thing we ran into was AI doesn't necessarily know medical terminology. So we've had to create a whole library of phonetic spellings of drugs and conditions and things to train it up before we start doing the audio enhancements that we do for a lot of our online learning center things. And the second question related to that is about translation. There is now a way to do real live translation to any language you choose at your meetings, if you wanted to expand your audience say to another country. So we definitely are recognizing the challenges with pronunciation of medical terms. And one of the challenges, because we wanted to try to use it, we're not at the step yet of translating into other languages. But in terms of the sort of live capture, there have been too many errors in terms of key things that it just really kind of undoes the kind of extra benefit because it's distracting. And I think that also goes to the point of why we, no matter how sophisticated these are, you always need a human element to correct these things and catch. Because there was even one case where something was very stigmatizing. We're like, oh, wow, okay, that definitely cannot be used. So it's still a work in progress, but I'm impressed that your son was still able to catch it. Which one was you? Yeah, right. So the mic on the left here, if you'd like to go ahead, please. Hi, Erica Flynn, American Society of Colon and Rectal Surgeons. Quick two questions. You mentioned, you know, kind of looking at the current content you have, making it shorter, so it fits in this 25 minute or less interval. Do you have content that just doesn't fit into that box? And what are you doing with that? Like, is it somewhere else on another platform, not accessed anymore? Or what are you doing with that extra stuff? I mean, we really have taken most of our content is 60 minutes or less. So we have very little that's over that. But ultimately, we just split it up into parts. So we haven't left, like, content somewhere else. So we just, you know, if it's an hour and a half or whatever, then maybe we've split it up into multiple parts of 15 minutes and made a collection of it, and then just put it into the platform that way, rather than leaving it somewhere else. Yeah, just to add to that. So we can take a data-driven approach. So we now have the data, what's the average length of the session when someone logs in? It's like between 15 and 20 minutes. And also a lot of conferences are organized in sessions, and sessions have individual presentations. So it's usually about 10 to 15 minutes. So that's a fast, easy way for conference recordings to do that. But definitely repurpose, shorten, you know, at least chapterize. If you cannot cut one hour webinar, because it's one speaker, it's harder to know where to cut, then the chapterizing with the little dots, you know, on the player that at least helps user to know what parts of the webinar to go into, yeah. And then one other question about CME with the attestation. You mentioned that, you know, they can print out certificates. Are you also tracking that internally, like to make sure they're not watching the same thing twice or duplicating? Or, you know, in some specialties, certificates are kind of obsolete, and it's up to the, you know, CME provider to send the information on through the ACCME system. So just curious. So we're not policing what our users are doing. It's basically an honor system to make sure that they are not double dipping or watching repeated content. But we do have a CME tracker that can, like, basically print out everything that they've done. And so if a certificate's not good enough, there is, like, a way to get the data and to submit that instead. Cool. Thank you. Thank you. And we are at time. We will transition to the third floor King Arthur room for lunch, where we continue the conversation. Please join me in thanking our presenters.

medical education

artificial intelligence

ACP

AI integration

Descript

CapCut

multimedia library

personalized learning

content monetization

ethical considerations