All right. Hello, everyone. Thank you for joining us this morning. My name is Devon Courtright. I am Deputy Chief Officer of Learning and Educational Research at the American Society for Clinical Pathology. I have the pleasure of introducing this session on Driving Meaningful Change Through Education, Education Evaluation, and Mission to Meet Learner Needs. We'll kick it off first with a presentation on education evaluation by Chris Presta, who is the Director of Digital Learning at the American Academy of Dermatology. We'll then separately flow into conversation with our two panelists, Dr. Vince Lafredo. Come on up, Vince. Vice President of Continuing Professional Development at the American Academy of Family Physicians and Ann Grupe, Director of Continuing Education at the American Society of Clinical Oncology. We'll hear their perspectives and other thoughts on this. Chris, I'll go ahead and hand it over to you to get us started. You can leave it on. I guess I'll use it. Thank you, Devon. I appreciate that. Good morning. Good morning. How's everyone doing? Okay. Well, welcome. So, as Devon said, we're going to be talking today about Driving Meaningful Change Through Evaluation. Let me see if I can work this. Okay. And followed by a panel. So, all of the questions we'll hold until we have our panel discussion. None of us have any relationships to disclose. And here's our learning objectives. I will not read them to you, but I'll give you a second to look at the slide. And we'll get started. So, before I go into the meat of the presentation, I just want to acknowledge that all the work I'm going to talk about in this presentation is the result of a team. So, this is the evaluation team from the education department. There's 14 members of the department, and we had six of us who worked together, are working together on this project. It is a team effort, as well as stakeholders throughout the organization, which I'll also talk about. But I wanted to acknowledge my colleagues who couldn't be here today, but this work wouldn't be happening without them. So, what is a holistic evaluation? We've been saying it's a program evaluation, and we all know that term from our ACCME work. I'm sure all of us are accredited providers, and we do program evaluation as part of our accreditation. We look at outcomes. We look at our delivery mechanisms. We look at our procedures, and we look at, you know, what's working and what's not. That is all part of what we do every day. This is different in the sense that it's taking a step back, looking at that 30,000-foot view, and not just what's happening with our own program, but what's happening outside of our program, what's happening in a competitive analysis. And so, the dilemma we wanted to address is the changing landscape of medical education. We wanted to keep up with the times. Learning formats are changing. I'm sorry, learning preferences are changing for our learners. Trends in member demographics. We're seeing members not wanting to go into solo practice, administrative burdens and costs. They're going into group practices. It's affecting their learning patterns and how often they're attending conferences and so forth. Increasing costs. We're seeing increasing costs when we deliver education. They're seeing increasing costs in their practices. We see competition. There's more organizations and for-profit companies wanting to provide education to our learners. And then we have new technology like AI. How is that going to affect the way we deliver education? We wanted to take all of these concepts, examine them, and integrate them into the big picture. How does this inform our strategy going forward? Then we organized these concepts into three buckets. What is the gold standard in education for dermatology? Are we meeting that standard? And then what should we focus on going forward? One of our members said this at a conference and we lifted it. Now it's become a mantra for us. Don't get mad, get data. We're going to get data. It started with a business case. When we started thinking about writing a business case to conduct an evaluation, to get buy-in across the organization, we thought about how is this work going to support the organization? How does it link to the strategic plan? The AED's vision, I don't know if I can see this from where I'm sitting, but the AED's vision is to ensure healthy skin and end suffering for all. We want to advance excellence in dermatology. And then our core values, patient first, member centric, lifelong learning, visionary leadership, professionalism, scientifically founded, thank you, and unity. This is what happens when you get older and you need glasses to see far and you don't need glasses to see close. Anyway, so those are the core values of the organization. It all links to the pillar of designing and delivering the gold standard of education. We made sure when we were talking about our business case that we weaved these messages throughout our work, that we were going to be doing work in this evaluation that was going to connect back to the organizational strategy that supported the pillar of gold standard in education and the organization's vision, mission, and values. Next we thought, what's the goal of the business case? What's the purpose of the project and what are the objectives of the project? So the goal of the business case was to provide a rationale for why we were doing this work. We wanted to be captivating, both written and verbally when we talked about it, when we were writing the business case, because we wanted people to feel compelled to support it. We wanted to demonstrate the value, again, the value of this project at the organizational level, why we'd be making an investment in this work to help the AD advance its mission. And then we talked about the purpose of the project. We were going to establish baseline data, enhance alignment of the education program with the AD strategic plan. We wanted to gain understanding of our members, understanding of and feedback on our program. We wanted to assess the competitive landscape and then demonstrated our commitment to continuous quality improvement in education. And then the ultimate objectives were to get feedback on the content, diversity, modalities of the program, again, understanding our members' experience, and we had a part of the plan to get a SWOT analysis, really needing to understand the competitive landscape, our opportunities, threats, as well as our own strengths and weaknesses. And then we said, okay, now it's time to bring in other people. How do we sell this to others and gain support across the organization? So we started engaging stakeholders. So that work really started with our own department, talking about this at our department meetings, talking to the team about what they hoped to gain out of this. What do they want to know? What do they know about the programs that they worked on? What was their input? Really having them, so if they were asked about it, they can talk about it passionately as well and having their support. Then it was brought to AD leadership. Our vice president talked about it very openly at her meetings with our executive director and the executive team. And then we brought in the chair of our council. We needed a champion who was a member who was going to also then help us get enthusiasm from the Council on Education and ultimately the board of directors. And then, of course, you have to navigate the politics. There's always going to be naysayers, negativity, people who maybe don't understand why this would be important work or think that we need to allocate resources. We were bringing in a third-party vendor. There was questions as to why we would need to do that. I will say we didn't have overwhelming resistance to this. There was a lot of enthusiasm for it, but we still needed to be open to hearing all the feedback, positive, negative. All of that gave us a clear understanding of what our current state was, and we used it to help build our business case. And the other piece we did at this step was thinking about our stakeholders on how we were going to give feedback. How were we going to continue to engage them throughout the process? So we were always going to start with our own department, our Council on Education, our leadership, and then the board of directors. Other items to consider were the budget. How much money were we going to ask for to have allocated to this project, making sure that that ask fit in with fiscal year planning? And then what were we looking for in a third-party vendor to help us? We wanted a credentialed evaluator who had the knowledge and skills to conduct the work. So our timeline. All this started in the summer of 2022. We worked on the business case. We started the work in engaging stakeholders and talking about it, deciding what we wanted for a budget. By the end of the fall, we had the budget approved, and we hit the ground running in January with our RFP and vendor selection. Then the work was happening throughout the summer, and we're now in the middle of the So next we identified the actual deliverables in the business case. The vendor was going to deliver a qualitative and quantitative analysis, a value proposition. This was very important to us that we have a statement that clearly defines the value proposition of the AD in the education space for our members. but our vision, mission, and core values, because all of that supports designing and delivering the gold standard in education. And again, this work wasn't just. work that's done in advocacy. So yes, while we're the education department and we're responsible for the certified education we provide, we also partner with stakeholders across the organization on delivering non-CME activities that are all part of the education portfolio. So the work truly supports the organization across many areas. So this is, I did upload the slides, I don't know how CMSS is distributing them and I can give you my email if you want me to email these so you don't have to, you don't have to read a checklist. But I do have a checklist of what we included in our business case. Some things I didn't talk about but are on here are that we included legal considerations. And the other main thing here is that having a defined business case basically wrote our RFP. So when we were ready to go to do the request for proposal to vendors, the business case was the foundation for that. So a lot of the work had been done. So the vendor selection process, as I said, started in January. We had seven vendors, we identified three expressed interest, and then we developed a scoring rubric and that project team I showed at the beginning, they, I'll use the rubric to score the proposals. And we selected a vendor in April. The vendor we're working with is a credential evaluator, I hadn't heard of that until we started working with this vendor. And they specialize in evaluation of continuing education. So here's the proposal, the scoring rubric. It may be hard to see on the screen, but basically we established all the criteria we wanted to look at, the company background, how well they understood our proposal, their approach to the project, the timeline, of course the price. We also included a place for comments. We collected that information and we consolidated it and then we met as a team and there was one clear standout, which is the vendor we're working with. I really liked this process, it gave us a way to look at everything very objectively and not just, oh we know this person, we've worked with them before, and it wasn't clear by just looking at them that we were gonna go with somebody. This really made us look at it critically and it got input also from the entire team anonymously, so it wasn't like, oh the vice president really wants us to use this one company so that's who we're gonna use. So it really also helped with that engagement. So this was another important step. Defining the scope, this was something we did with the vendor. What was gonna be included in this work? And the vendor really helped us hone this in. When they looked at what we wanted to do, what were we gonna actually include in the analysis? So we said with them, we were gonna analyze existing data. So we sent valuation data, demographic data, our self-study, I mean we sent a lot of data. There's a long list of files they were sent. They were gonna gather additional data through surveys, interviews, and a competitive analysis. And then the SWOT sessions. So they would also do this SWOT analysis and we would have a session with stakeholders to go over that together. And then finally it would result in an actionable report. And just as important was what was not in scope. They were clear that these were things they would not be providing. And this I felt was an important step because for staff members, especially in the department, who thought oh, maybe I can find out more about my project, it made it clear from the beginning, don't be disappointed because we're not taking a deep dive into one offering. We're not taking a deep dive into a single, a subset of data on a certain subset of learners. And we're not doing an evaluation of our processes. We're not gonna see if we should improve the way that we write self-assessment content. And so it kept everyone very much on track and what to expect at the end. So again, our timeline, we had given the vendor all of our data in July to August. And then August to October, they conducted the SWOT. They had interviews, we had an online survey. And that was done through our Learning Management Center. I do have some results to share with you. And the vendor conducted an environmental scan and they are preparing the report now. I can't wait to see it. We're a few weeks away. And then again, engaging stakeholders. So this was a big part of the interview process in identifying who the stakeholders are that we wanted to have interviewed. So we thought about, well, who impacts our work? Who has influence over our work? And then who has an interest in the conclusion of the work? Successful or unsuccessful? So this is a list of the representation we had. A lot of departments across the academy from education, meetings, marketing, membership, science and quality, development and revenue, the executive team and our CEO, as well as our council, other committees under the council, members of our board. We had some industry partners we talked to, and then the most important stakeholder, our learners. We really needed to know what they thought. And again, feedback. Making sure that as we were engaging those stakeholders, we were giving them feedback. We were telling them, giving status updates. It wasn't just ask them something, and then they never heard from us again. How are we using their feedback, engaging them in the process? Even with members, they haven't heard yet how we're using it, but we are talking to communications. This is a chart that shows who was interviewed. So we had board members, some department directors, members of the education department, two industry partners, five members from the council and or a council committee, and five members of the executive team. And one of the things the members were asked was, what would you describe as a gold standard? And so here's the word cloud showing how they responded to that. So evidence-based is the one that came up most consistently. Current, engaging, proven. So we really need to understand when you think of gold standard, what is it that you think of when you hear that term? Because that's what we're aspiring to be. We want to be the gold standards. We need to understand how that's interpreted. And are we? Well, 61% said, yep, you are doing it. You are the gold standard, yay D. But they were recognized for excellence. They said that when seeking high quality education, they think of the AD. They did note we're not the only ones. There's other societies and organizations providing gold standard. But we were meeting their expectation. So the ones who responded no, the note was from our vendor was that it didn't mean we weren't doing quality education. Those people who said that didn't say that we were at the bottom. They just said you have room for growth, which is what we were exactly wanting to get from this process. There's more you can be doing to meet what I would say is gold standard. Can improve our experience at events. You can make the education more engaging. And it was also noted that this term is very subjective. At interviews, most people said that we're meeting their needs and that our content's up to date and timely. We have something called question of the week, which is sort of self-explanatory, but it's a case-based self-assessment activity that goes out every Thursday. We started in 2018 and members just love it. It's quick, it's easy. They get CME credit. They get credit for the certifying board. And it has remained very popular. And some opportunities that came up was practice management, especially for young physicians, which we do a lot in practice management. But I think there's more anxiety around that area for those entering practice and that we could do more. And then also content selection for annual meeting are large events. I think what came back from this is people wanna present at those meetings and they don't know how to get their foot in the door. So how do we make that process more transparent? How do we create those opportunities for them? How do we help them develop and grow to become thought leaders in dermatology? And then the online survey, it was in our learning management center. We had it open for three weeks. Only members could see it. Went to about 2,800 respondents. And 88% said we were very or extremely valuable to them in our opportunities. And 85% said we were meeting their needs. So I think part of our long-term strategies are gonna try and get those moderate and somewhats up into those higher brackets. And then important qualities. This I may need to look at my notes. But the important qualities when choosing live education was appropriately priced, convenience, and the opportunity to earn credit. And then I just noted that when you look at digital, we didn't ask, the green is events and the blue is digital. When you ask for digital, we didn't have location and convenience because they weren't applicable. And so when you look at that, those drop off. So credit and up-to-date information and appropriate price were the top three for digital. I think I got that from memory. Okay. So overall, our feedback, the positive feedback is that there's a lot of respect for the organization. The quality of our program was recognized. The lot of support across the organization, from the board, from the executive leadership, and members for the program. And we have strong brand recognition. Opportunities include more engagement in events, digital learning, how do we make the content and the experience more engaging, looking at more technology to use at in-person events, keeping things modern, fresh. And then, again, how do we help create those opportunities for younger members to break in and get a speaking spot at something like the annual meeting and growing those future leaders. And then pricing structure, there was a lot of feedback on cost being too high. And so that's surprising, right? Every time you ask about cost, everyone's gonna say it's too high. But I think some of the red flags there weren't necessarily that we were charging too much, but maybe we're not communicating the value of the education and why the prices are what they are. Yeah. So our next step is continuing the analysis. We're gonna get the final recommendations in. We'll continue to update our stakeholders. We're gonna continue to try to speak about our work at events like this. We plan to write a manuscript and try to get the work published. But internally, we have a meeting. And then my key takeaways. Do the business case. It was a little time consuming, but so worth it. It made our job easier when we were talking about it. It put us all on the same page. And when you're doing it, make the why so apparent that not doing it is seen as more risky than doing it. Engage your stakeholders internally and externally. But I think a key part of that isn't just engaging them, it's keeping that constant communication with them, giving them the feedback. What's happening with the work? How are their contributions impacting the work? And then again, use data. Data, data, data, right? Use your data to drive strategy. So I am happy to now move to, let's see how I was on time. I timed myself at 28 minutes. Did I get about there? Okay. So now we're gonna move to our panel discussion. I'll be joined by Anne and Vince to take your questions. Thank you so much. Thank you. We might have moved to the round in the front, but I think we need to stay because it's being recorded. We need to stay here, right? Okay. Yeah. So Chris, thank you so much for that. It was great to have that as an example for us and a model to follow. Just some opening thoughts as you guys are thinking about your questions. The first is that we wanted to design this session to be actionable for you. So we've got a. that you have as your What I would say to add to your key takeaways is that consensus around definition is so important for these evaluations. education. I can say when we did ASCO's strategic planning here's the way they're defining education. Be the trusted source for timely and high impact evidence and continuous learning. You know what's not in there? CME. It's you know not that we don't have a CME portfolio but that's now become one slice of a much larger pie within our organization. We're engaging multiple departments. We're coordinating how we are delivering education. So you know there's that old adage if you build it they will come. I feel like our annual meetings you know kind of lean on that. But now it's not just if you build it they will come. It's build it and take it to them. Right? So you need to think about meeting our learners where they are and our learners are changing. You know I'm gonna lean on Vince for his yeah his generational comments that I know he's gonna make. But our organizations are learning and changing. We've seen this already as a result of the pandemic but it's not just as a result of the pandemic. Right? So be thinking about how you start a conversation around getting consensus on how So what does that mean? It means also that we have putting the same efforts into non-accredited programs so that they understand ACCME accreditation is not the only way to have quality education. Because I think it's really important. A lot of people at the dermatology conference don't claim credit. They are there to, quite frankly, bond. I've never seen so many people in the hallway. But they see if this is accredited, this has value. It's not all societies that have that. When I worked at the Society of Thoracic Surgeons, I took credit away from a conference and people recognized because we made sure they were clear. We haven't changed the rules other than industry can talk. Our members are saying that our education needs to be accredited for CME, the opportunity needs to be there, but both third claim credit. And then the other thing be. So it's very interesting but we don't see a deviation from that. Maybe a small regional course will get half but it is a third. So Damon, your recollection is accurate. One thing I can put out there as a potential challenge though is to be very out front and explicit when you're not giving credit. So we have a meeting that we call Breakthrough that we are very deliberately not accrediting and the reason why is that we wanted to design a meeting that created an intersection between oncology and medical technology. And so we wanted to have a meeting where we could invite industry partners to talk about the future of oncology and have a dialogue with our experts on the clinical side. And so yeah we were very explicit that we are not designating this for credit because we want to have these people at the table to have these conversations and to your point Damon, we are using the exact same educational processes and we have the same degree of rigor and our learner feedback evaluations are telling us that that meeting is not promotional. We are meeting our same educational metrics with that meeting in terms of learner outcomes etc. And in fact we updated our COI So they rely on us. Hi, so come from the technology partner side and we see a lot of data on what doctors watch and don't watch so we'll help the ACC launch the Netflix of cardiology, which was mentioned yesterday twice at a panel and What we see is this trend we call it personalized on-demand Microlearning and those three words are really important because personalized is if you watch this you may want to watch that Why is that important because there's a lot of content and overwhelm is a problem So if you put the content on a linear learning platform like classic LMS, yeah, doctors Doctors don't know what to watch next Independent of seeing me or not seeing me this platform that I'm talking about it has seen me But personalization like we all experience if we go now Netflix all of us will be shown different things on our home page Because it's personalized It's a huge huge huge word and everyone Expects it's nowadays. So we see this in the data. It's clear. Second one is on-demand, which is really convenience anytime anywhere I mean again all of us do the same I think and the third one is micro learning We found in our data and we see a lot of data but people watch is that it's 5 to 15 minutes is the time they spend and Watch something and maybe there's a lecture from a conference because it's all in one kind of experience all conferences they watch 550 minutes they continue with their life and So I would like to add to that one point which is you mentioned if you build it they will come We see that quite often like we see that the biggest missed opportunities. Hey, we have thousand videos be recorded from the conference Here is a link to watch it free or $500 or whatever But they will not come here is what we see in the data If you do that, they come once or twice total spend time average 15 to 20 minutes and they never come back Right, so what we had is a layer again Netflix is just a good analogy. We call it active content curation So if there's thousands of videos and doctors watch 550 minutes of their time They're busy. The question is how do you allow them to watch over time and digest that content over the weeks in a non-obtrusive way? So Netflix are the same so we would send once a week email that says hey, dr Smith in a personalized way, you may want to watch this content next So when they're on the page persons when they leave and and we've seen that work so well I'm about dr I'm about shared four hours time spent in the first seven average busy cardiologists, right? Because of personalized on-demand micro learning combined with active content curation. So I hope it's useful for the group I had so excited about these things because it's for me. It's an example how technology Amplifies education in the most beautiful way. So just wanted to share this because I know the data we were aligned because I think People know me know that that that's one of the things that I that I advocate very much is to one Using Netflix and we can use So, we see this active learning as a big part of it. One of the things we have done is take our education and make it 365. When you go to a national conference and you sit there and you say, hey, thanks for coming, we'll see you again next year in wherever. There's a lot of things that happen between, you know, in that 364 days. So, having that active learning. their medical school, in residency. team figure out what the modality is, if that modality is to package it or put it in a subscription or whatever. But the education content library does exactly what you're talking about. It allows us to put content in micro and even nano level, because we're, with our LMS, pulling out slide 27, 28 of a 78 slide deck that allows us to pull that out when they do their search. And that goes back to your taxonomy and your ontology. is that going back to live courses like natural courses in our research that we've been doing has come back not to the level that it was but also the on-demand products have gone up so what we see is a very Yeah so I would actually argue that needs assessment now is no longer just about the what but it's about the how and the when. So you know if you're thinking about any of our disciplines they are only getting more complex. There's more and more information about our different disease areas. I mean every month in some cases and so our members need help navigating that. That's a learner need. It's not just about you know keeping up with the clinical competence but it's even understanding what that looks like anymore. So you know part of what we're doing within ASCO it's kind of similar to what you're doing. We're calling it the ASCO content experience because we're recognizing that our learners don't necessarily even know what we have and so it's how do you connect the meeting content with the journals, with the guidelines, with the the career development resources we have and create this more holistic picture of what it means to be a clinician in this current time we're in. And so yeah I mean I your point 1,000% and it's also to your point you know it if you build that engagement that also creates a win for our organizations that are looking to create that opportunity for interactions with our members for the full 365 days rather than just seeing our societies as the you know three to five days One other thing to add to that too, one of the things that we do is we do leverage. and use some content marketing to it. So in other words, give them that snippet, give them that little piece of micro nano-level learning to say, here's something we're gonna tell you, if you wanna know more or you wanna see more on this, come, here's a link to where you can get that information on our site. So that's how we're leveraging social media to bring people to the nano-level learning and keep a constant relationship with our learners. I don't want to talk about it. So because you said social media, so it reminds me. But one other thing, and I don't know if everyone feels it, I have this unique perspective because I talk to multiple societies, so it's a different way. But oftentimes, they tell me, hey, when it comes to on-demand conference content, we have all this new, but members feel nickel and dimed. I heard that so many times. Why? Because they're being sold one by one, a library, which is linear. There is no personalization, no AI-powered algorithm. And so they struggle because they don't only go to a society's website for content. They go Medscape, they go Doximity, they go other places. So when they go to societies, they would prefer all-in-one, like subscription-based, all-in-one library, last three years of content, number of conferences with the algorithm helping them find. So just by putting that together and cutting these sessions into individual presentations, because most of the associations, the length of their presentation is 10 to 15 minutes. So that fits into what I said, 5 to 15 minutes. So then you have all that. It's cut into individual presentations. Suddenly, it's all-in-one. And then the content experience matters. And then that makes it so much possible to do what you said about social media, which I love. Because then you don't actually need to create any new content. Because you know what are your top 50 lectures from a recent conference. And if you create 5 to 30 seconds, because usually Twitter, LinkedIn, want 15 to 30 seconds video, you basically take the best part. Maybe that's a slide that shows efficacy of a new drug. Take those famous professor XYZ, efficacy data for new trial XYZ. And that is a screenshot or a short video with two lines of copy, just to tease people. And then they click, and then they want to go and get access to a subscription-based library. The beautiful thing, no creation of new content. It's just putting it together in all-in-one, better delivery, AI-powered. And again, I don't want to just dominate. But hopefully this gives you some ideas, because this is what I spent all my day. And come to our booth if you want. We're down there. But really, I just want to be of help. I learned so much over the last years. And I work in pharma. I'm MD by training. So just feel free. I could share much more. No, that's good. Because you bring up some good points. We did a really in-depth research with our membership. We used a company called Geostrategies to do this. And what we found out was, you talked about nickel and diming. Our members had roughly about $5,000 pre-pandemic, on average, to spend on their continuing education. Post-pandemic, we found that it's at $2,400 that they have from their employer. 80% of the family physicians are employed. They have $2,400, on average, coming from their employer. And they're willing to spend $600 out of pocket on education. That's still far less than the $5,000 they had, coming from the employer previously, plus whatever else they were going to add. So we see this shift in, hey, don't nickel and dime me. Give it to me all in one space. And we have a new LMS that's coming out. And in our LMS, our goal, our end goal, we're not going to be, it's going to take us a few years, but to get there, our end goal is to have just two forms of education that are out there, on the ground, face-to-face, and just a few of those, very targeted. We used to have 20. We're down to six. And everything else be in our subscription, our online, on-demand, our live stream, everything else. Again, take a look at, we're going to go back to some of our social media things, look at Prime or Apple. Everybody else is putting football games now, right? Thursday Night Football is on Prime. You have to go there. You have to be a subscriber to get in to get Thursday Night Football. What are we teaching our consumers to do? We're teaching them to go to these subscription models to get what they want, live and on-demand. And that's the way everything's going. And we have to replicate that in how we consume, in a lot of ways, that to our members. Let me just add, especially if you want to attract new members. Because new members, that's how they consume. I'm glad we brought up pricing, because that's been back here on my mind. We all have revenue goals, right? We have to put that hat on. At the same time, I feel CME, we're the source. We're the gold standard for CME. However, we also hear, well, there's a lot of free CME out there. So how do you compete with that in terms of, as you said, budgets have dropped. So that makes that more appealing. Just some insight or guidance on messaging. You can say you're the gold standard, but I welcome feedback on that. How do you compete against free? It's very, very difficult. And you have to be smart about it. I can give you an example. We lost a lot of money last year, flat out. And our big packages selling for $900 aren't selling at the level they were. We did a triage program, which is we created five CME activities over the course of six months, releasing them every month. So that was our triage bag. It's come up with 30 things in six months. Crazy, crazy amount of work for us to do. Those units are up way over where we were. Our net is down. Why? Because they were cheaper. So people are buying more of the one-offs, and they didn't even touch the stuff that we had that was more expensive. So you have to find that right pricing now. And you have to give it to them in smaller amounts. And that's why our long-range goal is to get to a subscription. Because if we actually can make more money, if we're, right now, our course that we offer is $900. If we sat there, and what I've talked to our marketing team about, is we charge $100 a month, we're at $1,200, and give them everything, our current, it would put us, even if we had a third, and we have 130,000 active members, even if we took 20,000 members, it would put us at almost more than double what we're making right now. And they see value in getting more for less. Because instead of paying $900 for one course, I get $1,200 to everything. So let's go. Well I did have a do you mind if I insert a comment. I have no doubt this is where things are going and I have no doubt that it's going to increase engagement. But as I commented yesterday I worry that atomized education it might actually diminish learning. But I think you put your finger on I think the key thing that these societies can offer that is really helpful to learners which is the curation and active curation and and that is something they can't get on Medscape or Doximity or any of the social learning platforms. So I really see that's the distinct value that the societies offer and related to that I want to. And this is really a question for you and because your content the content evolves so quickly in your field. How do you retire items from your library that may have become outdated with emerging evidence. Do you have a mechanism for dealing with that issue. Yeah so you know we are yeah content creation is a huge thing for us but we you know I think with. communities of practice. We're launching virtual. Yeah, I just had a follow-up observation. When you were talking about the subscription model, I put my consumer hat on. And how many streaming platforms do I subscribe to? And I still sit there at night, and I go through all of them and can't find something to watch. I'm sure I'm not the only one, right? It's like you're on Prime and Apple+. And so it just makes me wonder, as I think about the subscription model, which is really intriguing, is maybe they don't find 10 things, but maybe they scroll and they find the one that's just enough to keep them going. Because as a consumer, it keeps me going. I still keep Netflix. So I think that's an interesting model to go to. And let me just add on that, too. Our data shows, because, again, we asked our members this question, as an individual, they're willing to purchase it. So the individuals, but we got to, so we really wanted to get into institutional sales. It's something that would not work with an institution. So we go to the hospital and say, is this a subscription you would want to buy? The institution says no. Why? Because they can't choose just family medicine. They need everybody's. So they don't want one. They want a subscription that'll cover everything. And so the institutions aren't looking for the subscription at the moment, because there's not that one that covers everybody. So they leave it up to the individuals. But the individuals overwhelmingly want it for that one purpose that you're talking about. And Tom, just to answer your question, too, we go through the accreditation cycle to begin with, right? So in terms of what we keep on that, we are going to have an archive, so that when we update the data, there is going to be an archive. If people want to see what were the standards or what were the policies previously, the guidelines previously, they can go back and look at it. So we're going to keep the first and second editions in our archive, but the most updated stuff. And then we're also going to be tagging it for when did we post it up there. And we're going to be having review periods. So everything, every month we look at what just, again, to pick on the Netflix model. What does Netflix do when you see it? What's coming, what's going, right? You see that every month. Here are the things that are leaving Netflix. Here are the things that are coming. So we have thought about that. And we have thought about creating and tagging with a time date stamp of, and asking our staff who are helping coordinate it with the faculty, what's the appropriate time? Is it six months to review? Is it a year to review? Is it two years to review? What is that? And then leveraging that to tag it so that we go back and review and say, hey, is this stuff still relevant? Is there new things to have? Or we may know that a guideline's coming out in six months. So we're going to tag it and say, hey, we need to review this because that guideline is coming up six months next year, whatever it might be. Yeah. We unfortunately had to start late because everything got late from the other way. So I know there's a little bit of a break. So I'll pass it down and see. Any other questions? Any other questions or parting words from the panelists? We'll be here. Did it help? Something to take away? Yeah. Thanks. Excellent. Yeah, appreciate it. Thank you so much.

education evaluation

medical education

changing landscape

learning preferences

gold standard

dermatology

areas for improvement

engaging stakeholders

high-quality education