Alright, welcome everyone to today's webinar. I'm Helen Burstyn, the CEO of the Council of Medical Specialty Societies, and we're really delighted to have a chance to share with you some of the work and the results from our joint PCORI project with the Patient-Led Research Collaborative on the promise of patient-led research integration into clinical registries and research. We're going to talk about two of the scorecards today, research organization readiness and integration into the research process. We are trying to figure out how to add captions to the webinar today, but we promise the recording will include captions, so apologies to anybody who is expecting them today. Our bad. We will make sure that gets fixed in the future. If we can go to the next slide, please. Excellent. Just wanted to point out this is work we're really grateful for funding from the Patient-Centered Outcomes Research Institute, PCORI, as part of their Eugene Washington Patient Engagement Awards. This work is solely our work. It doesn't represent the views of PCORI, but we're really grateful for them for taking on this project and allowing us to have funding for this really most unique collaboration. Next slide. Just want to briefly give you an overview of the agenda for today. We'll do some quick introductions, a little bit about project overview and goals, and then really we'll spend the most time today talking about the review of the scorecards in general, and then take a deep dive into two of the scorecards around the research process in particular. We'll have an opportunity for questions and answers, and then some good opportunity for discussion that'll be moderated by Susanna Fox, who's a member of our advisory panel. She will serve as an ombudsman today. She'll keep an eye on the chat, interrupt the flow if there's a really burning question and make sure we get to as many of your questions today as we can. Next slide. Again, encourage your participation. Feel free to put your questions, your comments, your ideas in the chat throughout the webinar. If you have an immediate question, you can raise your hand. Susanna and Liz will moderate whether we need to stop the flow or kind of hold that for the end. Next, please. Just want to introduce the team. I'm really delighted to introduce the joint team here today. You'll hear from a couple of the folks from the patient-led research collaborative team today, including Gina Assoff, Hannah Davis, Lisa McCorkle, Aneesha Sekhar, and Hannah Wei. Gina and Hannah Wei will be presenting on today's scorecards. Yes, the slides will be available after the webinar. Thanks. And then from the CMSS side, myself, Liz O'Keefe, and Suzanne Pope. Next slide. We are really grateful to have an extraordinary advisory panel, as you will see listed here. More than half of the members came directly from the patient community, as well as people who really cross those lines beautifully, folks like Susanna, of course. But as you'll see here, people who are very engaged in clinical research, clinical registries, as well as those who have been in this game for a long time, thinking about how to effectively engage meaningful collaboration for patients into clinical research, really moving beyond the token patient on a committee, which was really our goal from the starting point here. Next slide. And as I mentioned, Susanna Fox will be our ombudsman for today from our advisory committee, and I'll turn it to Suzanne just to review the goals of the overall project before we pass it on to the PLRC team to walk through the scorecards. Thanks. Sure. Wonderful. Thank you, Helen. Yes, so our goals for this project were obviously to support patient-led outcomes research, very important to us, and we had a lot of key aims that we wanted to focus on. So we really were interested in incorporating patient-led research into patient-generated outcomes research, and really into the comparative effectiveness research. Very much there was an emphasis on collaboration with diverse patient populations. We wanted to make sure that those diverse patients were represented and that we heard their voices, and we really were able to do that much through our advisory panel. They really helped us connect with different voices and make sure that those voices were obviously well represented. We also wanted to make sure that it was patients leading comparative effectiveness research efforts, and really emphasis on leading, and you'll hear that throughout our presentation today. And then we wanted to focus on pathways for sustaining this, you know, not just a one-and-done, but really how can this be sustained? How can patient-reported outcomes and comparative effectiveness research data really be sustained throughout the efforts of specialty societies, registry teams, research agencies, you know, academic research? And then finally, we drew upon the lessons that were learned from COVID-19 and long COVID from the PLRC team, and they had such great success in this and really leading patient-led efforts, outcomes research. So we drew upon that valuable experience. I will now turn it over to Hannah Wei to provide a little bit more on PLRC and get us kicked off here. Thank you. Thank you, Suzanne, and thank you, Helen, for that wonderful introduction. I'm really grateful to be here speaking to you all and representing PLRC along with Gina today. Just a little bit about the Patient-Led Research Collaborative. We formed in April 2020 out of a Slack support group called BodyPolitik, and specifically in a channel called DataNerds, where we often assess and create conversations around the latest research. And so the Patient-Led Research Collaborative is made up of patient researchers and advocates, and currently we have folks across four continents, and we are an interdisciplinary team specializing in sciences, technology, that's speaking for myself, policy, and media. We have an IRB from the University College London. I won't read through all our accomplishments because there's a lot of it, but we have been doing this for over two years now, and we are both publishing our own papers and doing our own studies, but also increasingly getting into collaborations with major institutions to co-author and co-write studies and reports. And so these scorecards are directly drawing from our experiences doing this with our own team, but also with our institutional collaborators. Next slide, please. Great. So why scorecards? When we set out to do this project in the summer of 2021, we surveyed across the community for expert feedback and trying to see what has been done already. And what we found was that there have been models created to describe and to improve on patient engagement, but a lot of that was done from an institution's perspective, and also sometimes you get consultants to write these complicated reports. And we wanted to create something that a community can immediately use, but also to create a shared language for our collaborators to use as well to assess themselves and to improve on the collaboration dynamics. And we specifically honed in on assessing collaboration dynamics because we think that is kind of, a lot of times that is a barrier to getting more patients involved. And so some of the inspirations you see on the slides here are from work that's been done in citizen participation, involvement in the public and the research steps. And in particular, there's one that's, you see that these four circles, that's about resilience planning. And so we drew from each of these models, things that we liked, for example, negative labels to name and frame what citizen participation means and don't mean, and to help establish new baseline. From the disaster resilience model, we drew from that learning how to measure multiple domains. And so we kind of broke things down in our context, what does collaboration look like in different contexts. And for the levels of participation, we really liked how it's in a gradient and there is an ideal to aspire to. So we set out to take all this work further in our scorecard development and to define from a patient perspective what collaboration can look like in each domain. And the scorecards are meant to be immediately actionable, that it is agile, we can validate and update it, and we can use it as a way to create a shared dialogue. With that, I'll hand this off to Gina to talk more specifically about the dimensions of the scorecards we developed. Thank you, Hannah, and thank you everyone for setting this stage. I get to do the fun part of going into the scorecards themselves, and I'm excited to go over it with you all. So first of all, I'll be talking about the framework of the scorecards, and then we'll be talking about each scorecard in more detail over the next two sessions. As we mentioned, I'm going to be going into the research organization readiness, and Hannah, after I go into the structure of the scorecard, will go into the research organization readiness. Hannah will go into integration into research process, and then we actually have a next session where we'll be going into the other two. And so we released four scorecards with the playbook that we developed over the past year in the process that Hannah talked about. We focused on four major domains, and they're subdomains as well, and we provide measurement guidance on a five-point scale. So we iterated multiple times on these with potential users of the scorecards, both patient groups and research organizations that are partners to patient groups, and we feel pretty confident about these four. However, as Hannah mentioned, we do think these are going to be agile and open to updates. And so the research organization readiness one is about the readiness. It's allowing patient groups to measure the ability of a research organization to engage in a meaningful patient partnership. So it's all about the readiness and assessing that. The integration into the research process is about really diving into every detail of the research process and measuring how effectively the patients are engaging in each or participating in each phase of the research process. And then the patient partner governance one is measuring the decision-making power and the governance model and how that should be shared between partner groups, between the research partner and the patient partner group. You know, patients are the experts of their condition, and they bring value and skills to cross-domain collaboration. And then lastly, we have the patient burden scorecard, which measures the degree to which patient burden and their associated trauma is addressed, including accommodating patients who are dealing with illness and symptoms, and then also compensating them for patients' time and skills. So we, you know, just to provide context, we actually had worked on two other scorecards, but we ended up like deciding that we wanted to focus on these four because we felt like these would give the most value to the community and, you know, we'll see if we'll release more in the future. But yeah, now let's go into more detail into the structure of the scorecard. So the scoring model for this, you know, we developed a five-point scale ranging from negative two to positive two. Notice we're using the word collaboration as a keyword to label the scorecard measures. You know, this frames the, you know, it's framing it around collaboration. And then we anchored it on zero. And so we're saying we're creating a new baseline of acceptable collaboration. And we have negative scores, which we actually went back and forth on. But we decided, you know, to keep the negative because, you know, we're saying that, you know, zero is the new baseline. And below that is somewhat, you know, it's basically, you can call it minimal or no, and it's somewhat not acceptable. And if you're at one, you're great. And if you're two, that's ideal. That's what we're going for. That's our vision for what we think a patient-led model partnership could be. Notice also the light blue on the right, those are the subdomains. So, you know, to add more depth to the scorecard, you know, when we did the analysis of what was important to all these communities, we, you know, we dissected each area to sub areas, and this is providing more depth. So, for example, here's the integration into the research process. So, we go into each of the parts of the research process, and we provide assessments for each of those. Now, I'm going to go into more detail. So, next slide of the first one. So, the research organization readiness. So, this one measures the ability of a partner organization to engage in meaningful patient partnership. You know, we realize that the problem isn't always motivating patients and training patients to be more involved, but also whether our collaborators have the values, priorities, and the resources in place to carry out this, you know, this mutually, you know, beneficial or fruitful partnership. And so, this has three sub dimensions. One is the recognition of biases. The second is collaboration process. And the third is knowledge and disease subject. You know, a code that captures kind of this recognition of the biases subdomain, which I included here, is from one of the research, actually patient research groups that we talked to. And I felt like this was very, you know, what anchored our, you know, decision to create this scorecard, where they said, you know, some people say we have a conflict of interest because we have this disease. And they kind of look at them as less than or because they're patients. But they push back and they say, hey, as researchers, you also have your own conflicts. You have grants, you have money, you have a career in this. So, everyone is conflicted. So, the idea is we all have biases. Let's come into this in a way as, you know, humble and be like, okay, we all, let's recognize that. And I think if we see that in our partners, we see that as something that is a positive and an ideal collaborative environment. So, I want to kind of go into each subdomain a little bit. So, let's go to the next slide. So, I'm going to start off with the recognition of biases, which I kind of had that quote that we started off with. And, you know, here, if we look at the zero and acceptable collaboration, we are saying that research organization is, you know, at the minimum aware of their own biases, is open to feedback from patients. And not only that, they're actually implementing some of the feedback. So, you know, this is, you know, this is what we're saying is our acceptable collaboration. And then if you look at the, you know, let's go to the number two, because I think it tells you the range of what we have here. You know, we're saying in addition to being aware, implementing it, they're actually iterating on it. We heard from other patient groups that this is happening. And I think the most important one here is we recognize that there's a systematic way that they have and processes for accepting input from patient groups. So, it doesn't happen one-off here and there on specific research projects. But there's actually a way where it's documented, there's a process, and, you know, they're kind of working on it and iterating it. And this to us would be the ideal collaboration. So, this is really about a research partner having the awareness culture and understanding that they have their limitations as well and looking at patients to provide, you know, this input because they're valuable partners in the research. Yeah, once they have that culture and mindset, listening and acting upon that feedback. So, that's it for the recognition of biases. But, you know, and, you know, there's more on the negative and the positive, but I don't want to, you know, I think we have to go through all of these. So, you all will have to spend more time on them, but I kind of I'm trying to pick a few of the, you know, the areas to kind of talk about here. So, then we have the collaboration process. So, here this is, you know, if you look at the zero acceptable collaboration, here we're saying that, you know, the research organization has dedicated resource and infrastructure for collaborating with patients, whether they have patient panels, they're coordinating personnel on this, and they're also, in addition to that, they're doing a little bit of training to build skills. Now, patients, you know, are coming in with value of just understanding their own disease. But, of course, there may be some additional training that can help them to be more effective partners. And so, this can be where the research partner can provide that. And we're saying here as an acceptable, you know, we understand this is like a, you know, somewhat of a effort. So, we're saying at least limited training. But then when we're coming to the ideal collaboration on a process, here we're saying that, you know, there's the infrastructure for doing it. Of course, there's like more, you know, vetting on this by other patient groups and bigger, you know, it's also there's extensive training. So, they're really investing in making a patient being, you know, coming to the table and working in a way that, you know, is effective and helping putting in the resources if the patient needs it or asks for it. And so, that is the collaboration. Of course, on the negative here, we're saying that there's no infrastructure for collaborating with the patients. And lastly, under this scorecard, we can go to the third one, the next slide, please, is the knowledge in the disease subject. So this one, you know, you all may think is obvious, but, you know, the reason why we included this one, and I think this is something that, you know, as patients with long COVID, when we first started researching our own illness, you know, we realized that there's a whole patient community that are in the post-infectious disease world, ME-CFS, dysautonomia, where it's under-researched, not understood by many, but the research, the patient community had a lot more knowledge on this than anybody else. And we had, you know, realized this early on and kind of pointed this out as an important aspect of understanding long COVID. And so, you know, I think when we talk about knowledge in the disease subject, we're also talking about it coming from a perspective of the patient. And so here, when we say acceptable collaboration, you know, we're saying we want them to have at least one year worth of knowledge experience with the disease being researched, which you may think is obvious again, but, you know, we're talking about diseases that are maybe chronic or not as extensively understood. You know, this is not an easy thing, but sometimes, you know, where, you know, researchers think they know, but patient community can have a lot more knowledge in this. And then also like on the ideal collaboration, if you look at this, it's really about keeping on top of the information continuously and as well as from the patient community on the latest research. And so this is very important. Yeah. And, you know, I think, you know, this covers the, you know, the major points I wanted to cover on this scorecard, but I kind of want to summarize kind of what we, you know, honed in on in this scorecard, which is about the research organization readiness. So this is basically looking at more broadly, you know, does the research organization have a systemic process to include patients and take action on patient input? Are they, you know, investing? Do they have resources? Do they have infrastructure to support the patient partners? And then lastly, I think this one is maybe the most important one for this scorecard. It's about being humble about your own biases and being ready to learn from the patient community. You know, we saw this, you know, with a lot of some of our research partners. I mean, I think Helen and team are kind of an example of this, you know, where they kind of like came to us as like experts in this topic and be like, hey, you'll teach us how you did this. And we really appreciated that, of course, but we learned a lot from, you know, Helen and team as well. So, you know, these are kind of like a summary of, I think, the takeaways of the research organization readiness scorecard. And I'm going to pass it on to Hannah now to go into our actually most extensive scorecard. So take it, Hannah, from here. Great. Thank you so much, Gina. I think I'll echo a lot of what Gina has mentioned, because all of our scorecards kind of work together synchronously. So I'm really excited to introduce you to the integration into the research process scorecard, which is very dear to my heart. And I think a lot of PLRC folks will say so as well. I think in the comments, there's some questions coming up about whether we've used these scorecards with our partners or not. Just to answer that really quickly, this is a pretty early version. And I think a lot of the values are ones that we already practice pretty implicitly. And so I think what we're hoping for to release this into the world is for other organizations to use it, to adopt it in their own day-to-day engagement with patients and give us feedback, because obviously not everybody are structured like PLRC. And so we really want to know the skills. Anyway, so let's get into the scorecard. Integration to the research process is about accessing how effectively a study team involves patients early and also creating opportunities for collaboration every step of the way. It's broken down into five sub-dimensions representing each stage of the research process with ideal roles for collaboration, but also roles that are not so great. And we'll get into that as well. The point of having this is to have us try to evolve towards a process of co-written, co-led research that empowers the patient community and makes sure that this work is sustainable. Next slide, please. So the first sub-domain that we'll talk about is the hypothesis generation phase of the research. And the thing that we want to anchor on, and this is the new baseline that we want to set, is that research goals take into consideration patient priorities and patient inquiries and lived experiences are included when generating research hypothesis. So what we notice a lot is that when researchers approach us to get involved into study, sometimes what might happen is that they have a hypothesis already formed and were given maybe two days or even less to comment on these. And sometimes that doesn't result in a reciprocal and respectful collaboration because there hasn't been any time and space for us to include our perspectives. And so the ideal collaboration, what that looks like is that research goals are based on patient priorities and also co-written with a patient organization or patient researchers who are highly knowledgeable. Patient inquiries and lived experiences share equal weight with the organization's interest. And here I echo back with what Gina mentioned about biases. We all come with our own biases. And so it's more equitable to treat patients as partners with our own set of expertise when generating hypotheses. I want to touch on what is not acceptable. What is not acceptable in our books is siloing patients from this process and outright dismissal. There's also a bit of a trap, which is the minus one, that is attempting to include patient inquiries, but the communication somehow isn't adequate and there's little follow-through. It's almost like ticking a checkbox or having patient involvement be an afterthought in the hypothesis generation phase. Next slide, please. So then we move on to the study design process of the research. And for the context around this is that when we started PLRC, we were mostly conducting our own research. And as we've gotten more and more requests for collaboration, we've been really fortunate to be involved in the study design process of many of our research collaborators. And we noticed that having patients provide input on the study design is an opportunity that a lot of teams can adopt immediately. And it's highly scalable to build into funding opportunities, to build into bigger research organizations when they want to have more patients involved in this. And so how we're anchoring the baseline of acceptable engagement in study design is to have select patient voices approached to inform the process. Patients are involved in reviewing the study design and have an impact on how the methodologies play out in the study. Ideally, where we want to aim for is that the study design is co-written and is reviewed by a diverse array of patient researchers who are representative of the study's subpopulations. And this is really important in order to not, you know, not include patients who have these expertise, but are part of your study. So we also recommend protocol testing to be done by the patient community. And we do this ourselves as well. We've implemented this recently with the PLRC's Biomedical Research Fund. So as part of the fund award, each project that doesn't already have a patient representative on the project receives one or two paid patient representatives from the PLRC team to inform study design. And this will ensure that the lived experiences of patients are incorporated. So there's more information on our website. So I think I want to also talk about a trap here, which is having patients only involved for recruiting other patients. This in our books is not considered patient engagement. I want to emphasize that. Leveraging patient trust and having that be the only contribution is not a reciprocal relationship. So for this scorecard, what is not acceptable is not creating the opportunity for patients to be involved in study design, but still asking patients to promote your work, basically. And so we scored that negatively. Next slide, please. So now we move on to the analysis stage of the research. I think one of the things that we want to kind of establish with regards to patients doing analysis is that we want folks to see patient researchers as having expertise of their own, and patients being involved interpreting data and carrying out analysis in some capacity should be the baseline. In this day and age, patient researchers read many papers, have tried combinations of treatments. Some keep meticulous records of what worked and what hasn't. And all of that knowledge is so indispensable on a research team. Patients can ask well-informed questions and bring different domain expertise in analysis, and also to review manuscripts. And so at a baseline, we would like patients to be involved in interpreting data and carrying out analysis at some capacity. In an ideal situation, patient researchers can also co-lead on the interpretation analysis and or work concurrently with the partner research team to carry out the analysis. The trap and the non-ideal stage of this is that patients might have been asked to review manuscripts last minute, or have no say in what data to prioritize, or have no room to inform methods of analysis. This is both draining for us, but also results in lower quality of for us, but also results in lower quality of research. I see a question in the chat about authorship guidelines. We'll get into that soon. We have a subdomain on that as well. So next, publication. This is also really dear to my heart as well. As a patient-led group, a lot of our knowledge comes from freely accessible study results. Open science principles are really important to the patient community because we don't have subscriptions to major journals, and we learn from the results that have been broken down, usually in layman terms, from open access papers and open discussions around them with people who are inclusive, but also knowledgeable. So I think this quote describes it really beautifully, that it's important to feed this ecosystem if we want to see more effective involvement from patients. So the acceptable collaboration baseline is that study results are freely accessible to patients and the public, and the findings are summarized in lay terms in ways that are informative to the patient population. The ideal where we want to go is that study results are freely accessible, but also patient organizations invite others to co-write findings and reports, and there's a channel of communication that's open for patients to ask questions of the research organization. What we don't recommend, or what we don't like, is having study results be locked behind a paywall, findings that are not communicated in lay terms, and that just makes us feel like the patients aren't invited to the conversation after our labor and our data has been contributed to the process. Next slide, please. So now we're getting into attribution and how PLRC does it is that we make it possible for patients to be co-authors on collaborating papers. We don't have that as a baseline, but we have that as an ideal. So at a baseline for the scorecard, patients are acknowledged and credited in major public-facing communication, so press announcements, papers, if necessary, to the extent that patients wish to be named, and patients are consulted on how they prefer to be attributed. So I kind of think of this as taking a page from journalists that we love working with, so great journalists do fact-checking, asking your sources how you'd like to be credited and quoted and where. That's really important for us. The ideal is to be acknowledged for what patients did throughout the engagement process and recognizing that we all have our diverse skill sets and also include as co-authors on papers when appropriate. The trap here is kind of talking about us without us. So terms like patient-centered research, patient-led research, it's getting more and more popular, and sometimes what we see is we aren't consulted on how and where patients are being attributed, both in papers and also in public communication. And so what we prefer is to have organizations check in with us on this process. Some key takeaways for this scorecard is to invite patients into the process as early as possible and break free from tokenized roles of patient engagement and work with patients at every stage of the study, check in with them, creating opportunities for a meaningful contribution. Our vision of this scorecard is to evolve towards a process of co-written and co-led research with patients, and this will help create even more opportunities for patients to get involved and contribute meaningfully. And to come back to that quote about building a garden, to support and give back to the patient community to sustain future collaborations. And I think we'll hand this off to Suzanne. Yeah, so I'll jump in because I've been monitoring the chat and taking notes about some of the awesome questions. Thank you so much. And Jean and Hannah, I just want to say like if it was possible, you would have been interrupted by applause multiple times during the presentation. People are really fired up and it's great to see. There's also some great ideas for how to expand or some questions about some of the aspects. So I'm going to jump right into one of the detailed questions. And Gina, this was on the scorecard that you presented, and it was about the length of time that someone has been familiar or involved in that disease or condition. And this person writes, Sabrina Poirier writes, the length of time doesn't necessarily measure knowledge. There are researchers with years of experience who are still behind. Could you talk a little bit about how length of time was chosen as a measure and whether you considered other measures or where your thinking is going? And of course, Hannah, Lisa, anybody else can jump in as well. Yeah, and that's a really good question and a really good point, actually. I mean, when I introduced the scorecard, one of the things I mentioned was the subject, I think defining the subject or the domain of the disease adds more depth to that, right? Because if it's just saying, oh, do you have experience in long COVID, but how do you define long COVID? And that should come from the patient community, right? And so, yeah, I mean, we know from literature on mastering a skill or whatever, it's like the time you spend, right? But of course, it's how you're spending it and also which parts you're focusing on. And I think those will be defined by the patient groups or as the community in general. But I do think maybe we could revise that one to emphasize those things a little bit better. Again, I think we did quite a bit of revisions. We reviewed these with, and emphasize this, we did multiple rounds of research on these, showing them to both research organizations and to patient part groups. And we iterated on them, but I think they're agile and they're open to the community giving us feedback on it and they're ever evolving. So that's just what I wanted to say. And Hannah or Lisa, if you want to add to that. Yeah, just briefly as like an explanation for why we chose a year and kind of like amount of time was because we felt it was difficult to apply to multiple different diseases. If we focused on something like for long COVID, I could identify personally what I would be looking for that a researcher must know. So that's knowledge of ME CFS, knowledge that great exercise therapy is contraindicated, all of that. I think that probably in the next iteration, we need to somehow account for that. Like the researcher needs to have a certain level of understanding. We just couldn't quite figure a way to make that a measurable thing, but totally hear that. And I think that'll be useful to, to iterate on Hannah, anything to add or, or standard calendar list. Okay, great. So what I was going to say is the great thing about the chat is that we're going to save it and, and that'll become like a to do list for the team. So keep adding things to the chat and, and I'll pick out another piece, which was about the, the patient liaison, that there's somebody who works at the, the research organization that has, you know, the, the, the baseline is that there is a patient liaison and MC call it suggested that the liaison have access to the C-suite and that it's a powerful position that there's some way to, to know that, that this is not just a window dressing kind of position, but that this is actually somebody who has the ear of leadership. And so I wondered if you could talk about that and, and talk about, you know, how we might add that to a scorecard or add a measure related to what kind of power that person has. Yeah, I mean, I, you know, I think that one was in the research organization readiness one. And, and, you know, that's a really good point because yeah, you can have the patient liaison, you can have somebody dedicated resource, you know, but it has to be like, you know, effective resource, you know, not just putting it for the name of it. And I think, you know, a lot of this, you know, I think with the scorecard, you know, I think there's a lot of nuances that we can't include everything right. And we're trying to like make it succinct and like, but, but it definitely, that's a really good point. And I think one of the things that we have to, we think with the scorecard in it, you know, it's, it's, they're not going to stand on their own, they're going to have examples. And I think use cases, and I think that will make them more, you know, elaborate on these points. But, but, but yeah, I mean, I think potentially, you know, you know, when we say we have the infrastructure for it, you know, to us, that means that infrastructure, you know, is effective. It's just not there just for the lip service. Right. So, you know, and maybe we can include more examples that shows what that means, and whether it's like being on the C-suite, or, you know, whether they have a dedicated funds, there's many ways to ensure that it's not just lip service, right. And it's just, you know, it's more effective. And I think we can do those with examples. And I think part of our playbook is going to have examples and quotes from experiences that may elaborate further. But it just starts, it raises, it starts bringing up these questions of how do you make sure that it's like infrastructure that is effective, right? And hey, I would just wanted to also say that, that because this is a zoom, if you haven't had time to type a question, or would rather say your question, instead of typing it, please raise your hand using the zoom function of raising your hand. And Liz is going to be able to recognize you and let me know that she's going to unmute somebody. So yesterday, when I was posting to my blog and posting to LinkedIn, trying to sort of garner questions in advance, we got a really interesting question overnight. And I wanted to hold space for this question, because somebody might not feel necessarily comfortable in this space to ask it. But somebody asked genuinely, what's the difference between patient engagement and patient led? And so I wanted to open that to this group. And then if people want to start raising their hands to ask another question, please do. Does anybody want to take that about what, why did, why, why is it patient led versus patient engaged or patient centric? What's the difference between those, those terms for, for you in the, in the patient led group? I'll just jump in and say, like, there's a quote that just keeps coming back from our interviews, which is that, you know, having patients at the driver's seat of research is really important. And that kind of, for me, defines what patient led means is to have patients actually be at the driver's seat instead of being like a consultant on the side. And having, again, it comes back to the power dynamics. And just to like build off of a Gina's comment before that, what we find to really work, at least in our context is to build that relationship very early on with, especially the, like the PI of the study, the principal investigator. And they're the ones who has the power to, to drive the research, but also, you know, if we have a partnership with them, then we're kind of on equal grounds to co-lead. And that is, you know, that to us is patient led versus just engagement. Oh, Helen, you're on mute. Of course I am. Just a brief comment to that. This was a really interesting discussion with PCORI, because so much of what they do is patient centered, patient engagement. And we really pushed this idea with them that this was distinctly different, because this was patient led. And just, I think that's what enabled the scorecards to have a scale from minus two to plus two. I think if we had taken the more sort of vanilla terms, you probably wouldn't have seen the extremes in quite that same way. So really proud of what the team has done here. I think it's, it's really transformational. Awesome. I see a question from Andrea Downing. Oh, I was, hi Susanna, and nice to meet everyone. I'm Andrea Downing and founder of an organization called the Light Collective. I was going to answer that question by posting, first of all, I'm just really, really thankful to this community and to the leadership for creating these resources and continuing to validate problems across patient communities that we need to solve, because I directly see this for the cancer and BRCA community as a useful tool. I'm going to post this in the chat. And this is from, if anybody knows Dana Lewis, who is not only one of the original leaders in the e-patient movement, she has led a lot of patient innovation. She wrote this piece called the patient in the cage. And to your question, Susanna, I think it has a lot to do with, we often see this blind spots, the pain points and the priorities that researchers don't, because they don't have the lived experience that we do. And having gone through engaging in research based on researchers' priorities versus driving our own in the BRCA community, I find the latter, patient led research to be much more powerful and impactful. So definitely read that and thank you. Great. Thanks so much. I'm going to pull a question from the very beginning of the chat that was very nuts and bolts and I think is a good one. Can you talk a bit about how you, how you guys got access to an IRB and what advice you have for working with an IRB as a patient organization, patient led organization? Yeah, I can answer that. So we got our IRB through University College London, which one of the co-founders of PLRC, Athena Akrami works there. So it was very useful to have someone already in academia and had access to an IRB and that's how we did our first few surveys. So I think my recommendation for someone who doesn't have necessarily that connection or someone in the patient community who has that academic connection would be to try to find a patient partner that would be willing to bring you on to their IRB and someone that's willing to do this type of patient led research with you, that's a collaboration. The other thing, so we also have IRB through, for another project that's through just a regular one that we paid for. And so once you're able to maybe get some paid collaborations, you might just have to unfortunately pay for that, which it is expensive and pretty limiting, but it would be nice to find ways to make it more accessible and make it so that patients can have a way to do this research for a discount or even free, but still adhere to IRB and ethics. But yeah, I think finding that partner is at least right now what is a necessary step. Great, and I'll continue to invite people if you have a question that I haven't gotten to, raise your hand and you can restate the question, even if you put it in the chat and I'm not getting to it, please go ahead and raise your hand. Another one that came up in the chat, which I was interested in getting your perspective on is this word patient is really interesting when we know that of course there are people who are not currently patients who are survivors who could be part of the expert lived experience group. There's also care partners and caregivers and family members. So could you talk a little bit about how you had that perspective as you were building this and talk about the name patient led and how you see that maybe as an umbrella term? If you could, if anybody can take this question. Yeah, I can take that. I think we actually, this was an area where I remember this very vividly. We had multiple discussions about how we wanted to define it because we did actually talk to folks who were advocates and patient partners that were caregivers or had survived and their voices were part of the research and they informed a lot and they had very similar viewpoints to the patient groups and how they were thinking. But I mean, I think the idea of the patient represent that term to us represents you are the lived experience of the disease and that could be all encompassing to all of those, to those folks. Now, you know, do we need to maybe have levels of that? You know, we've discussed some of that, like, you know, what does it mean to be a patient? And we, you know, I think we ended up for the scope of this project to, you know, just say, you know, use the word patient and maybe like define it to be a little bit, you know, all-encompassing, but to be really about, you know, how much, you know, the lived experience do you have with this illness and that sort of thing. All right. Well, I think I'm going to throw it back to the slides because we, this is only the first of three webinars. So tell your friends, come back for the other two. And if we didn't get to your question, we're saving the chat and, you know, maybe these questions can actually kick off the beginning of, of, of what we're going to talk about next. Oh, I see. Somebody just raised their hand. Do you want to, do you, can we squeak in one more question? Veronica? Oh, thanks very much. Yes. Well, first of all, thanks very much for all the great work, because this is really going into the best direction. And we've been in touch before with a patient led research because we apply for one of their grants, which was, you know, a very good process. And one of my points, well, first of all, I'm a researcher, but I'm a science communicator at the moment, but I used to be a researcher. I'm trying to go back and bring part of this long COVID research to Mexico because there is absolutely nothing going on here. And what we've been facing, I think is something that we might really include into all this vision. And it's the fact that unfortunately, as a researcher, when you ask for a grant, there is kind of very defined guidelines of what they are expecting for them to fund you. So we want to include patients. We actually have patient led groups in Mexico, which are part of really our board, let's say the group that we're forming at the moment. But it's very hard because to really match what we would love to do with the patients coming from the patients, and I'm a patient at the same time, you know, it's very hard because that's not necessarily what the funding boards are looking for. So unfortunately, this really means that we have to change the whole philosophy coming from the granting boards, coming as well from the medical community, because we need medical doctors to be involved for a clinical trial. Like as researchers, we cannot, like I'm a biologist, so I cannot really, you know, do like a clinical trial, I need a kind of a clinical partner. But the problem is within clinical partners, they also have their own conflicts, not within a group, but in general, there is this conflict about each single clinician is looking into their own speciality. So you know, they want to see lung COVID from the heart if they are cardiologists, or they want to see it from the lung if they are pulmonologists. So we really need to change the whole philosophy and the whole culture of not only the researchers, but also the medical community, because we need them all to really work in together. Because otherwise, it's very hard to get this research done. Because without funding, there's nothing we can do really. So that was just what I wanted to comment on. Thank you. Really appreciate that comment. Thank you. Great point. I'll just add in, I mean, ultimately, for much of this to happen, we need that funder buy-in, right? Like we need funders to change what they're prioritizing. And hopefully, these scorecards will help them do that. Our hope is that funders can use these scorecards. But we need to move those incentives that funders have so that it's focused on patients. And I think that addresses some of the things that were brought up in the chat, like researchers don't have time for this, they don't have the staff, all of that. If funders changed their incentives and had buy-in and really valued this, then they would encourage that. And researchers could incorporate that in grant applications. They could have the staff, they could use deadlines that are more flexible that allow for patient relapses. It needs to be a paradigm shift on every level, but definitely getting those funders to be bought in is huge. And Suzanna, if I could add one quick point as well. I think it's also about getting, there was a lot of the chat about how do we get the researchers and the medical establishment on board? And I think that's part of the reason we're doing this collaboratively. But I think there's also really important, these scorecards could be used by the patient groups to assess who they want to partner with. But really importantly, they're also intended to be used as self-assessment tools by the research organizations, by the medical organizations. How are you doing? And how do you need to improve? So that's really the goal of these webinars as well. So if this community has suggestions as well about how to get that out into that broader community, we'll certainly work with PCORI, but we'd love to work with other funding agencies and others, as well as certainly all the 50 medical specialty societies, part of our organization, but we've got to shift that power dynamic for sure. Yeah, Veronica, thank you so much for bringing that up. It's been a topic that we've been also discussing and thank you, Kellen, for summarizing that so beautifully. For our own projects with the Patient-Led Research Fund, this is what we've tried to do because we recognize that we need those incentives coming from the fund itself. And so built into our process is having patients involved in the study at every step of the way. And we hope to set a good example of this actually being done and so that funders can look at what's out here and the effects it has on the quality of the research and on the patient community and say, wow, we want to adopt something like this and really be able to use these scorecards as a means to shift the conversation. Great. Well, I'll just say congratulations again. This has been a great first introduction. And what you can see on the screen is the next two events that are being sponsored. So please tell your friends, register for those. The other scorecards are really going to probably generate just as much a conversation as these have. So stay tuned. Great. And thank you all for participating. Just want to reiterate, we will make all of these materials available to you. The recording, the slides. We want the scorecards to be widely disseminated and used. And we want your feedback. We're continually improving and revising and iterating. So thank you so much for all of your feedback today. Have a good day. Thanks Susanna for facilitating. Thanks team. Bye.

Patient-Led Research

Scorecards

Medical Specialty Societies

Research Organization Readiness

Patient Engagement

Power Dynamics

IRBs Access

Funding Challenges

Collaboration