And as mentioned, this will be recorded, so next slide. So briefly, this is the CMSS and PLRC team. Really delighted to have the five folks from the PLRC team, four of them joining us today, Gina Asaf, Hannah Davis, Lisa McCorkle, Aneesha Sekhar, and Hannah Wei, and then from CMSS, myself, Liz O'Keefe, and Suzanne, who's been the project manager moving this forward. Next slide. We've also had an extraordinary advisory panel. More than half of the panel was, in fact, made up of patients, and really wonderful insights as we presented the ideas, the concepts, and the scorecards all the way through, so just thanks to them. Susanna Fox moderated the last webinar, and she will also moderate our final webinar in early February as well. Next slide. Excellent. So the goals of this work, and we already do have a nested playbook that we had put together in draft form before the scorecards came out, really was to think about how we could support patient-led research. And we had a good discussion about that on the first webinar as well, of the distinctions between patient-led, patient engagement, patient-centered, and we'll happily talk more about that today. But very much how do we incorporate patient-led research, patient-generated outcomes into the broader clinical research enterprise, including around comparative effectiveness. Very much thinking about how we can collaborate specifically with diverse patient populations, thinking about how patients could lead these efforts in partnership, and really the keystone here really is very much about how do we achieve meaningful collaboration, respectful collaboration between the medical community, research community, and the patient research community. So we'll hear lots of the patient-led research collaborative here very much has been focused on long COVID, but I think what you'll see, what will be presented here, I think is widely applicable across many, many different disease groups. So next slide. And at this point, I'm going to turn things over to Lisa to give us an overview of the PLRC and then launch into the work. Thanks. Thank you, Helen. Hi, everyone. I'm Lisa McCorkle. I'm one of the co-founders of Patient-Led Research Collaborative and a person with long COVID. So just as a brief intro to who Patient-Led Research Collaborative or PLRC is, we're a group of patient researchers and advocates across four continents. And we originally were founded out of the BodyPolitik COVID-19 Support Group on Slack. And we actually did the first research on long COVID by sending out a survey documenting symptoms and impact on work and life of long COVID to the BodyPolitik Support Group. We've continued to do research since then, our own research, as well as consulting on research studies. And that's kind of where we developed a lot of these ideas, in addition to doing interviews with other patients, researchers, funders, other stakeholders, and doing a literature review on what currently existed. Next slide, please. So we really wanted to, you know, develop something that was built off of what we've learned from our experiences. There are many models of patient engagement that exist from what we've learned from our literature review and speaking with people. But it seems like there was kind of a need for something like a scorecard, something that could be used by funders to assess research teams and how meaningfully they're engaging patients, for research teams themselves to assess themselves, improve their own engagement. And then for patients to know what to ask for in collaborations. So this kind of scorecard where you can evaluate where you are now and how you can improve seems like something that would fit the need for the current moment. Next slide, please. And now I'll jump into the different scorecards. So we ended up creating four different scorecards that we have public now, and they are four dimensions of collaboration. We covered the first two last week in last week's webinar, which was on research organization readiness and integration into the research process. And today we'll cover these last two, patient partner governance and patient burden. Next slide. So we, to touch on the scoring model again, which Gina went into depth last webinar on, we have a five point scale ranging from negative two, which is non-collaboration to two, which is ideal collaboration. Each dimension includes subdomains that are also evaluated. And we scored it this way in this negative two to two way, just to indicate that getting a zero is really the new baseline. We want to change what the baseline is for acceptable collaboration when working with patients. Anything below that, you get negative points, you're penalized for that, and anything above you score positive, up to two, which is the ideal collaboration. Let's jump into patient partner governance scorecard. Next slide. Perfect. So something we came across a lot in our own experience is that patients can be in some parts or even at all steps of the research process, but they might not have any real decision-making power or they're treated poorly in interactions. So patient partner governance scorecard measures how decision-making power and governance is shared between patient groups and their research partners. There are two sub-dimensions. One is meaningful decision-making between groups. And the second is accountability between groups. And it really evaluates the working relationship between these two partners, how decisions are made, what the culture is, and how groups are held accountable. We have found this in our own experience and hearing from others that this is really a key part of making sure that engagement is meaningful. Next slide, please. So diving into more of the meaningful decision-making between groups sub-dimension. So acceptable collaboration is in the middle there, the yellow zero. Decision-making process for significant decisions, including funding, study design, publications is well-communicated and it's agreed upon between patients and the research organization. So this means that the patient group is aware going into the collaboration, how much power they have to sway decisions. They are finding out as they go and they really agree to the collaboration and the dynamics. In the ideal situation, the ideal collaboration, this gives deference to the patient group with sufficient support provided to make these decisions. So what we don't want to see with this is for patients to go into a project not being communicated what the decision-making structure is and maybe having the understanding that they can sway the study design, but it turns out all of their ideas and suggestions are ignored. Patients deserve the right to know the situation they're going into so that they can decide is it worth their time, their spoons, their energy. Ideally they have that decision-making power. They have a vote. They really have a meaningful seat at the table and they can influence the study. Next slide, please. The next sub-dimension is accountability between groups. This more looks at the culture and accountability between the different groups. Acceptable collaboration here is that there's a shared understanding and ideally a written agreement of the rules of engagement and the culture with defined consequences for not following through between these groups. From the get-go, it's understood what roles everyone has and how all of the different groups involved will be treated. For example, everyone is treated with respect. Everyone's input is valued. Maybe deadlines are flexible within X amount of days. Having all of those types of rules of engagement so that all parties are aware of basically what they're signing up for and how to expect to be treated and then having those defined consequences. If someone is disrespected, it's not just glossed over and ignored. Ideal is the number two is basically the same thing as acceptable, but deference is given to the patient group to define the rules of engagement and with sufficient support to do so. What we don't want is a lack of understanding and agreement to what the culture is. We want agreement on what's acceptable for communication, how to treat each other, etc. What I've personally experienced, interactions, and I'm sure many patients who have consulted on research studies where you're asked to consult on a study and then you are dismissed in the meeting, you are maybe even yelled at, and there's just not necessarily a positive working relationship. Having those standards and rules of engagement from the get-go can really help ensure that there's a positive working relationship. And now I will pass it over. Oh, sorry. No, not yet. The last slide on this is the key takeaways. So just some key takeaways from the scorecard is that for true partnership, we need to give patients decision-making power, ideally at equal levels as the research partners or giving even deference to the patients, but minimum having that decision-making communicated to all parties. And we agree ahead of time on what that process is, what the rules of engagement are, and what the culture will be, and really ensure that it's a safe and positive working environment for all parties. Now I will pass it over to Aneesha. Thank you. Great. Thank you so much, Lisa. My name is Aneesha. I'm with the PLRC, and I'm a person living with long COVID and associated conditions. And I'll be talking to you about the patient burden section of the scorecard. So this part measures the degree to which the patient burden, which can be the harm that's caused, any trauma, cost, is both addressed by the research organization, minimized when it's possible, and compensated when it's not. And we're starting from two central tenets. The first is that, as Lisa mentioned, patient researchers are often treated as second-tier contributors. Maybe their experiences and contributions are devalued. Maybe the challenges and trauma that they face as part of their participation gets waved away. And secondly, even in a world where we have perfect parity, patient researchers are just going to be exposed to more harm and risk that other researchers are not. That might include just exacerbating your illness simply by participating in research. That might be the weight of being emotionally as well as intellectually invested in the research question. These things are really central to the patient representative experience. So with these burdens in mind, we created a scorecard to capture how well an organization can minimize what is possible to minimize and compensates what cannot across four sub-dimensions, which are accessible engagement, trauma-informed practices, responsiveness to patients, and compensation. Next slide, please. So we start with accessible engagement, which asks if patients can fully and safely participate in the research process. What we want to avoid here is requiring that patients incur some sort of harm, whether that's needing to create expenses, cause harm to themselves mentally, physically, emotionally, in order to fully participate in the research, and especially not viewing patient harm and hardship as proof of their commitment to the research. We want to see that as something to be minimized whenever possible. So at a baseline level of collaboration, what that looks like is designing avenues of input with the patient population's needs in mind, such that they can contribute in a minimally harmful way. And also, if you're hearing from patients that they need additional accommodations, you grant them whenever it's physically possible. So what's great about this level of collaboration is that you are being mindful of patient access needs, and you're responding well to hearing that there might be gaps. But what's missing between this and ideal collaboration at two is going to actually to the patients for them to identify their needs. So when we look at ideal collaboration here, that starts with patients co-designing the engagement avenues. And specifically, this means including all impacted patient subpopulations, so that you can hear from a diverse set of people to tell you what their access needs are. We interviewed a number of expert patient researchers for the scorecard, and one of them brought up the risk that undocumented people can face when driving to a research center or traveling by plane to a conference. They said that this is an access need that is very rarely brought up when discussing patient needs, because that subpopulation is so rarely heard from. So we really need to make sure we're talking to a diverse set of patients to make sure that everyone is able to engage safely. Next slide, please. Next up, we have trauma-informed practices, basically asking how well the research organization minimizes the potential for trauma as part of participating in the research process. The phrase is a little bit unspecific, which I think speaks to the fact that different patient groups, different areas of research have very different contexts, which again highlights the need to go directly to the patients for them to say what they need and what potential problem areas might be. Some examples of trauma that patient researchers can face, Lisa mentioned this as well, is invalidation or straight-up hostility from researchers discounting your lived experience, watching your fellow patient representatives decline in health or seeing yourself decline in health over the course of the research process, or an overstudied patient population being tapped over and over to participate in research without seeing any visible benefit to their community. So at a baseline, what acceptable collaboration looks like is you recognize the need for trauma-informed practices, implement them as best you can, and listen to patients when they bring up more needs. And this really speaks to a fantastic mindset from everyone who's participating in this because you're understanding the need for trauma-informed practices, you are bringing humility and a spirit of wanting to meet the patient reps and have them succeed. And once again, the gap between that acceptable collaboration and ideal collaboration is the element of patients co-designing. So in an ideal collaborative environment, starting from a place of mutual respect, patients and non-patient researchers are co-designing what a trauma-informed environment looks like throughout the research process and are given sufficient resources to minimize harm and trauma whenever possible. Next slide, please. So third, we have responsiveness to patients, which measures the extent to which a research organization genuinely seeks and responds to patient input. What we want to avoid here is tokenizing patient contributions, patients risking retaliation or re-traumatization for communicating, and asking patients to give from a deep and meaningful part of themselves without seeing a benefit to them, to their community, or to the research question. So at a baseline, acceptable collaboration looks like the organization thoughtfully creating methods for patient feedback that influence the current study and acknowledge the feedback and show them their impact. And this speaks to the two previous sections, because number one, those communication channels need to be accessible to the patient population. And two, because of the inherent power imbalance, these channels need to be safe enough that patients can feel safe voicing their opinions without risk of retraumatization. Once again, in the ideal collaboration, though, patients are co-designing from the get-go communication channels that are safe, accessible, impactful throughout the study, and close the loop to acknowledge the impact of patients' input. Next slide, please. So finally, we come to compensation, which asks two key questions that go to the central tenets that guided the patient burden scorecard. So the first question is, are patient researchers at parity with other researchers? Are they given fair compensation for their time and expertise? And are they rewarded with visibility, professional growth, and the knowledge of their impact that nonpatient researchers will enjoy? Secondly, are the added burdens that patient researchers must assume acknowledged and compensated? So that might mean meeting the patient's needs in terms of payment methods, both in the delivery mechanism, you know, is it by ACH transfer, check, cash, and the timing of payments, which can impact eligibility for benefits and other needs that patients might have. And then secondly, any expenses or harm done to patients while participating in the study, is that being compensated as well? For example, let's say that you participate in a one-hour session, which causes your symptoms to flare up so badly that you can't cook and you have to order in, is that being factored into your compensation? So at a baseline, patients are going to be compensated fairly for their time, for their expertise, any expenses and harm that are incurred by their participation, and with flexible payment methods and timing. And I want to narrow in a bit on the expertise. As Lisa mentioned, patients are often dismissed or seen as less credible simply by virtue of experiencing the illness, which is the exact opposite of the dynamic that we want to see. Patient experience is incredibly valuable in itself and deserves to be compensated fairly. And there are a number of resources out there, including the Fair Market Value Calculator from the National Health Council, which can help you assess either as a patient or as a non-patient researcher, what fair compensation looks like. So that's what we're looking at as a baseline acceptable collaboration. In an ideal setting, patients are given decision-making roles in setting that compensation so that they can ensure that people are being compensated fairly, that any harm done is being taken into account, and that patients are enjoying the same professional development opportunities and knowledge of the impact of their work that non-patient researchers enjoy. Next slide, please. So finally, some key takeaways. Number one, just work with patients from the get-go, creating an environment that is trauma-informed and accessible, and is therefore going to be minimally harmful for patients to fully participate. Second, again, this is a theme, work with patients to co-create communication channels that are accessible, safe, impactful, inform the study from end-to-end, and close the loop to acknowledge patient impact. And finally, compensating patients fairly for their time, expertise, and harm incurred in the method and cadence of their choice, and with the professional development and authorship, visibility, and other non-monetary rewards that other researchers can enjoy. So taken together, these show that you understand and respect patients' body of knowledge, approach the potential for harm and trauma with an element of humanity, humility, compassion, and that you're proactively working to minimize harm. Next slide, please. Finally, I wanted to say a very huge thank you to everyone we interviewed for the patient burden scorecard, providing a lot of insight and expertise, and sharing insights that really helped shape the scorecard. And with that, we will go to a Q&A. Wonderful. Thank you both. So much rich information, already a great discussion happening in the chat. I saw Hannah, you started to answer one of the questions, which I thought was a great question, specifically about how do you ensure that patients have the education they need to participate? And I know that was one of the scorecards you guys were working on, and we sort of tabled it for now. Do you want to just answer that one? I think it's a really important question. Yeah. Thank you for bringing that up. I think when we did the interviews, we both touched on how partners can educate themselves to work with patients, but also patient groups can upskill to work with research, especially research partners. And I think one of the major points that we got out from that is that there's both hard and soft skills for patient groups to really focus on. And so a lot of times it's about creating the right environment to develop these skills as well. That's why it's really important to have a collaborative relationship. That way everybody can learn to be more effective and more helpful towards the ultimate goal. I think specifically we've seen models where patient groups develop training amongst themselves in order to sort of place their folks into clinical trials, for example, but also on advisory boards and do talks at conferences. So it really depends on the capacity and the interests of the patients themselves as well. I hope that answers the question. Absolutely. I think you indicated your willingness to share that draft scorecard as well. If anybody wants to kind of take a deeper look at it, I think it was a great start and I think really important. A couple of questions came up about how these would be used and I think particularly how the scorecards could be used by funders. Lisa, I wonder if you had any reflections on that, particularly given the experience that PLRC has just gone through of actually being a funder. Yeah, so I think it's a really good opportunity for funders to evaluate how meaningful patient engagement is. Some of it is hard to see in a proposal, just a written document without seeing and interacting with the PIs or the researchers involved. But I think that it can be really important as a funder to really incentivize this type of collaboration. It's very easy for a funder to say, we value patient engagement, but what does that really mean and how does that actually translate into whether something gets funded or not? And would just saying, we have patients on our advisory board, kind of qualify that proposal to be funded. From our perspective, that's not necessarily enough. You have to have a legitimate plan that covers a lot of things that we have in our scorecard. And we also recognize, I mean, I think it's often brought up that researchers don't have the time to do this. And that's where we're trying to really flip the script here or just expand and change what the baseline is currently. I totally understand that it's difficult in this current environment and what funders fund and incentivize, and we're trying to push that. We're trying to move that needle so that this type of collaboration is really what's incentivized and prioritized by both funders and researchers. Yeah, absolutely. And just again, a couple of comments in the chat. Obviously, this will be shared with PCORI. We've sort of indicated all the way along the path that this is potentially something they could use, but we also do hope to share it widely with NIH and any other funding agencies. I think in many ways, I think one of the brilliant pieces of this work, I think, is actually the use of the negative numbers. I think in many ways, there's been this sort of assumption, particularly I think on the part of a lot of medical organizations and research organizations, we have a patient on the panel as being sufficient. And I think it's great to see that that's not an acceptable, that's actually a negative number. And hopefully, if that gets built into scoring and that gets built into true collaboration as a way that these get used to decide funding, then hopefully people will at least have to find the time because otherwise they in fact won't get funded. There's a question, Rivka, you've been adding a lot in the chat, but feel free to speak as well if you'd like. Oops. Hi. Yeah. I just want to say I'm so moved both by this presentation and the last one. I mean, like moved that I feel like crying because I feel like you are recognizing the value of us patients on a level that I just haven't seen before. And I'm deeply, deeply appreciative. And I asked a lot of questions in the chat already. It's been really hard as a patient involved with research efforts. And I know a lot of you know what I'm talking about. We really respect the researchers and really respect the burdens that they are struggling with. And it's really hard to watch them struggle. And they're often our friends in the end because we've been working with them. And so we don't want to be like a pain in their butt and say, meet with us, meet with us. So this incentivizing that you're talking about is really great. So that they want to meet with us. And also, I just want to do give a shout out to of course, the team that's worked on this, but also I see researchers on this webinar right now. And I really appreciate the researchers who have joined. And some of you are already wonderful with including patients. So in your work, that's it. Thank you so much. Wonderful. Thanks, Rivka. Anybody else want to ask a question? Otherwise, we'll keep going through the chat. There's plenty that's already there. And thank you for so many of the accolades for the team. Really just extraordinary work. There are a lot of questions and maybe anyone on the team could answer. I think there's been a lot of questions about the, and praise for using trauma informed within the scorecard. And I think a lot of questions about how do you provide the kind of, what could researchers do to provide the care that patients may need related to the trauma of talking about their illness, talking about their symptoms or anything else related to being engaged in the research. I'll turn to anybody on the PLRC who'd like to respond. Sure. Happy to jump in. So I mentioned in the chat, some of the ideas that came up in our interviews, one being credits to speak to a trauma informed counselor, help setting up and paying for a visit with a clinician who understands your condition and is able to help you manage it. Other, I think broad picture, one thing that came up from a couple of different people, and I personally strongly identify with is that seeing the impact of your work is an incredibly meaningful way to address some of the trauma that you incur because you're asking to give something, you know, very, very deep of yourself. And if you just are kind of tossing it into a void and never see what happens, that's one thing. But if you're hearing, Hey, there is a change, it is impacting my community. It's impacting research question that I deeply care about. That can be incredibly meaningful. But again, you know, it always depends on the context and the research question and patient population. So I think that speaks to the importance of having patients involved from the get go to kind of anticipate what are some of the needs? What are some of the resources that need to be provided? Excellent. Anyone else on the team want to respond? Okay. Lots of good questions around this and thanks to the team for responding on many of these as well. I think one of the questions was really, have we started to use these yet? And I'll say these are pretty hot off the press, not yet used, not yet translated, and not yet psychometrically sound assessments done. I think all of which we'd love to see if we can get additional resources to potentially move forward with, but a sense maybe from the team of maybe from even reflecting on your recent experience of awarding some grants, do you have any sense of how this might have influenced the way those decisions might've been made? Sure. I can answer that. Yeah. So I think, so for context, we had our patient led research fund, which was done last year. We announced the awards in November of 2022. And we had a panel of patient researchers, an amazing panel of patient researchers who, you know, scored and evaluated the proposals. So in our request for proposals, we identified that involving patients was a priority. And not only that, but we provided as part of the award, if they didn't have plans to include patients on their, on their grant, they would be able to patients on their study design committee from the get-go. We are paying patients to be part of the study design committee of each of these studies. So for, for us, you know, it was a bit easier because we are a patient led organization. We have, you know, contacts with patients, but we're making it super easy. We're not even asking them to take it out of their budget. We're providing the patient researchers to these different teams. And we, you know, part of the evaluation criteria was how meaningful it seemed like the engagement was going to be for these different teams. And we tried to use some of the ideas from the scorecards to implement that. But as I said earlier, I, you know, I, I think it can be hard to evaluate that just based off of proposals. So I think there's some iteration that can be done there and how, how can you really apply it given you know, proposals are usually pretty short in length or some of our proposal tried to be, cause we had patients reading them and you know, what you can write looks different than what's going to happen. So I anticipate that a lot of the work will be you know, working, the patient researchers working with the researcher and identifying those rules of engagement and using a lot of the stuff in the scorecard now that we have this, these certain funded studies up and going. Yeah, that's great Lisa. I also really like the specificity of the scorecards, you know, it's myself and the sort of the medical community. I mean, they're being very explicit about sharing funding, sharing publications, expecting patients to be authors. I think it is pretty radical. I don't think a lot of that has been done to date. So we're also hoping that in our role in terms of being with all the medical societies will help inculcate that as they think about their research models, they make sure this is part of the puzzle. A question, actually a few really interesting questions, a little bit more about developing this resource. So this is a question from Sarah, the process of developing the resource, what were the most difficult challenges and what was most unexpected? I can, I can talk to that a little bit. Yeah, thanks for that question. I mean, we followed a, you know, pretty rigorous process where we both reflected on our own work that we did at PLRC and asking members of PLRC as well. And then, you know, with a survey and some just reflecting on our own work. And then we also interviewed a few folks in both the research world and patient led research or patient contributors. And I can't remember the exact number of folks that we talked to, but it was quite a few. We did a, you know, one-on-one interviews and we learned a lot from the community as well, in addition to like comparing it to our own work. And then we also did a bunch of desk research. We came across some of the resources that were, some of them were shared here. And so we'll be in our, in our playbook, we'll have, you know, we'll reference all of those. There is a lot, there's a lot out there that has been done in terms of patient-led research, even if it wasn't called patient-led research. And, you know, and we learned a lot from those resources and then they're reflecting on our own works. And that's kind of the summary of kind of how we reached to this. And then we did a lot of synthesis. I think the hardest part, in my opinion, and I think everybody will probably have different hard parts, was defining the scope of this and really coming out with something that is like gonna be actionable and usable. We could have went in so many different directions, even picking the scorecards. We had to eliminate a few that were, you know, I think Hannah mentioned one, which was not eliminated just for now. We had to prioritize which ones we wanted to put out to the community. So that was really hard. And then the nuances with like the terminology, that was also really hard. Every single, we revised like the text multiple times for all of the scorecards. For the patient burden, when Aneesha did a great job of interviewing and testing it further and getting feedback on, whereas the other ones we did a little bit of that, but, you know, we could do more. So I think the nuances around the definitions and how you describe it, that was really difficult. And then the scope, those two things. Yeah, I think that's great, Jean. I'll just add for anybody who wasn't on the first call, what was also interesting was our initial vision was creating two playbooks, one for researchers, one for patients and two sets of scorecards. And I think one of the really important contributions here is, you know, the language needs to be the same and we have to get the medical community to adopt the same language that we're talking about here in terms of patient-led and engagement in a way that we haven't seen before. And I certainly, I think Dave's made some really important comments about power and balance. And I think we talked a lot about that as well. I think one of our advisory committee members said she learned more in an hour of one of our advisory committee calls than she'd learned in the last year just in terms of engaging with patients. So this is, I think, it was a very powerful experience overall. Stacey Hurd, who I know is on the call, made a comment about how patient-centric needs to be replaced with patient-led immediately. And this was something we talked a lot about with PCORI in terms of the definition and even trying to come to consensus on what that looked like. I wonder if someone on the team wants to give a sense of why patient-led is the term here and how it is in fact different than patient-centered or patient engagement. I can talk a little bit to that. I mean, I think, as I said, I think that was one of the big struggles or big challenges was like these nuances with like terminology we were using. I think patient-led is how we viewed ourself as a group. When we did our work, that's what we defined ourselves as just because we were a group of patients trying to understand the disease. And it kind of was how we defined ourselves. So when we started doing this work, we realized there's like a spectrum of patient-led in many ways. And that was like, and again, not always going to be the case where everything is going to be completely patient-led and throughout the whole process. But we felt like that was like from a goal perspective or an ideal perspective, that's what our work is trying to define. So that's why we chose the patient-led word because it's kind of an idea and a goal and what we would like things to move towards and evaluate things on that spectrum versus the patient engagement, which is sometimes feel more passive and less in power, the patient is less empowered. So yeah, I mean, that's, I'm kind of like just talking based on just what's coming to mind, but if anybody else has anything to add, that would be great. Yeah, I was just thinking, I mean, I think a big part of it is shaping the outcomes. So what outcomes do we want to see from research? And what I think we all at least believe is that those outcomes should be shaped by what patients prioritize because these illnesses are impacting patients. And so the outcomes and the research should really be focused on what do patients care about? What outcomes do they care about? And what type of improvement or therapeutics are they looking for? I think too often research in medicine is focused more on just a researcher's perspective and what they're interested in. And that might not actually have impacts on the patient's experience for many years potentially. So really kind of, and that type of research can even be patient centered, right? Like it can have patients involved throughout the process, but it's really kind of changing it so that the entire research process, including like what outcomes we're focusing on and what questions we're asking are prioritizing and being driven by the patients themselves. Excellent, wonderful. Sabrina. Hi, I'm just wondering, this is so awesome on so many levels. I'm sure you're gathering that from the chat. I'm wondering one of the things that I know Rivka has mentioned and it's been my experience as well is that even the best intention researchers are so short on time and short on knowledge and capacity around these issues that they don't know where to start. So I'm wondering if you guys have thought about, for example, templates to do things like agreements, discussions, like how do we find market value for compensation? Because I know Lisa, we worked together recently on something and there was a huge discrepancy between where Canada was and where patient led was. And so like, how do we start to like get some of those things gathered together and put it in front of them so that then the hard part is just the discussion and the process of moving us forward versus the, oh, before we can even do that, we have this huge lack of knowledge, this huge gap in terms of how do you create an agreement with patients? Like, what does that look like? I've never done that with any of my other partners and yet everybody benefits. So that's something that if you guys aren't doing, it might be something the community can get involved with or start to think about. I mean, there are lots of questions. That's a great comments. There are lots of comments in the chat about creating the resources that go along with the scorecards. And I think that's a great example of pulling out some of the key tools like agreements and compensation that we could bring forward. I don't think the team has done that. We've certainly not done that work yet, but I think it's a really interesting idea. Anyone else? Yeah, I just wanna add that we've been having these conversations as well as to how to provide resources for our patients, not specifically agreements, but like how to negotiate these points. And because we've been doing this for two years and we figured out like at certain points you should ask for these stuff or like you can actually ask for compensation for showing up to meetings, for example, and things like that. I think it would be really helpful too. So I agree with you. It would be really helpful for the community to develop something like this so that patient groups can borrow and use. Excellent. It's a great question as well by a patient, Dave, in the chat about whether the scope is really limited to individual research projects or could this also potentially help support the development of patient-led research agendas? So patients guiding what gets researched. I would also extend that. I think you could also use this to look at agendas around policy between medical groups and patient groups as well. Reflections from the PLRC team. Yeah, I responded to the comment saying that in our, I think in our previous session where we talked about the integration into the research process, we touched upon the different phases of research and how it's important for the patient-led concept to be from A to Z. So it's not during the conducting of the research, it's also in the design and the hypothesis forming. I believe that addresses that. But I do think there's even before that, maybe we need to consider, like you said, where it's the, or like the policy and there's maybe we didn't touch, maybe there's even beyond the integration of the research process that we need to consider that's wider. But definitely, I think the idea, because when we touched upon the governance, we're touching upon the entire research process, we're saying this is not only bringing in patients at the point where it's just, you're conducting a very specific thing. It's also in the decision-making process. So, yeah. And that's certainly been a lot of PCORI's interest in this space has been ensuring that the outcomes get looked at are the ones most important to patients. So thinking about how to use the scorecards in a way that actually gets at what gets studied rather than how to conduct a study, makes a lot of sense. And we can talk with them about that as well. Just wanna see if anyone else wants to jump in in person before we continue to go through the chat. The PLRC team has been so good about answering the questions. It's hard to pull them out for discussion, but I think there's just a lot of them that I think what we're seeing is this is just such a rich community of groups who would really love to help build the resources around the scorecard, even an offer to potentially test the scorecards in an environment in Karolinska. Sarah, that was you. I'd love to hear more about that. Yeah, of course. That's the offer. Definitely stands. Of course, I can't promise that the others will be on board, but I can promise to ask them. But I wanted to ask another question as well, apart from all the ones I've put in the chat already. So will you also be publishing this academically since the way to influence researchers is by publishing academically? I haven't gotten to it yet, but I would say the answer is yes. I think it's gonna be important that this get highlighted and the PLRC team has been amazing about getting their work into high-level medical journals. And I think collaboratively we'll make sure that it gets seen at the highest levels and really gets, really importantly, seen and gets used. And as Dave points out, it's gotta be something you can cite. So absolutely something we'll push forward. But literally these things were just done in the last, finally completed in just the last couple of weeks or months. So I think there's a long tail in terms of how we actually get them out there and used. And we just wanted to take an opportunity to get it out there to have everybody see it before, even before it gets used to get your input. And even as we've put forward things to PCORI, we've indicated every one of these is still a work in progress. It's an opportunity to adapt and change as we get more input from the broader community. Any response to the PLRC team about how to get this out there or putting this in the literature? We haven't added that to our to-do list yet, but it's gotta be there. I guess we won't get to it. Wonderful. All right. Anybody of the team say any other good questions in the chat? Otherwise, I think we probably could go back to the, or if anybody on the team wants to add something that wasn't asked that they think is important to add to the mix. We'll certainly take a look at everything in the chat, all your suggestions. Because in fact, the next webinar is about how to use the tools for improvement. So I think we've talked a lot about how patients would use these tools to evaluate potential partners of whether this is a group you wanna engage with. But I think we also wanna use these scorecards as a way for researchers and medical groups to really do a self-assessment or have others assess them. And then I think what we're really gonna try to focus on in the next one is that if you know how you score, what are the strategies to improve on some of these elements? So Helen, this is Pawan Goyal from ASAP. Hi Pawan. Hi. So one of the questions as we start talking about tools would be, where is the data? What data is available? And how do we create a baseline when we talk about improvement? Do we have understanding of variability and improvement in the baseline opportunities? Yeah, thanks Pawan. And certainly the PLRC team has been amazing about collecting all that data really across the world on patients. Anyone on the team wanna talk a little bit about the strategies you've used to collect the data from patients? Is this the context of running our own studies or is this in the context? Yeah, okay. So we advocate for open science principles. So the data that we collect, it's anonymous and we store it in a GDPR compliant server. And we try to loop in our participants at every step of the way. And I don't think this is covered too much on any of our scorecards, but it's a very important value for us to give the information and knowledge back to whoever participated. Because you have to understand they donated their time and data to us. And so the least we can do is to update people on where the research is and summarize our findings back to them. So this has been our PLRC practice. Great, excellent. And I see there's a question from L. Siebert if you wanna come off mute. Hi everyone. Thank you so much for this work. It's so important. You can't really improve patient engagement if you can't measure it. So this is all awesome. I have a question about kind of incentivizing research groups to pick up methods like this because I think that can be challenging. I'm wondering if there's any interest in exploring kind of connecting level of patient engagement throughout the process to the outcome in terms of impact factor for the paper or media coverage. Because I imagine this would really directly show researchers how they're not doing patients a favor. It runs in both directions. We're both helping each other and we're both valued stakeholders. That's all. That's a great comment. Any reflections from the team? Wow, this is a tough one. Thank you for bringing that up. We do need a lot of help to help frame the practical use cases of these scorecards. And one of the things we've been looking into is to have solid case studies going forward with organizations using the scorecards or evaluating themselves with the scorecards. So I wonder if there's an opportunity there as well to really present this work to the public. Yeah. I would also add on the journal side, most of our societies have journals that they publish. I think it's an interesting question as well as does it get built into the evaluation of clinical trials for publication? So even getting beyond the impact factor, just in terms of looking at the actual method sections, method sections of most clinical research are not this detailed and don't include a section, for example, about how patients were meaningfully engaged. Maybe that's part of the push here as well in addition to getting it built into the way assessments of grants and other things are used. So great suggestion, Ellen. Thank you very much. All right. We are at 3.55. Maybe Liz, you could just pull up the slides again just so we could show people. So they have it in front of them, the next webinar, which I think, yep, there we go. So we've already had, this is our second webinar. The first webinar was on two of the other scorecards on research organization readiness and integration to the research process. Both of those, this one and the prior one, the recording will be available on our website by next Monday, Liz says, and then Thursday, February 2nd. Again, as I mentioned, we're gonna talk specifically about how to improve your score based on where you start, where you're going. We're gonna give a couple of examples that we'll talk through of maybe things that didn't go far enough and what could have been done to have the research group, in fact, get further along. So with that, I'm just gonna thank everybody, thank the team, thank both teams for really just extraordinary body of work. Again, incredibly amazed by the response from the broader community about how innovative and needed this work really is. So thank you all. Please join us for the next webinar on February 2nd, and we'll post the information about the recordings next week. Thank you all, bye.

patient-led research

scorecards

research collaboration

patient engagement

open science

governance

research impact

incentivizing

webinars