All right. Go ahead and get started. Delighted you could join us today. I'm Helen Burstyn, the CEO of the Council of Medical Specialty Societies, and I'm really honored to be joined today by our collaborators from the Patient-Led Research Collaborative to present our work on the development of these patient-led scorecards that you'll hear more about today. This is the third webinar in the series. The first two recordings are available on our website and, Liz, maybe you could put that in the chat as well. We're really excited today to kind of talk more about the scorecards, how they can be used to improve your score, and use them really as a learning opportunity. We're also especially delighted to have Susana Fox here, who's been one of the champions of this work and a member of our advisory panel who served as moderator. She's already popped her first comment into the chat. Please use the chat liberally. Let us know if you have questions as we go through it. I'll give some quick background and then turn it over to Hannah and Lisa, who will walk through some of the examples. Just briefly, this is a joint project, as I mentioned, between the Council of Medical Specialty Societies, an organization of organizations, an organization of the 50 medical specialty societies in the United States, and the Patient-Led Research Collaborative, really coming up with this approach jointly. This was funded by a Eugene Washington PCORI Engagement Award. These views are ours and not necessarily PCORI's. I will tell you they've been delighted by this work. Next slide. So what I'll do briefly is we're going to, for the agenda for today, some quick welcome and introductions. I'll then turn it over to the team to walk through a bit of the scorecards, case studies, and then all through this, Susana will keep a track of where we're going and make sure we get all your great questions answered. So next slide. I very much encourage your participation. As Susana has already mentioned, we had the first two webinars just filled with wonderful questions back and forth, lots of dialogue, and we expect nothing less for today. So next slide. So briefly, this is the project team overall. We're joined today by a couple of the members of the PLRC team. In particular, presenters today will be Lisa McCorkle and Hannah Way. And we're also, as I mentioned, delighted to be joined by Susana Fox, who's a member of our advisory panel, in addition to our staff, Liz and Suzanne. Next slide. This is just a wonderful advisory panel that we had as part of this effort. Importantly, more than half of the members of the advisory panel were patients, and we brought some really innovative thinkers to the table, both on the clinical side and the patient side, to really help us build, really, I think, as we talked about in our little prep discussion, a new baseline for what's expected in what meaningful collaboration really means between patients and clinicians and researchers. Next slide. At this point, I'm going to turn it over to Lisa to walk through the PLRC. Thanks. Thank you, Helen. Hi, everyone. I'm Lisa McCorkle, and alongside Hannah Way, we are two of the co-founders of the Patient-Led Research Collaborative. And just as some background, Patient-Led Research Collaborative is a group of long COVID patients formed out of the body politic COVID-19 support group in April of 2020. We came together. We're a very interdisciplinary team, really all motivated to conduct research for our own health purposes and researching our own condition and what was happening to us in a very uncertain time. And we had a lot of urgency for getting our data and our story out there. And a lot of our early data helped inform some of the early stories on long COVID, including one of Ed Yong's pieces. And from very early on, we consulted on a lot of the research studies on long COVID, consulted with government agencies. And since then, over the last two and a half to three years, have done a lot of our own research, as well as consulted on many long COVID studies. So we've come to this work from a very patient-led perspective. It's in our name and really learned a lot of lessons over the last few years on how to meaningfully engage patients and to really drive research from a patient perspective and move away from kind of tokenizing patients and just putting patients on panels for looks and not really engaging with them and using their input and moving more towards, you know, how do we actually use the vast expertise that patients have about their own conditions and bodies and ensure better outcomes for all patients. So we created scorecards alongside CMSS. If you could go to the next slide, please. And we covered these, the four of these, we did a deep dive into these the last two webinars. So really encourage you to review those webinars if you didn't join us or review those before. But I'll go through each of them very briefly here. So we had four scorecards that we ended up developing. One is the research organization readiness. So what this scorecard does is it measures the ability of the research organization to engage in meaningful patient partnership. We have different sub dimensions on each of these scorecards, which help to measure how meaningfully these research organizations engage with patients. For this specific one, one of the sub dimensions is a recognition of biases. So this is whether the research organization recognizes their own biases and implements feedback provided by the patients. Another sub dimension is the collaboration process. So is there infrastructure and a process in place for collaborating with patients? And then the knowledge in the disease subject. So how much experience does that research organization have in the illness? The next scorecard is integration into research process. So this scorecard measures how effectively patients are involved in every phase of the research process. Oftentimes patients might be kind of putting up the end as the writing phase or in really only one area. And what we want to see is more patients involved from hypothesis generation, study design, analysis, publication, and attribution. So we have sub dimensions for each of those. And the scorecard really looks at how much are patients leading each of those aspects at every stage of the process. The next scorecard is patient partner governance. So this measures how decision-making power and governance is shared between patient groups and partners. Sub dimensions of this scorecard include meaningful decision-making between groups. So for significant decisions like funding, study design, publication, do patients have a say and is the decision-making process well communicated even before a patient group gets involved with a partner group? Another sub dimension is accountability between groups. That's a shared understanding and written agreement of rules of engagement and culture with defined consequences of not following through between both groups, between both the patient group and the partner group. And lastly, we have the patient burden scorecard, which measures the degree to which patient burden and associated trauma is addressed. So sub dimensions of this scorecard include accessible engagement. So are patients able to contribute in a way that's accessible to them? Another one is trauma-informed practices. Is it a safe, mutually respectful environment where patients receive sufficient resources and support? Responsiveness to patients. So are there clear, accessible, and safe channels for input with also acknowledgement of the impact on the patient of providing that input? And then compensation. Are patients compensated fairly for their expertise, their experience, and any anticipated expenses as well as the harm that providing that input may cause? So these are the scorecards that we published. Definitely take a look at the links that we put in the chat, both to the scorecards themselves and the prior webinars. And today what we really wanted to do was kind of do a deep dive into different case studies and use cases for each of how to use these scorecards, how different audiences can use these scorecards. And so we'll go a bit deeper into that. But just to give kind of in case you missed some of the previous webinars, the way that we scored the, created the scoring model for these scorecards is on a scale from negative two to two. And the reason that we did this was that we wanted to create a new baseline of acceptable collaboration. And that's at the zero, that's the yellow zero in the middle. And we wanted to do this because negative two and negative one, there is some patient engagement in those, in those scores sometimes, especially a negative one. But what we often see is a very tokenizing level of engagement. And we want to move away from that being considered as acceptable and move towards a new standard, a new baseline of what's considered acceptable to engage patients. So when we look at the scorecard, you're going to be rewarded if you're even better than the baseline at that, better than that acceptable level, you're going to be rewarded with more points, one or two. And then if you're at the level of tokenizing patients, only including them, you know, at certain areas of the research process, not addressing the harms that are put on patients, you're going to get a negative score. So we did this so that hopefully research partners would be incentivized to move towards that at least zero, but ideally the one and the two scores. And I will turn it over to Hannah Wei to go through some of our first few case studies. Thank you, Lisa, for that wonderful overview. I'd like to talk a bit about how we thought through use cases for our scorecards. And obviously, our intention is to create a shared language between groups. So these are patient groups, potential research partners, and funders. And so the first case study addresses how patient groups can use our scorecards to evaluate potential partners. And we say evaluating, but also the scorecards can be used proactively before the engagement, during the engagement, when there's a conversation about feedback, and also retroactively after a collaboration, when there's a conversation about how things can move forward and parties can improve for next time. So from the patient group's perspective, we thought that, you know, when patient groups are considering partnering or are in the middle of a partnership, they can use the scorecards to essentially help them negotiate for needs in the different areas that Lisa succinctly described. So when a collaboration is in its planning phase, all four scorecards can help guide discussion around some shared expectations. In particular, I want to highlight the research organization readiness scorecard. That one, I think, is a really good scorecard to help determine whether or not you want to partner in the first place. And if there's no infrastructure to accommodate patients in the first place, and it's very hard to work out the other details, the integration into the research process scorecard can help patient groups be even aware of where they are in their own resources and how they can get involved with the research process and use that as a way to open discussion for opportunities to contribute at every stage of the research process. Also, we really enjoy having the patient burden scorecard that Anisha helped to develop for us. That one can be used to give feedback to potential partners to help them understand what things feel like and what are the burdens that the patients take on in the process of partnering for any research project. The patient partner governance scorecard is especially helpful to assess power dynamics before or after the collaboration to help give suggestions to how to move forward. Next slide, please. So, ideal outcomes of this use case is to have the patient group be better equipped to decide on who to partner with. A patient group knows which areas to ask for in order to get more decision-making power, to have more resources and accommodations, and quickly align with expectations from the other side going to collaboration. The last one is patient group has more resources and support to help themselves become a more robust organization. And we always, you know, as a patient-led research collaborative, we're still struggling with this part as well. So, we know this is a process. And I want to shout out a great example of a partner that we have, the Light Collective, who's been already using our scorecards to move forward on conversations with potential partners. The Light Collective is a coalition of experts in digital rights who are also patients. And they do wonderful work in that space, putting out lots and lots of resources for patient groups to understand what their rights are and how to negotiate for them. And so, we decided to swap resources, and now they're using the scorecards to help land better partnerships. Next slide, please. So, getting into the other side, the case study where the research partner can use the scorecards to assess themselves, we imagine this use case to be very relevant when the research team has some resources and capacity to improve on patient engagement and would like to know where to set aside these resources, ideally before the collaboration takes place and alongside conversations with the patient group. Suggested use of these scorecards, again, all four scorecards to help guide the process of grant writing, establishing a mutual collaboration dynamic even before the work begins. And we find that especially helpful. As PLRC, we always try to be as early as possible in collaborations so that we have our resources set in place in the grants and that we know exactly what we're getting into. And so, having the research organizations do this planning proactively with us is incredibly helpful, and we'd like to see this happen more. If the study is already underway, the research group can use the research organization readiness scorecard and the patient partner governance scorecard to help them assess the level of engagement that they're capable of and opportunities that they can improve. And so, that will help have more meaningful meetings and discussions with the patient group and hopefully have a more sustainable collaboration if something goes wrong. The research team can also use integration into the research process scorecard and patient burden scorecard to find ways to plan for patients' accommodations and such. The integration into the research process, when sometimes I think that certain groups want patients to be involved, but they don't know how they can come in. And so, I think this scorecard can give them ideas of asking questions like, hey, does your team have an expert in XYZ, say like somebody who can do analysis and clustering, and be able to bring them into the analysis phase and be involved in more ways than usual. Next slide, please. So, ideal outcomes for this case, we would love for the research partners to apply for funding that prioritizes the scope and budget for meaningful patient involvement. As we mentioned in the previous webinars, it's really important to incentivize this dynamic from the beginning, from the funding perspective as well, because oftentimes we see that patient engagement becomes an afterthought, and there's no budget for it, and that's really awkward for the patient group sometimes to step in and say, hey, we want to do this, but we don't want to do this at the expense of taking care of our family and having another job and whatnot. And so, the research partner can use these scorecards to proactively address the needs of the patient group, and so hopefully make the study stronger with better engagement and better data, more meaningful. A great example of this, I want to shout out our partners at Long COVID Physio and the Episodic Disability Team that myself and Lisa are also on. And so, I'll turn the mic to Kelly O'Brien to talk a little bit about what she's doing and how they're planning to use the scorecards going forward. Thanks, Hannah. Thanks, Lisa. Yeah, I've had the opportunity and the pleasure, as Hannah mentioned, to work on a project funded by the Canadian Institutes of Health Research, or CIHR, exploring the experiences of episodic disability among adults living with long COVID. And this is a collaboration with Hannah and Lisa, who are part of the team from PLRC Long COVID Physio, led by Darren Brown, who is co-leading the study, along with other community experts and members from Long COVID Ireland, Long COVID Support, and the COVID Long-Haulers Support Group Canada. And I think that what's really fantastic is that we've, I think, benefited from having these scorecards released. And then, given that we've been working together as a team now for about two, going on almost two years, being able to have the scorecards released, it allows you kind of retrospectively to look back and evaluate, how have we been implementing the benchmarks within the scorecard across the four different dimensions? What are we doing well? What can we be doing better? And how might we use the scorecards moving forward? So I think, as a research team, just really continually understanding and revisiting, who you're working with as a team evolves, their expertise, capacity, what their interests are, and what their limitations are, and the potential unpredictability or fluctuating nature of one's health condition. And so, in terms of the patient burden process, having a structure in place where people can step away or return, depending on their health and capacity allows. From a governance perspective, I think continually revisiting our process as a team will be really helpful. Many establishing terms of reference, but not just leaving in that as something that happens at the beginning of the project, but continually going back and using these cards to check in on our process. How is it going with respect to a collaborative decision-making model? And how is the team working together? And having a process set up where there may be aspects of the project where it makes sense for the community to just meet independently as a team and have that safe space in order to flush out analysis or interpretation of the data. But then also having those mechanisms of the team coming together as a whole, community experts, patients, and academics to really sort of prevent, I think, those silos from developing within a team, but also allowing largely teams to come together and have that cross-collaboration and decision-making process. I think then lastly with patient burden, I'll just mention is, I think what reflects to me and thinking about is what practices are in place for members of your team in order to support those who may be going through different periods of fluctuating levels of health, but then also what practices and procedures and supports are in place for the research participants. And I think this is where the scorecards can be reused actually quite collaboratively with research ethics board or institutional review board processes as well to make sure that those processes are in place. So overall, I think a real strength of these scorecards is they're not specific to one target population. There's opportunity for many of us who work across different patient populations to implement them in our work, which is really exciting. It's something where they can be continually revisited throughout the research process. As someone who also teaches in graduate programs and in the physiotherapy program, we can use these in health professional curricula and graduate trainees. So they're learning early on how to embed these principles and values into their research proposals. And then as for folks who are in the position of launching or reviewing funding applications for graduate trainees or operating grants, embedding these scorecards into that structure for individuals as they apply for funding, but also in a way that those are the way those applications are actually evaluated as well as part of this criteria would be great. Thank you so much, Kelly, for all your insights and thank you for the wonderful collaboration that we have. Turn the mic now to Lisa to talk about, last but not least, our case study for funders to use these scorecards. Thank you, Hannah, and thank you, Kelly, for providing that. So one of the use cases that we were considering when developing this was how can funders use these scorecards? So in this example, let's say a funder wants to support research projects that are directed by patients or at least have meaningful engagement of patients to really ensure the best and most patient-focused outcomes. And in this example, the funder would have the ability to set criteria for rating proposals, they can add patients to their review panels, and they're willing to set aside adequate funds for meaningful patient engagement. So in this type of scenario, which is kind of the ideal scenario for funders, they can really use all four scorecards for many of these aspects. So they can use the four scorecards to develop criteria when they're evaluating proposals. They can share the four scorecards with applicants as they're developing the proposals. It's kind of like a hint, hint, this is what we're going to be evaluating your proposal on, and this is what we value. Funders can evaluate proposals based off how well the proposals score on each of the four scorecards if they don't have a separate criteria for it. As funded projects launch, the funder can check in on each project to ensure the execution of a patient-led research plan by evaluating the projects using all four scorecards. I think this is an important aspect because there's only so much that a proposal can get across. You can say a lot in a proposal, but what's actually executed can be quite different. So having that follow-up and making sure, checking in and making sure that that patient engagement, that patient-led nature is actually happening is a crucial part of a funder's role here. And then they can also use the research organization, readiness, the patient partner governance, and the patient burden scorecard specifically to evaluate their own engagement of patients in the review process. So ideally a funder would have on their review panel of proposals, patients themselves, so that patients can be in charge of deciding what research is being funded, what is being prioritized, because that's really where a lot of the decisions are happening and where we can make sure that research projects that are funded are of what's important to the patient community. Next slide, please. So that gets into kind of the ideal outcome. So by providing funding for patient engagement, the funder is really incentivizing that patient-led research. It's an indication to researchers that if you have the capacity and set aside funds to properly and meaningfully engage patients, we're incentivizing that. We're providing funding for that. That's something that we value. Additionally, in evaluating proposals, the funder can prioritize projects who have better practices with this. So it's not just providing the funds for it, but it's incentivizing and prioritizing projects and researchers who value patient-led research themselves and have a plan and capacity to meaningfully engage patients. And ideally a funder can help give guidance on how to meaningfully engage patients, especially with some researchers who may not have done this type of work before, having a funder to go to for advice on how to do this work, how to meaningfully engage can be very helpful if the funder can be kind of the expert in this and help guide the research projects that are funded through this. And lastly, an ideal outcome is that patients have decision-making power in what projects get funded. That's another ideal scenario here. An example of all of this is a very special example, and it's our own, which is our patient-led research fund, which was a $5 million biomedical research fund looking into long COVID and ME-CFS and associated conditions. I'll put a link to that in our chat. So you can take a look at what we have funded and some of the criteria we set. So this was something that we did in fall and summer of last year, 2022. And this is while our scorecards were still in development, but we still use the key themes to develop criteria that would reward proposals who had researchers that had lived experience of illness, so was truly patient-led. And we would reward proposals that based their research question and priorities of the research community, who developed proposals in partnership with patients, as well as who had a plan to meaningfully engage patients throughout the study if they were chosen, if they were funded. So those were some of the criteria we had and part of how we chose the research studies that we ended up funding. And additionally, for the projects that we chose, if the projects did not already have patients on their study design committees, we are providing paid patient representatives for two to four hours per month to advise on study design, analysis, writing, and distribution. So this really enabled the research projects to not have to consider the cost of patient engagement in their own proposal budget, as we as the funder actually provided it for them. An additional aspect of a patient-led research fund that would be ideal for funders to incorporate is that the entire panel that chose which research projects to fund are patients themselves. So it was an entire panel of people with long COVID and ME-CFS. So having funders incorporate this type of meaningful engagement and having patients on their review boards is another really critical way of ensuring patient-led research and that the research that is done is really the priority of the patient community. And I think we are ready to go to Q&A. So I'll pass it to Susanna, I think. Yeah, yeah, that's me. So thank you so much. And Sarah Rigger, never apologize for having a lot of questions, because we love questions. And I'm actually going to start with the last question, that you asked, because I think it's a good one just to level set. Sarah wrote in the chat, I know that the first webinar was dedicated to terminology and definitions, but I haven't been able to watch it yet. Could you please state your definition of patient-led research? So I'll open that for Hannah or Lisa. Yeah, thanks for that question. I think in our definition, it's patients being at the driver's seat every part of the research process. And this ties back into the integration into the research process scorecard. If you look at all the sections that were acceptable, those are places where patients have the agency that are equal or are directly leading that part in the study itself. And we apply that to every process in the study. So from the hypothesis stage, for example, patient-led means that patients themselves with lived experiences come up with the hypothesis and consult with the researchers. And then researchers might be like, well, this is really interesting and it speaks to our expertise. We'll investigate that one. And so that to us is patient-led. Whereas in comparison to something that's more patient-involved or patient-consulted, that's when the researchers give us a list of, hey, do you think these are interesting or they already have a hypothesis formulated? We see this quite often. And they should give it to us like the night before and say, hey, can you let us know if these are of interest to your community? We're like, well, you kind of gave us not much room to work with. I hope that answers your question. You said you have anything to add to that. And what I was going to say is one of the things I love about Zoom is that if you raise your hand, your little square will pop up to the top and we'll be able to see that you raised your hand. So if you have a question you want to pop into the conversation, please do raise your hand using Zoom. The other question that I think garnered a really interesting conversation in the chat that I just want to bring forward for everybody to talk about is another question from Sarah. And she said, do the scorecards also address and acknowledge patient's biases? And Lisa, you wrote a great response to that, that we do have an unpublished patient group readiness scorecard, which I think people are going to be really interested to see when that is developed. But this is a tension between researchers who might have some skepticism and what this project was about, was about saying like, let's talk to patients and this is a patient led project. But Lisa or Hannah, could you begin to talk about patient bias and that challenge? Sure, yeah. I think, I mean, I think it is valuable for patients to acknowledge and understand that they have biases, but at the same time, part of the whole motivation behind these scorecards is to let that bias come out of it, I guess, because that bias is based in our experiences and based in the, you know, just the lived experience of being a patient and living with an illness that too often is not respected in these research spaces. So I think it's just a kind of a tricky balance of how do we make sure that we're not gonna penalize patients for having that quote unquote bias that's too often not in these spaces, but is very much needed, while also kind of acknowledging patients, you know, I think need to understand where researchers are coming from and maybe understand all of the different biases in play and what they maybe don't have a background in and kind of acknowledging where their knowledge base may be lacking that could be paired well with a researcher. And if I could just add to that, Suzanne, I think it's a really interesting question. And this whole concept of what's bias in any research really is a question of sometimes bias is expertise, right? So we say somebody is biased, but it's actually because this is their area of knowledge. They know this area really well. They may not be biased. They may just be bringing that special expertise to the table. So I think in this context, what we're saying is the patients are bringing that lived expertise to the table. So I'm not sure it's necessarily bias, but I think it's analogous to what we often talk about for any researcher. So I just want to normalize that, that it isn't just patient bias. It's really bias expertise for all of us. I love that. Thank you. Hannah, anything to add to that? No, that was great. Okay, cool. Sabrina, I'm gonna call on you. Hi, I just wanted to talk a little bit about that tension, which I think is what's coming up around the patient piece. And I think that what you guys are saying about the bias piece is really important to that, but I think it's almost bigger than that. And in the ME community, it's something we've had to kind of figure out and address. And I don't think we've nailed that yet, but it's a work in progress. I think sometimes it's about more about representation. So it's the readiness of representation. Are these individuals ready to represent the scope of an illness, for example, versus their personal anecdote? And it's not that their anecdote isn't incredibly valuable, but if you only get to have two patients sit on a committee and one has a very particular way of viewing the illness that really excludes a large majority, there can be harm accidentally done in those cases. And so I guess for me, the patient scorecard that I'd love to see will kind of address that a little bit because we've certainly seen amazing patient representatives come out through this long COVID experience, the likes of which are on this call. But we've also had some people who've unfortunately caused some harm accidentally because they didn't understand the connection with ME or they didn't understand the full breadth of the experience. So how do we kind of measure that, look for that, start a conversation about that? I think that allows us to kind of tackle that tension point, but in a way that's respectful. Yeah, absolutely. No, that's a really great point. And so part of this scorecard that we developed had that the patient group or the patient that is going to be part of these research project is accountable to the patient community. And that's something that, I mean, one of the reasons we didn't publish it was it's like, how do you measure that, right? Like as I could say for myself, yeah, I'm totally accountable to the patient community, but the patient community might be like, you don't represent me at all, right? So it's how do you measure that when you're kind of self-identifying there, but that we totally see that dynamic and completely agree that that's a really important thing we need to measure when thinking through patient group readiness for being on these types of projects. I think there's something here that could be expanded and explored where there are older patient groups, patient groups that have a long history of working with researchers, that there might be something that we can learn from them and how they conduct the work. For example, the National Breast Cancer Coalition since the 90s has had this essentially a school that they send survivors to, to learn how to sit on a congressionally appointed committee. And there's other groups like that in patient communities. And so pooling that knowledge, and again, like breaking it out of the silos of the patient groups, this is something that's very well established in cancer, but let's make sure that's established all across every condition. Which actually brings me to another question from Sarah. And it was her first question is how might we use our energy to spread this information across the world and across many conditions? I'd love to hear from anyone on the call. You can raise your hand and be unmuted, but I'll start with Lisa, Hannah, Helen to talk about what are the ways that we can make sure that this is really going out to as many people as possible? I think what Kelly brought up earlier was really fantastic, having this taught in training programs and in schools. And I think somebody else, Linda, mentioned this as well, to just have people understand that this model exists and you can make use of it and make it your own, I think is really powerful. I think for us, the biggest challenge sometimes is to work with people who are not in the community and work with a certain partner and they already have a set way of doing things. They have their governance in place and we're just kind of coming in and consulting and trying to do this one part of the study. And so I think it's really important to start from the beginning and go, well, we need to negotiate this type of dynamic from the start. And there was another question that was related that like which types, groups of researcher do we find easier to work with? And I'll go directly into that. So like kind of cheat and answer that question as well. The great opportunities that we've gotten where we were really sitting at the table were from PIs like Kelly herself, who understand this dynamic of fair collaboration. And so I think what's really important is to get the buy-ins from the people who are making the decisions, the PIs, the funders in order to, and that way they can think in this model and help implement it in the projects that they manage. Lisa, Helen or anyone else, anything to add on that? Yeah, I was just thinking, what Laura just put in the chat, work with community engagement offices and departments at research universities. I think is a really great idea. I think also, I mean, a lot of this is so reliant on a funders buy-in, you know, and we've talked about that in previous webinars for in order to even, you know, meaningfully engage with patients, you have to have the capacity, you have to have the resources. And so that comes from the funder. And so I kind of think targeting funders, getting their buy-in on this and, you know, having this be the new standard for how they choose what research projects to prioritize, you know, might be a tough way to go, but I think it's gonna be the most successful way to go. And that's at all levels. That's at, you know, private foundations, that's at NIH, that's at, you know, government agencies. And I think there is appetite for that. And we just need to have kind of a coordinated way of making that point and getting their buy-in on this. And Linda, I was just gonna call out and see if you would join the chat because your answer in the chat was so great. I wanted to make sure to bring it forward. So please share. Thanks so much. Yeah, I just wanna say, I mean, from my perspective, I'm a patient and I do a lot of patient engagement work, but I'm also a retired healthcare professional, retired healthcare executive. So I see it kind of from the organization side, but I also see it from receiving the care side. So I sit in the Institute Advisory Board for MHA, the Institute of Musculoskeletal Health, and arthritis and do a lot of work. And I mean, we co-created the PIRA, you know, patient engagement modules that some people have mentioned there. So I really think it should be shared with, you know, I'm just talking about CIHR right now from a Canadian perspective, all the research institutes nationally in Canada, I think patients should share it with the Institute Advisory Boards. All of them have the patient representative on there, but we should really discuss it and use it and say it really should be part of any funded research. I'm a patient co-applicant in a CIHR, you know, multimillion dollar research project. I'm gonna be bringing this because I'm co-chairing the patient engagement group of our, you know, research. So I think we really have to work together with those that are on boards of healthcare organizations. They can be a very powerful tool, especially if there are academic health science centers that work in conjunction with research organizations and also take some of the learnings from some very advocate powerful groups like community advisory committees and hospitals and patient and family advisory committees and hospitals. I live in Ottawa in Canada and I'm gonna be sharing this with a lot of the patient advisory committees that I know and I work with in different hospital settings, which in conjunction then work with their foundations that sometimes fund them, but also work with the research institutes and the universities that fund them. So I just think the more that we can do and the more you allow this public document and this scorecard to be shared broadly, the more powerful and the good way we can to be working as partners to let people know. I so appreciate you. Thank you so much, Linda. And if I could just add to that and Linda makes an important point, but it puts so much of the onus on the patient community to spread it. And I think that one of the sort of, I hope, you know, secret sauces here is the fact that we're doing this collaboratively. I think it can't all be on the patients. It's gotta be the medical community, the research community has to push this as well. So I think- I totally agree, Helen. Yeah, I think I completely agree in that my caveat was not that it should be the patient at all, but I think if we start somewhere and then we have powerful partners that are also healthcare professionals or executives or administrators or whomever, they can bring that along and it has to be a co-partnership, absolutely. Yeah, absolutely. One way that I'm taking this home as a researcher is that I am now using the scorecards to look retrospectively at projects that I've been involved in to say to myself, I consider myself someone who's really, you know, interested in patient-led research, but goodness, my scores are actually not as good as I hoped they would be looking retrospectively. I'm involved in a project that's gonna be launching in about a year that is focused on teens and young adults. And so I did wanna bring up that. One thing I noticed when I got into quality improvement work was that pediatrics is a leader in quality improvement, you know, infection control, all kinds of really interesting stuff about improving the care in hospitals. And so I will say personally, I have seen some really forward-leaning work in pediatrics and in anything having to do with teens and young adults because there's this sense of there has to be a different way of doing things. We have to involve teens and young adults in designing the research because how can we know what the right questions to ask would be? And so I wanted to open up and see if there's other researchers here who wanna talk about how they're gonna use this work, you know, in their own projects going forward or retrospectively how they would score themselves. And yes, I'm putting people on the spot to raise your hand and that's okay if you're not ready to. So I'll leave that hanging. You can think about whether you wanna raise your hand. So I also wanna say, I loved what Kelly O'Brien said about the potential and limitations of patient survivors and caregivers participation. Because I think that something to really keep focused on is the huge potential, how this is gonna be something that can be a competitive advantage for researchers who want to move faster. I've got great stories of clinical trial recruitment going much faster because there was a patient-led design. And so I think that that is really important. I wanted to ask Lisa and Hannah, as you were designing it, did you have a vision of what the potential future state could be because of this work? We outlined a few of these in the case study in our ideal point form. So I think overall it's just more collaboration between groups and just more understanding of the patient perspective when entering collaborations. And also just like, I see some comments on this already, but harnessing the capacity of patients in a way that makes sense. And so if we have, what's fortunate about our space and along COVID is that, and fortunate or unfortunate is that we have quite a few healthcare workers and folks who have previous academic experience. And so these folks can jump into analysis right away and move research along faster. And I think that applies to more than just patient-led research collaborative, applies to a lot of other groups as well. We all have had, especially in Chronicles, we all have had fantastic expertise and careers to bring into these projects. And so part of the big vision is to make sure that everybody's expertise are all included in moving forward and in having competitive advantage towards studies. So there's a great question in the chat that I want to bring out and that Angie writes, I'm hoping it's okay to share your scorecards at society and research conferences. I'm helping as a patient partner with registry development plan to use that opportunity to share some of your scorecards. And the answer that Lisa immediately put is, yes, please share widely. We want these to be really a living document. We want feedback. And what we're doing is taking all of the chats from these three webinars, we're taking the recordings, we're absorbing all of the input that has come from this community and that will continue to come from this community. So I don't want to speak for the group, but I think our radar is really wide open and want to hear from everybody and really want this to be shared widely to get the most input possible. Completely agree. And if we could help at all on the medical society, the registry research front, let us know. And I, since I was talking, I didn't see something that came in through the chat. Helen, I see that you're writing something about- It's a great question. Could you call that up for us? It was Laura's question about, you know, what concerns, pushback or challenges do you think you'll get from funders and how can it be overcome? So I thought it was just a great question and Hannah answered as well. You know, they're not validated yet. We'll work on that. I still think they have a huge amount of face validity here, but you know, one thought was I think they're going to try to figure out how this fits into their overall rubric of how they decide on awards, how much this gets assigned overall compared to other pieces of it. I hope at least we can encourage them to start testing it and just kind of using it as sort of a process of is this informative? Does this add additional value? And then eventually can they build it into their rubrics for funding so that you have to meet, you know, at least that zero and hopefully the plus one or plus two to get funded. Okay, great. And I see that Sarah, who's coming to us from Sweden has a question. So Sarah, you can unmute. Thanks, hi. Yeah, it's pitch dark out here. It's like middle of night in the rural part of Sweden. But I was wondering when you've done these case studies, how have you used the scorecards? Have you added the scores up? Have you used them individually? Have you made averages? How have you used them practically? Personally, I actually scored myself on a project that I'd done in the past and got a score. So I tried to use the scorecard as I would if I were going into a new research project. But, you know, it would be interesting just as a discussion guide to say, you know, let's put this out on the table and have it as a discussion guide of whether or not it's used as actually a formal scoring tool. What were you guys thinking as you were developing it? Lisa or Hannah or Helen or Suzanne? Yeah, I'll speak from my perspective. We kind of didn't want it to be added up because really ideally someone is scoring on the plus one, plus two, or at least zero on all of these efforts. And so, you know, we don't want a situation where they could be ideal in one area but they're not paying patients at all, right? Like that's not actually ideal. So that's why we didn't necessarily give instructions for adding or even averaging. It's really because they're also individual and ideally you are scoring on zero to two on each of the sub dimensions. I think that that's why we didn't give instructions for adding up or averaging or anything like that which may be confusing and maybe we need to give more instructions like specifically to that. But that was the, you know, what we were thinking is that we didn't want someone to be rewarded for having really good, but then they're like awful on these other areas and then they look like their average. Completely agree that it should probably best be used as a discussion guide. Yes. I see a risk though with the name of a scorecard and that organizations and groups will be starting comparing themselves and sort of competing in this and using it differently in different ways because that's, we know that often things are turned out, tend to be used differently than they were intended to be. Interesting. Yeah. They're also an improvement tool too. I'm sorry, Lisa. Yeah, no, I mean, I see like a version where it's, you know, maybe we do have it so that it's added but if you get negative one or negative two on any dimension, you like automatically actually get negative 10 or something, you know, like somewhere, somehow where we have a scoring system that makes it so that people are really penalized for being in that negative range. But yeah. Some sort of weighting system for different cards maybe. Yeah, I think that's a reasonable way forward, yeah. But I think this will be a living document. I think it'll be evolving. Yeah, thank you for, we're running out of time. So I should say that we're just gonna have some closing thoughts, I think. Yeah, I'm just gonna add one more thing. So when we developed the scores, we were actually, we were inspired by disaster planning and disaster planning, there's like several domains of resilience for cities, for example, if you're assessing a city, whether that's, they're resilient or not. And so the domains that kind of are independent and that's why we structured our scorecards in a similar way because we figured out that, you know, if somebody is really good, as Lisa mentioned, somebody is really good in one area, or somebody is really good in one area doesn't mean they're really good in the other area, but it's good to get that feedback to know that I can improve in that area. All right. Awesome, thank you so much. That's a great thing to bring in. So I don't know if Helen or Suzanne wanna close us out with the last thoughts of where people can find more information. Absolutely. So I've posted it in the chat as well, but on the CMSS website, we will have all the recordings available. The first two recordings are already available and then we'll have this one shortly, but we'll also be contacting all participants with information to send you information as well as to get additional feedback and as well as an evaluation of our webinars because we're definitely interested in feedback. As we've said, this is, these scorecards are living documents. So please use them, share them widely and continue to provide your input to us. And also just point out that the scorecards are available there as well and that we also have a draft playbook that we'll be kind of bringing you back out as well about how you use them in this overall effort to have your work be much more collaborative with patient researchers. So thank you. And thank you, Susanna for a great job and Lisa and Hannah. It's a wonderful webinar. Great. Thanks for spending the hour with us, everyone. This has been great. Thank you. Bye. Thank you. Take care.

patient-led scorecards

Council of Medical Specialty Societies

Patient-Led Research Collaborative

patient collaboration

research integration

patient partner governance

patient burden

case studies

research outcomes