Hello, everyone, and welcome to our webinar today. I'm Helen Burstyn. I'm the CEO of the Council of Medical Specialty Societies, a coalition of 45 specialty societies working across the House of Medicine to improve care. And we're really excited today to have this third in our sixth webinar series that we're sponsoring and collaborating with our colleagues at the AAMC with generous support from the Gordon and Betty Moore Foundation to really focus on how we can advance clinical registries to support both pandemic treatment and response. We launched this session, these sessions at our meeting on May 1st, and all that information, the recording from that session is available on our website. And we've also got all of the prior slides and recordings from the prior webinars as well. So we encourage you to take a look at that offline if it's helpful to you. And really, our goal here is to, in some ways, not let any crisis go to waste and think about how we can use this as an opportunity to transform clinical registries, clinical research through rapid cycle learning, development, and technology. And so these sessions will continue for the rest of the summer, and we'll have all the details available to you on our website. We would very much encourage you today, as you're listening in and have questions, to go ahead and enter any of the questions into the question box on the right-hand side of your screen. We'll come to those at the end of the discussion. We'd also encourage you to continue the conversation on Twitter. We've created a hashtag for these webinars, hashtag CAHPSCOVIDregistries. And feel free to follow at CMSS or myself for frequent updates. And we'll give you an update on the next one coming up at the end of this session. Also note that the right-hand side of your webinar box, you've also got the CMSS handout section that includes the combined PDF of the slides from the speakers today. Prior slide decks are available on our website. We didn't have that quite ready the last time we met. So just last housekeeping at the end, you'll get a short evaluation, and we'd love to have your input as we try to make these better and hope that they're contributing to some shared learning. So without further ado, I'm going to introduce our panelists, and then we'll run through presentations across our panelists, and then get to you for some great Q&A, I'm sure, as you could tell with a very provocatively titled webinar on reflecting on our COVID-19 failures, what's a new vision for integrated registries. So just going through the list here, we've got Liz Garrett-Mayer, who is the Division Director of Biostats and Research Data at ASCO. Cliff Koh is the Director of the Division of Research and Optimal Patient Care at ACS, as well as a professor and surgeon at UCLA. Greg Martin is a professor of critical care medicine, and he is also the incoming president of the Society for Critical Care Medicine, in addition to Director of the Division of Pulmonary Allergy and Critical Care Medicine at Emory. And we recognize he's pretty busy, sadly, right now, as they're getting pretty hard with COVID. So thank you so much, Greg, for joining us today. And last, but certainly not least, we'll have Michael Howell, a principal scientist at Google and a pulmonary doc himself, kind of take us home with his vision based on what you've heard from our esteemed crowd of what they've tried to do to stand up registries, retrofit registries to address what's happening in COVID. How can technology and sort of the socio-technical aspects of what we need to fix help us be ready for this continuing or for the next crisis, or just to make care better in general? So with that, I'm going to turn it over to Cliff to begin the webinar. Thanks, Cliff. Great. Well, thank you, Helen, and thank you, everybody, for joining. And I really look forward to a great session together. There's some great speakers and great knowledge, and I'm hoping that we can come up with some solutions. So I hope everyone's staying safe and doing what they can to really help to end this pandemic. And as we're going forward, and we've been three or four months into this, and it looks like it's going to continue, I think that a lot of organizations and groups have really tried to do their part in collecting data and analyzing and figuring out what the science is telling us and where to go from there. And as we're into this, and it looks like it's going to be further, I think the group was thinking, how would we evaluate how we've done so far, and what can we do to make things better in terms of data as we go forward? So this new vision of registries and integrating them is something that I think all of us believe is the right thing to do. And hopefully, with these presentations and the discussions, we can get to that point and come up with some great ideas. So Julia, may I get the next slide, please? So as Helen said, I spend time at the American College of Surgeons. For those of you who don't know the College of Surgeons, it's the largest surgeon membership organization. It has 80,000 or so members. It's worldwide. The college was actually started 100 years ago and was founded on principles of quality of care and education. Surgeons would meet and figure out how do they learn from each other in order to take care of patients better. And this has largely progressed over the past century or so. And so now, in 2020, the largest division within the college is the quality division. Almost half the staff in the College of Surgeons work on a quality program. There are over 3,000 hospitals in the country that participate in at least one of the college programs. And there are 15 accreditation, or we use accreditation and verification interchangeably, those terms. And so you may have heard of the Trauma Centers Program, Level 1, Level 2 trauma centers, or the Cancer Center, the Commission on Cancer, or Bariatric, or Children's Surgery. There's a number of these verification programs. But we also have, and more apropos to this session, is we have clinical registries. Probably the most well-known is NISQIP, the National Surgical Quality Improvement Program. And so from that standpoint, I'd like to focus the rest of this presentation, short presentation. Julie, may I get the next slide, please? So I think as a starting point, why are good data essential? And why do we need good data? So the first three things of, we need to have information. We need to know how we're doing. We use data for investigation, figuring out how to do things better, and where the research leads us. And also, a lot of things we do at the college, and others do, is we use it for improvement. We know where we are. We know where we want to get to. Are we getting better? Are we getting worse? Are we staying the same? And so these are three good reasons to have really good, accurate data. And the last two things on this slide are, without data, we are basically flying blind. And unfortunately, too often, early in this pandemic, I think a lot of us felt like that. We didn't know a lot of things. We didn't know what we didn't know, but we knew a lot of things that we didn't know. So that's really important. And the last thing, I think all the speakers agree, that if we have garbage in, we get garbage out. If we have bad data going in, or it's not standardized, or it's not accurate, then we don't know what we're getting out when we're doing the analysis. So these five things are really important, as we think of data, and registries, and where we are, especially in the pandemic now, where we were in the beginning, where we are now, and where we will be going over the next several months, or years, potentially. Next slide, please. Thanks. So at the start of the pandemic, and from the view perspective of the College of Surgeons, we were hearing reports, there's international studies showing extremely poor surgical outcomes. So you may have heard that the mortality rates were three times, four times, five times higher in elective surgery in patients who were known to have COVID, or in patients who were not known to have COVID, and were asymptomatic, and then showed their symptoms post-operatively, and then the outcomes, the mortality rates were sky high, and it really scared a lot of folks around the world. If we add that to starting in March and April in the US, where when we didn't know a lot of stuff, and we were like, all right, we need to have PPEs, and there was a shortage, and we needed to have ventilators, and there was a potential shortage, and we needed to have ICUs, especially ventilated rooms, and there was a shortage that in the US, it was a pretty agreed upon thing to try to triage and stop non-operative, non-emergent operative care. So other than trauma and emergencies, that we should consider triaging them and trying to prioritize things that could wait, so that we could conserve resources. And this was government, the Surgeon General, the College of Surgeons, anesthesia, nursing, all agreed that that was the right thing to do. At that time also, the college, we were hearing from many of our hospitals that they didn't know how they were doing with COVID, what their real numbers were, who was sick and who wasn't, what was helpful and not helpful in terms of treatments, who to test, who not to test in terms of symptoms, what their outcomes were in terms of how they were doing, who's leaving the hospital, who's not, and what they should be doing. And they didn't know operatively in trauma and emergencies, and they didn't know in the non-operative patients and the patients who were not undergoing operation. So multiple, multiple requests came to the college that they need help and can somebody help to organize data collection so that they can start seeing how they're doing. And to be honest with you, the college in the beginning was like, well, there's going to be a lot of efforts in this. We should kind of stay with surgery and not do much else because we'll rely on others. And hospitals were telling us we need to have some data now. So we very quickly did two things. Number one, we put into our registries now, like the NISPPIT program or the trauma registry, we put in COVID variables into the current registries so that if somebody were having surgery or an emergency trauma operation, that we had COVID related information there so we could put it into the context of what was happening with the pandemic. And we felt that as hospitals were starting, when the pandemic was going to subside and hospitals were going to start to increase their surgeries, all told that we would have in, for example, in NISPPIT, some COVID related information. And that's happening now in many areas. And so we have that information. However, hospitals are also asking, you know, can we create a basic COVID registry for non-operative patients? Because as I said, most patients we're not undergoing operative surgery to know. So next slide, please, Julia. So what we tried to do and what we did do is this, is that we just launched very quickly a very kind of basic COVID registry for hospitals. And at that time, most of these hospitals were in New York that were, as we all know, was one of the initial hotspots. And so we heard about this kind of the last week of March, got all these requests and by kind of the second or third week of April, we launched this. And it is basically all COVID patients, operative and mostly non-operative, made it free. It's going to be hospital based. And so it's really inpatients. That's who we're hearing from. And that's where our registries are now. Most of our registries have a collection done by a registrar. So we designed it the same way. And a lot of our NISQIP registrars said that we absolutely can collect that and we will collect it because we're not doing anything for NISQIP or some of our other registries as well. And all the other kind of administrative IRB, it was approved by that and it's international. The variables are, we made it very basic patient demographics, presenting symptoms, comorbidities, treatments and outcomes. And this was kind of, the idea was make it expedient, put it out there so that hospitals can put in data and kind of see how they're doing. And subsequently hospitals have kind of seen those things and hopefully it's been helpful. But I know that there's a lot of lessons that we've learned and we can get to that in the discussion. Right now, or as of a few weeks ago, there were maybe 10,000 patients in it. It's from 30 states in the country and a number of hospitals. And I won't get into the data results because I think it's not really apropos for this, but maybe until the discussion. But this is kind of how I think a lot of organizations put things out there. And because we were in the early stage of the pandemic and we were flying blind. So it was a sense of like, all right, let's put something out there. Perfect is the enemy of good. Let's just get something out there that we think is good and then go from there. And so that's what happened at the College of Surgeons. Next please. So again, it was good. However, if we kind of now look back at the last three or four months and we ask ourselves, and we do this all the time at the College of Surgeons, and I know that this happens at other organizations as well. Some very simple questions. Are the data that we're providing good enough or enough for clinical care and giving us good clinical care information and epidemiologic information? Is it helpful? And are we making movements based on those things? Is it adequate for investigation? So people need to do research to figure out this treatment versus that treatment, or who do we test and who do we not test and what's the risk of all these things? Is it helpful for that? And finally, is it helping us improve? Are we making gains in our fight against COVID and the pandemic? So really, I think I'm not sure we are making huge gains in that, particularly if we agree that the pandemic is ongoing. And in many areas, we were talking to Greg in Georgia, or I'm in California, that numbers are increasing, and it seems to be worsening. So whatever we're doing, can we do something even better to help stem the tide? Next please. So some of the things that I know that we talk about within the College of all of our registries are these, and these are just some bullets that I hope we can get to. But the big why question, why are we collecting and what are the questions specifically that we're trying to answer? And I think the questions that will progress. So what are we trying to answer now and what are potential questions we need to answer in the future? What will we need to answer and how do we prepare ourselves for now and the future state? Settings is very important. I was saying that ours is hospital-based, but there's a lot of stuff in the community that we need to do or combinations. We spend a lot of time on the collection issues of case identification, case ascertainment. How do we find these COVID patients to put into the registry? And is it from the very beginning of testing or is it once somebody hits the operating room or for Greg in the ICU or for Elizabeth, somebody with a cancer diagnosis? And then how do we potentially do it in an automated way? Because taking staff time, people time with a registrar is how a lot of folks do it, how we do it, but is there something different and perhaps better? The variables themselves that make up the registries, we have issues all the time that we discuss about accuracy and standardization. If we have people doing it, how do we get to parsimony? How are they related to the aims and how do we prepare for things in the future? The analysis are very important and I will defer to other speakers and Elizabeth who does a lot of this, but when we think about this of who does the analysis and how is it done and is it a coordinated effort? Those are real questions that we're always trying to address and tackle. Timeliness is really important because we don't want the helpful data to come out months later if it'll help sooner. And finally, a big thing that is in the title is that integration and merging of registries and what's possible. And I think we will hopefully get to that in the discussion. Next please. As a final two overarching questions, if the pandemic is going to persist for a while, what lessons, good and bad, have we learned in these past few months regarding collection, analysis, reporting, and really to achieving actionable improvements? And then secondly, what things can we do to overall to do better? Thank you and look forward to the discussion. Thank you so much, Cliff. We're going to turn to Liz Merritt Mayer now to give her perspective from ASCO. Thanks. Great. Thank you. Thank you for having me and thanks for a great introduction, Cliff. So I'm going to be discussing the ASCO registry, the ASCO survey on COVID-19 and oncology registry. Next slide please. So why ASCO? Why did we get involved with this? So of course, this is a national medical emergency. ASCO, as a member organization with more than 40,000 members across the world, immediately started developing guidelines and guidance for clinicians and others to help address patient care issues. And we also thought, you know, are there other ways that we can do something to assist the oncology community? One of the ways we thought we could do that was by standing up this registry. And so I'm part of the Center for Research and Analytics at ASCO. It is, we call it CENTRA. And it's only been around for about three years. So we had developed some infrastructure. We had developed a group that was familiar with research, familiar with data, data collection, and the ability to analyze data. So we thought this is something that we can do. And so we immediately started talking to our members and talking a lot amongst ourselves to try to develop this registry. So we came at it from sort of a research perspective to help guide us in terms of what we wanted to collect. And so the primary objectives that we had were to analyze distribution of symptoms and severity of COVID-19 among people with cancer. We wanted to examine the impact of COVID-19 on cancer treatment and on outcomes. And we also wanted to document the adaptations of cancer care to the pandemic. So we actually have thought about it really in two different ways. One is focusing on what are the changes to the practice? And we do a survey on that. And then we also focus a lot on collection of data for cancer patients. Next slide, please. So the patients of interest here, we spent a bit of time figuring out who we should collect data on. And we can't say that we necessarily got it right. There are lots of nuance to deciding this, but we decided when we launched this that we wanted to have patients that had confirmed COVID-19 diagnosis based on a confirmed test. And we were looking at patients who were in active anti-cancer treatment. So patients who had recently been diagnosed, patients who had clinically evident cancer and were receiving treatment. We also include patients who are disease-free but would be receiving any type of adjuvant therapy within one year after resection, and those who have clinically evident cancer receiving supportive care only. So what this means is that we're not collecting data on patients who are more than one year out from their curative therapy. So there are many patients that are going to medical oncology office practices that would not be in our registry. Next slide, please. So the way that we've structured this is that we have relationships, ASCO has developed relationships or agreements with each of the practices that is participating. So we received a central IRB review for our study and our data collection efforts. And we have then signed data use agreements with each practice that takes part in this. Each of the practices completes a survey about their practice changes. And then for each patient, we have not a one-time data collection. We have baseline information around the time of COVID-19 diagnosis, or it could go back in time if the practice joins, let's say, in June, and they have patients that were diagnosed back in March. But then we've structured it so that we are collecting follow-up information over the next year about how those patients are doing with regard to both COVID-19 and longer-term sequelae and their cancer outcomes. So as you might expect in a registry setting, we are not getting consent from patients, but we have structured it as a limited data set, and we are able to connect patients over time using zip code and date of birth information. Next slide, please. So the way that we have structured our data collection is very similar to the way a lot of other groups that have been collecting data in this area in cancer are doing. We're using RedCap. We've structured it as a survey, and as we were developing the registry, we were fortunate to have a very active volunteer, Dr. Jazee from Saudi Arabia, who was involved with the MERS outbreak there in June 2015. So based on some conversations we had with him, he really helped us focus and understand how we should structure questions, what should be the anchoring time points, et cetera, for us to develop our data collection tools. So ideally, well, practices can submit data one of two ways. They can either submit the data directly to our RedCap database, or we provide them our RedCap project, which is basically the guts, the mechanics of the project, and we allow them to collect it locally at their own practice, because some practice would actually like to have the data themselves for their own purposes, for monitoring or doing their own quality assessment, and then we ask them to just send us the data at monthly intervals. As you can see here at the bottom, we have a link there. From the very beginning, we've been trying to be as transparent as possible, sharing our forms with anybody who wants to see them to see how we're collecting the data. We had early conversations with the NCI as they were developing their NCAP study, which is their registry for cancer patients with COVID-19, and we were pleased that they actually found them quite helpful and were able to incorporate some of the structure that we used into the NCI registry, which, of course, down the road should hopefully be helpful if we have things that are asked in similar ways and structured in similar ways as well. Next slide, please. So we launched the registry on April 10th. As of yesterday, we had enrolled 41 practices across 23 states. We are continuing to accrue practices. We have two others that we think will have signed on by the end of today. We did some targeted outreach to hotspot areas, which is very dynamic, so the practices we thought were hotspot areas two months ago are not necessarily hotspot areas now, and so we're evolving our approach to that. We also, ASCO takes great pride in its relationships with community practices, so we actually have most of the practices we have are community-based practices, although we do have some academic practices as well. So it might not seem like a lot, but we currently have about 175 patients who have enrolled or have data submitted, and that includes all of those where data is being submitted directly to the ASCO instance of our database and none that are collecting data at their local instances. Next slide, please. So in terms of using the data that we're collecting, we've been very aware from the beginning that this is very dynamic, and I think that this really speaks to the ideas for this session, is that we recognize the way we set this up, even in March and April, we would not be able to foresee things coming down the road. So we have formed a steering group that is made up of experts from different areas of interest that would help us figure out exactly how to move forward, how to use the data, what changes we need to make to the data that are collected, and analyses to perform in the short and long term. So we have infectious disease specialists, oncologists, epidemiologists, et cetera, that are all meeting with us regularly to discuss the path forward. So we plan to release reports periodically, publicly, starting in early September. We've talked about these from sort of a traditional report standpoint, also from sort of a more dynamic interface that we would make available online. And then we also plan to make the entire dataset available to external researchers sometime next year. Next slide, please. So from the very beginning, we've had lots of conversations with other groups that are also interested in collecting data on cancer patients. So I know that from an earlier webinar, the American Society of Hematology described their registry, they are collecting data on patients with hematologic malignancies in addition to other non-hematologic disorders. The structure of their registry is different than ours. You may have also heard of the COVID-19 and Cancer Consortium, referred to as the CCC-19 registry. They're collecting de-identified data where it's at the physician level, and because of the way that it's submitted, there are no identifiers to know where the patient was treated and who submitted the data. COVID-19 is an international consortium, goes by the name TERAVIL, and that's specific to thoracic malignancies. St. Jude's has developed a global consortium for COVID-19 and childhood cancers. And then, so we've had conversations with most of these groups, in addition to the NCI, as I mentioned in their NCAP study, and then we've also had conversations with different groups across the world, in Brazil and Mexico, Saudi Arabia, and also a European registry group that is focused specifically on neuroendocrine neoplasms. And so our goal in speaking with all these groups early on is to start thinking about how we can merge our data, whether it be in a real sense or sort of in a combining results of analyses on the back end. And my little figure here really is the idea we've been going with all along, the idea that really a lot of these different registries in the cancer world are collecting different pieces of the same puzzle, and we're hoping that we can fit them together to answer important questions at the end together. Next slide. So this is just a list of the participating practices thus far. As I said, we have more that are joining every day, every week. So we're hoping to get to around 100 practices would be, I think, our goal within the next couple of months. Next slide, please. And then I just wanted to acknowledge we had, in addition to a number of ASCO staff that have been really important in this, our principal investigator is Dr. Richard Shilsky, who is the chief medical officer at ASCO, and a number of volunteers that have really helped guide us along the way to help develop this. And that's it. Thank you. Thanks so much, Liz, that was great. So next up, we will hear from Greg Martin, who will give the experience of the virus registry at SCCM. Greg. Thanks very much, Helen. It's certainly a pleasure to talk today about the Society of Critical Care Medicine's virus registry, the Viral Infection and Respiratory Illness Universal Study. I appreciate the invitation, and as Helen mentioned, I'm a critical care physician at Emory University in Grady Memorial Hospital, and I'm also the president-elect for the Society of Critical Care Medicine. As all these registries, this is very much a team effort, so that we've got leadership at the Mayo Clinic and Boston University who are certainly helping and leading a lot of the efforts that are coordinated by our Discovery Research Network and the Society. So the first question to me and to us really was, why a COVID-19 registry? And some of our, some of the two analysts you've just heard raised this as well. For us, it was important because the Society is the largest nonprofit, multi-professional society in the world, dedicated to the consistency and the practice of high-quality critical care, and we have members in more than 100 countries, and our patients are particularly unique in being critically ill and injured, and thus requiring timely and high-quality life-sustaining care, and that leads to us, to the paradigm for SCCM, the Right Care Right Now paradigm, and it also informs our mission, which is to secure the highest quality care for all critically ill and injured patients, and when you put these together between our members and our patients and our goals as a society, it leads to the clear conclusion that having a registry to inform both our patients and our providers was an important factor. The other thing that's unique about SCCM is that it's a multi-professional society, so it's not just a physician-led society, it also has a variety of other practitioners like advanced practice providers, but it also has the pharmacists who are responsible for critical care pharmacy decisions and drug therapies, as well as the respiratory therapists and the nursing staff who work at the bedside, and informing that larger body of individuals is perhaps another major factor for us. Next slide. So, sorry, I just covered that as well, and I apologize. So that was sort of the main reason why we felt that a COVID-19 registry was critical for us. Go ahead to the next slide. So clearly, if you look back over history, we've had a number of epidemics and pandemics, and certainly we have failed ones, and it may be too much to expect to have real-time capture, harmonization, and sharing of pandemic data in historical epidemics, but we live in a very different era now, and accomplishing those goals is actually completely feasible. So I wouldn't think that that's available from the Black Death or the Plague of Justinian, but we continue to have epidemics and pandemics that occur in more current times, and those are the kinds of things that we need to be addressing. Next slide. So you all are familiar with more recently, we have continued to fail. So we had the SARS, first SARS epidemic in 2003. We had the H1N1 swine influenza epidemic in 2009. We had MERS in 2012, and we even had Ebola in 2014, and those are not the only years. A lot of those lasted more than one year, but it's clear to us that we've had several failures as we think about how to capture information that would be vitally important, both to providers and to the public, it would be helpful for people to both conduct research, but also to inform the clinical care that's needed for these patients. Next slide. So attempting to stand up and effectively run registries in those prior epidemics and pandemics unearthed several key elements about how to accomplish the goal of being able to rapidly run a registry when a natural or man-made disaster or another infectious pandemic comes along. At the Society of Critical Care Medicine, our critical care research network, which is known as Discovery, was already working to create C2D2, and C2D2 is the critical care data dictionary, and the idea was to have harmonized critical data definitions and data elements. So at the point that the pandemic began, we had actually harmonized the data points to an extent, and we were actually preparing to go through a Delphi process to ensure that those were really as consistent and as ideal if we wanted them to be, and that's when COVID-19 came along. So we were already thinking through some of these things about definitions, data points, the legal implications, and even from a research perspective, dealing with central IRB and how to get those things together. Next slide. So in one way, Ebola was actually fortunately more amenable to isolation and in some ways mercifully brief in the United States, but the emergence and decline in that in a different cycle from influenza led to, a lot of us, the realization that capturing vital infection information in the midst of a disaster or a national emergency or a pandemic is really less likely to be successful without already having those pieces in place. So from our perspective, we had started thinking about that, we had certainly seen these pandemics come through, we had even, and it's more on the individual basis of our members, we had seen attempts being made to try and stand up a registry, and we realized that one of the things that needed to happen is that actually needs to be developed in cycle or outside of cycle so that when the next pandemic comes along, we're actually prepared for it. Next slide. So in some ways, we almost missed the mark with COVID-19. We certainly, like all of us, were learning from those earlier failures. And as I said, the SCCM Discovery Network had already began creating C2D2, the Critical Care Data Dictionary, to have those harmonized critical data elements and definitions for quick deployment. Everyone was seeking to automate the data collection wherever possible because another major barrier, particularly in the midst of a pandemic, as you all have seen around the country and around the world, that when a pandemic is occurring, it's very unlikely to be able to stand up and collect additional information or to add more workload, particularly to the frontline providers who are in the midst of a pandemic. So using, in our mind, using middleware and other tools that can provide direct capture of those data elements and actually pull them from an electronic health record was a key element to having a successful registry as well. So part of what we had planned to do is to have hundreds of sleeping nodes of a combination of academic and community hospitals across the HHS regions, and then those could be activated for data collection within hours of an event. So they were already available. They were essentially already designed to stand up and collect the data, and they simply need to be activated or turned on. Next slide. So the description of virus is that it's, like a lot of these, is a cross-sectional observational study. It's a registry of adult and pediatric patients who were admitted to the hospital, either with a confirmed COVID-19 diagnosis or at least high clinical suspicion. We collect de-identified data and use that for analysis, and the data collection for us actually started as early as January 2020. So we did not have, I'll show you the timeline, we did not actually have the registry up and running at that point, but we were able to retrospectively capture data. You can find more information about it on the website, and if you're one of those people that likes QR codes, you can scan the one at the bottom and it'll take you to the website where you can read more about it. Next slide. So the timeline for this was really optimizing the scope of the C2D2 workgroup, so using that critical care data dictionary tool and the group that was already working on it to rapidly deploy the virus COVID registry. There's really some fantastic people like Rahul Kashyap and others who have been vital, both working on this for a longer period of time and really put a lot of the time and effort into making it feasible for the COVID-19 registry to come along. So the idea was to put all that in place, and then we really stood this up quite quickly. So you can see at the bottom from March 11th when the idea was conceived, beginning to develop the social media contacts and developing the study protocol and the other elements that were needed, and actually getting those ready so that the IRB approval was done within about two weeks after that. And in fact, the first patients and data were collected on March 31st. So from sort of inception to actually having it up and running was on the order of three weeks. And as you can see, we had, I'll show you the geographic picture of this, but we had about 170 sites who signed up and expressed interest in participating even within the first few days or a week of the announcement of the registry. Next slide. So one of the things that we chose to do is to use REDCap. A lot of you may be familiar with that. It's a research electronic data capture system that's created by the people at Vanderbilt as part of a CTSA NIH program. The advantage of it is it exists both in the cloud and it also exists in a variety of instances around the country and people are familiar with using it. So we use that as the tool for collecting and housing all that information. The data collection instruments we created, you can see here, so we have some core data elements about demographics and history and diagnosis of the disease. Particularly for critical care patients, we have information about fluids and infection and vasopressors and mechanical ventilation support. And then we have some very specific things to help make adjustments, both on the pediatric and adult basis to understand organ dysfunction, illness severity, support modalities that were in use, and even in some cases information for FEMA so that we could understand the sharing of ventilators and other things that was occurring in small circumstances. Next slide. I should say, so there's, here's our site recruitment, so we've got, we've got 170 sites that expressed interest in participating. And you can see that they're geographically represented. So we have a number of sites, particularly around the US, but we actually have sites throughout Europe and the Middle East, as well as Asia as well. So we've captured a number of those. We're also continuing to bring on more sites because one of the goals here is to have as complete a database as possible, and in fact, allowing that data capture to occur retrospectively when possible, because it can be pulled directly out of an electronic health record. So again, the data elements are going into REDCap, and they're based on, a lot of people may be familiar with this, the ISERIC case report form. So ISERIC is the Severe Acute Respiratory Consortium for Emerging Infections, and they've created some tools for harmonizing data elements and facilitating data capture and registry completion, particularly in the setting of an emerging epidemic. Next slide. So if you look at where we are now, so this is data through last week. We have about 15,000 patients that are into the registry. That's obviously gone up quite quickly, and it actually continues to accelerate for a couple of reasons. One is we certainly have more sites coming online. The other is that we're working to try and facilitate more and more electronic data capture. So as we begin to have those data elements and build the middleware tools so that the data can come directly from electronic health records, it facilitates the dumping of a substantial amount of data and allow us to make that available to both us and to others. Next slide. So part of what we do with this is we actually put it into the dashboard. So you can see here the sccmcovid19.org website has the dashboard, and in this case, you can see the typical demographics of patients who have COVID-19. Again, these are people who are hospitalized and or in the ICU. So we have patients that are described by gender, race, and ethnicity. We also have signs and symptoms and comorbidities present. You can see here that hypertension and diabetes are quite prevalent as expected. We have information about support modalities like mechanical ventilation and other supports such as respiratory, supplemental oxygen, and high-flow oxygen, and then we have outcome data in the right-hand column about survival from mechanical ventilation, ICU discharge, hospital discharge, and their overall status. So this is one of the tools that we continue to update. It's available on the website. Anyone can go there and look at it, and ultimately, it will be a portal that we can continue to use for other tools, particularly for investigators and people wanting to do additional studies. Next slide. So the other component for us is dissemination of this, and I mentioned the dashboard, but we also have publications. We have member-facing articles that go out. We have our social media channels and campaigns that are being used to share information. We also do a lot of news releases, webcasts, and other information to try and share through either public health agencies or through our own contacts, and then we have the Critical Care Societies Collaborative that shares across some of the other critical care societies, both the nursing societies and the physician societies in the U.S., and then we have these other collaborations and organizations. So we've been working with the American College of Radiology and Point Click Care and other registries as well as other societies, like the Infectious Disease Society of America, to try and grow and facilitate a lot of the work that we're doing. Next slide. So the current status is that we have, as I showed you, we have about 15,000 patients into it. We have the dashboard. We also have what will be the Cohort Explorer, which is being developed by part of our team that will allow people to dive deeper into the cohorts, whether it's for research purposes or for other quality improvement initiatives. We have 68 proposals that have already been put together for ancillary studies. We'll use these data to try and extract and inform other areas that would be useful for the public to know. As I mentioned, a couple of the collaborations and those continue to grow as well, and then as you would expect, we're developing publications, and some of those are already in press or in the public domain, and others are coming along. Next slide. So there's a couple of things that we've really learned along the way that I wanted to touch on. So clearly things that work is early buy-in is a key element. And clearly having really meaningful collaboration is a key so that when people are both getting the data or providing their data and are collaborating with us, we wanna make sure that they're getting something good out of that too. So not only do they get their data back, they can look at the data, they can do benchmarking. They're also eligible to do additional studies and ancillary studies. We try and have that social media presence, which is good for everyone really to see what's happening. We advertise and share the information about what's happening in the studies. We've learned a lot about IRBs and data use agreements and how to make those happen as quickly as possible. We're continuing to learn about the data transfer elements and making sure that that works smoothly and develop those because every EHR is a little bit different and the tools that exist are still in evolution for many of those. As you might expect with anything that is being asked in the midst of a pandemic and is asking for additional data collection, daily reminders and providing support to teams who are doing this kind of work is really, really important. There's clearly in terms of challenges, there's always resource limitations. The more that you can bring in resources that often freeze up time and you can either develop parallel strategies or you can provide additional resources to simply help people to accomplish their goals. There's always the question of scope creep. So how much do you wanna collect? How many data elements? What kind of patients? And even sort of what are the outcomes and the things that you're gonna do with the data in the long run are all things that we've come across along the way. And then there's always considerations like political barriers, whether people wanna participate in this. And particularly as we've all talked about, there's a bunch of different registries. So there's, our goal is not to add any additional work to people. It's really to try and harmonize and make this as feasible as possible. But as we know, there's other registries that exist. And if someone's participating in one that might capture similar data, there are both political and research and workload concerns about participating in another one. Next slide. So finally for us, this has really been a very collaborative process. Our registry was stood up and developed by the society, but then it's been funded by the Gordon and Betty Moore Foundation. And that's been really critical. We've also been fortunate to receive some additional funding that the Mayo Clinic Ventures Program has been helpful in securing that allowed us to create some additional infrastructure and provide a critical seed funding really to help additional data automation and more data transfer, data entry information. We're continuing to expand the partners in collaborative. And I think that's been one of the real lessons learned, but also has been important because it pulls additional people together to work on this and particularly helps to allow us to share that information back to other groups and make it work for as many people as possible, both on the member side, as well as on the patient facing side. Next slide. So with that, I'll really just say thank you and ask if you're interested in more, certainly come to the website, follow us on Twitter. We have a registry, COVID-19 registry, Twitter handle, and you're welcome to reach out to any of us. Certainly I'm happy to talk to anyone about it. I mentioned Rahul Kashyap, who is at Mayo and really leads a lot of these efforts and Dr. Vishakha Kumar, who is the lead at SCCM, the society who has been critical and instrumental in making all this happen. Thank you very much. Great. Thank you so much, Greg. So last and certainly not least, I'm gonna turn things over to Michael Howell, who is the principal scientist at Google, pulmonary doc himself, and he'll kind of give us his perspectives with no slides, having heard all this rich information we've heard from our panelists, having a sense of what might be sort of a combination of socio and technical issues to really move forward to a new next generation of registries, harmonization, what can we learn? So Michael, I'm gonna turn it over to you. Yeah, thank you. And it feels a little bit awkward to talk without slides after kind of three amazing presentations, but Helen and folks asked me to try to react to that and put a little bit of a lens on it to maybe shape some of the conversation later, which I'll try to do. And so maybe I'll tell 30 seconds about kind of my background and why I'm here. So between 1995 and 2017, I was either training or practicing as a physician. My practice was just critical care, like Greg's. And Greg, I will say, Ogie Gajek was a powerful social media force in encouraging people to join the registry. I had a couple of other parts of my career before I joined Google. I was a chief quality officer for the U Chicago Health System. And so supported and funded participation in some of the NISQIP registries and in COPE with ASCO. And then I had a big part of my career that was being a health services researcher with a particular focus on infection and some of the prior pandemics. And then finally, in terms of the ways that these kinds of registries touch our lives, my mom in early 2019 was diagnosed with gastric cancer, died later that year. And so her data is in some of the ASCO registries as well. And so the first thing, you know, our panel participants asked us to think hard about how have we failed and how do we need to move the field forward? And I do wanna stop for a second and say to each of the panelists and to each of the societies who have developed these registries so rapidly in such an unprecedented time, thank you for what's been accomplished so far. And don't sell yourself short. I think everyone who's participating in the webinar across all three of the registries which had been presented, I think all of us ought to be thankful for the work that's been done. And I don't think that you should sell it short. And so, you know, with that as a frame and probably gratitude as the kind of number one headline, I will say a couple of things. And I think the first is, you know, my current job is at Google, right? Which is a reasonably well-known technology company with some of the most advanced machine learning in the world, amazing data scientists. I thought I understood a lot about data science before I joined Google, but it turns out you learn a lot when you move. And sort of with that as a frame, I think I would say what's already been said which is that most of the barriers here are not technical barriers. They are social barriers and socio-technical barriers. A lot of this was brought up already that, you know, understanding what question you want to answer frames which data that you will try to get. We heard about three registries today, but there are many more. One patient may appear in many different registries because of and be unable to be disarticulated later because of decisions early on about which identifiers would be collected or other things like that. And those are important decisions, but they can be trade-offs. We've talked mostly about the U.S. context today. And so most of my remarks will focus on the U.S. context. It's worth saying that in the U.S. we've made the decision to not have a single identifier for a patient, unlike other companies which adds complex, unlike other countries which adds complexity for registries like this and for the hope to be able to see that a patient who had cancer then needed surgery then ended up in the ICU could be seen as a holistic patient-centered registry instead of a set of disconnected data. And I think it's worth saying that each organization that presented today has invested time and resources in developing their own registry and how those data are shared has to be weighed against the investment that you as a society have made. And so those things are all complicated problems with no necessary solution. The other thing that I think COVID has shown me as a health services researcher is that from all of the observational data, we've seen it's like a masterclass in classic epidemiological and biostatistical problems. Selection bias, confounding by indication, immortal time bias, the Will Rogers effect. We've seen major papers in major journals have to be retracted because of this. And so in addition to the social problems, there are fundamental really difficult statistical and epidemiological problems that it doesn't matter how much machine learning you put on top of that if you don't design your inclusion criteria and understand how those things roll out. Adding a lot of machine learning to it just makes it harder to find the underlying problem. And so I would say that there's some hope. I think there's hope both on a technical front and on a policy front that we've seen over the course of this. And maybe I'll start with some of the things that we've seen on the policy side that may make this easier in the future. We saw that CMS and ONC had a really thoughtful and forward leaning interoperability rule that came out that helps provide a framework for patients to be able to join their own data, raises the possibility of patient centered registries in the future that feels like a big step forward. We've seen a decade's worth of change in telehealth in just a few months, things that we thought would take a decade or more to get to. And in fact, now we're all collaborating like this, both in video and in synchronous ways, but also in tools that allow collaboration across documents and spreadsheets and presentations in ways that just a few years ago weren't feasible and weren't widely used. We've seen progress in the US context on the house proposing national patient identifier, which would then potentially allow the ability to link across registries. And I think that we've seen progress on the standard side with a broader and broader uptake of things like FHIR both on standards and on implementation. And so I do think that there's some hope on the policy side. And then there are also things that companies like mine or any of the major technology companies bring to bear. And so what are those? So the first thing, which I don't know if you'd asked a year ago, whether everyone would have agreed about this, but many of the companies have leaned into public health in the time of COVID, whether that's Apple and Google working together on a privacy preserving exposure notification tool, whether it's some colleagues of mine making available anonymized mobility data for researchers and policymakers to use, whether it's the ability to surface public health information at the point of care. And so number one is that there is will and action in this space. And number two are some technical capabilities that did not exist before. So one of them is ubiquitous, high reliability, global scale storage and compute that when a data center goes down or a machine goes down, the ability to rapidly treat that data set across the world as if it were totally untouched by equipment failure is something that all of the major cloud providers are able to do. And as you begin to join complicated registries or add new types of data, those techniques become really important for preserving the data that you've put the effort into creating a registry around. And then there are some new technical capabilities that have come online in the past few years around data types that can be handled in registries. And I'll take radiology just as an example that whenever you look at the inter-rater reliability of an interpretation of a chest x-ray for pneumonia present or not, that the inter-rater reliability tends to be quite low. And everyone who's ever practiced knows this, but it's actually quite shocking to folks outside of the medical profession, the divergence that we see. And there are now with things like convolutional neural networks or some of their slightly more advanced cousins, there's the ability to handle data types that previously we used to think of as unstructured, chest x-rays, CAT scans, other things, as pieces of a registry to do learning off of later without the intermediate step of reducing the data to a couple of sentences or a checkbox. And many people have studied those, the ability to interpret those on very specific things approaches or exceeds in some cases the human level ability. And so that's a new piece that exists. There's also the ability to handle free text from notes as an input into learning in ways that five years ago was just frankly unfeasible to handle through developments like LSTMs and some of their more advanced cousins. And there are a lot of groups working on the ability to do automated data handling and mapping into things like registries. But those are pretty tricky, I would say. And so if I turn the page of my high tech notes here, if I said sort of what do I hope for ASCO and for the American College of Surgeons and for SCCM and for all the other folks who have registries and are thinking about this, I think it may be a few points to frame the future discussion. So one of the things I learned moving to Google was the statement of focus on the user and all else will follow. And here, focusing on the patient and the clinician and the question that you're asking of the data and all else will follow. It's very easy to imagine more data will let you get whatever you want. But if you don't have the question that you're trying to answer framed, it's a big problem. The second hope is to translate the learning from COVID of moving quickly and in a resource light way to your other registries. Having supported FTEs to do data extraction for a number of these registries, it's quite a human intense event. It's really difficult and expensive and particularly as health systems have financial challenges over the next year related to COVID, this is a place to, I think, to lean in on. I think finding a place where, the topic of this is how do we bring the registries together, finding a place where one example of joining registries would be really useful and learning how the teams work together is a really good idea. I think that the Council of Medical Specialty Societies and membership has a very loud voice. And so thinking about what is it that are enablers of the ability to do the work you want in terms of registries and what are things that are barriers in terms of policies is a worthwhile endeavor because we have seen over the past year, five years, 10 years, forward progress in this field, much of it catalyzed by desires from clinicians who need to take care of patients. And then I think the other piece that I would say is that companies like mine, any of the major technology companies can help when you've thought through these problems, can help with the technical underpinnings of these, but it's only going to be successful when it's done in deep partnership with the specific clinical experts across different domains. This is not, as much as I want there to be some magic machine learning model that just takes all of this and turns it into a beautifully curated data set, that is not going to exist in any time soon. But that the partnership can enable a bunch of forward progress in ways that I'm hopeful can help patients and families and doctors and nurses and respiratory therapists and pharmacists and social workers to help take better care of people. Maybe I'll stop there. Thank you so much, Michael. That was wonderful, especially I loved your summation at the end. So we're really excited. We have a good amount of time left to focus on questions and several of you have already put some in the chat box. I welcome our presenters up here. They are to come back and come back on screen and come back off mute. And we'll talk through some of the questions. We've got a couple of specific ones, but let me start with one that is more general. I think it kind of builds on what you just said at the end there, Mike, about we have a loud voice. What are the enablers? What are the barriers? I guess if we think about this in terms of how can we, especially societies, kind of sort this out to increase the speed of what we need to do next time, what does it take for us to really start down that path? Greg gave, for example, on our pre-call is how can we collectively decide a cough is a cough is a cough regardless of registry? Any reflections on what we could, especially societies, really help do to drive towards greater integration? And that's for anyone. Well, I'll start and it's maybe because you mentioned cough and I also, like Mike, share the pulmonologist's background. So I think one thing that we all likely would agree on is that things like ICD-9 codes and ICD-10 codes are not gonna be useful for this because they're applied relatively for other reasons. They may be applied for financial reimbursement reasons, but they also are not applied consistently depending on the setting and the clinical needs. But I do think that there's ways to harmonize these things. So at least from our perspective, one of the things that we've spent some time on, even before the registry was stood up, is trying to define those critical data elements and then begin to figure out what that looks like. What is the data dictionary that goes with it? So that when it doesn't matter whether you're part of ASCO or you're part of the College of Surgeons, if we're asking you about a cough, we have a definition of that and you can use that. I think there's like part of what you and Michael were mentioning is that there's both a social and sort of medical and or a technical component of this. And it would be fantastic to figure out a little bit more of the technical capability behind that. Are there ways to more precisely capture cough by using tools that are natural language processing and EHR tools, for instance, when you say cough may be an interesting diagnosis that would be probably correctly captured if you're in a pulmonology clinic or an asthma clinic. But if you show up in orthopedic clinic with an ankle sprain, it's unlikely that they're gonna capture cough. And the absence of cough from that chart abstraction does not mean the cough wasn't present. So those kinds of things really lend themselves to perhaps a little bit more technical exploration. It doesn't mean there's a solution to it, but maybe there's some precision there that could be matched together with harmonization elements that are on the social side. Cliff, any response there? I know this is something you felt strongly about. We've talked about this in the past as well. Yes, building on what Mike and Greg have said, and I very much appreciate what Mike had said, because at the College of Surgeons, and I'm sure everyone who's worked with their registries and with the informaticists, they're like, the answer to this is not always technical. We can't just push a button and then magic happens and you can have this amazing registry, Cliff. I hear that all the time. To the point that it's kind of a social aspect of communication and the organization of all of us of getting these things set first and with Mike's first things like focus on the aim and the patients and the users, I think that the principles of what we need to do and the vision of what we can achieve needs to be set down if we're gonna move together and integrate these things. We had just written a proposal to, and this is, I'll give a surgical example, but I think you'll get the point, where we were working with our trauma folks, and Ronnie Stewart, and they were talking about this concept called RMOC, Regional Medical Operations Center. So if a region has, I don't know, five trauma centers, and they are all kind of working themselves and maybe getting overloaded some, and some have a lot of space, but they just don't know what anyone else is doing, that's a problem. And in places that have a regional medical operations center, so Ronnie is in San Antonio and Eileen is in Seattle, when they get it to work and the region benefits, because then they could bypass this place that's full and go to the next place, and then just everyone benefits just for a lot of these things. And I think that I view the same thing, we view the same thing in the college. We have six registries, and we're just like, all right, we need to kind of come together and not just work at our different cubicles and not talk to each other and not figure out what works for all of us. And so is there that opportunity here? And even the idea of that Michael and Greg brought up of how would we merge data? How do we link them? How do we potentially have that so that, you know, Greg can help perfect how we address cough and you don't ask the surgeons to do that, but maybe we do something about the operating room and then Elizabeth kind of focuses in on the cancer pieces. And how would we do that and not overlap all these things together? But I do think that having those discussions early on and what are the principles that we would each have? So that's the first thing. The second thing is that, is there going to be an organization? And again, to what Michael said, is CMSS going to be able to be that organization to help bring all this together? A clearinghouse of, all right, this is how we're measuring cough. This is how, if we're going to do this, these are the demographics that we all agree to use for, you know, any of those demographics that we're going to do. Because once those things are askew, and I can tell you from our experience at the college, we have four different ways that we measure readmission. And so we can't even agree on that. And so those principles of basic things, I think are maybe some of the things that Michael's talking about that, all right, once we agree on those things, then where can we put technology to those things? That's a great point, Liz. Great point, Cliff. Anything you want to add, Liz? I think the only thing I would add is just a little bit about sort of the, you know, why we collect data in some cases, and then what the registry provides. So I've had some experience working with, you know, real-world data from EHRs and trying to use it to answer research questions. So, you know, I think there has to be, to some extent, in the whole, you know, scheme of things, a bit of a cultural shift to think about. It is very important, obviously, to capture data for taking care of patients, but it would be great if when the data is being collected, it's also being thought of as contributing to the sort of the global database that we can use to, you know, help patients in the future. And so, you know, cancer, we have, I think, a lot of challenge with that. You know, progression is not well-defined, when obviously cancer progression is a really critical endpoint. So I think that's something, I think it comes up a lot in this registry discussion and the way that people are collecting data is different. Some are being extracted from EHRs. Ours is really more, you know, data collection forms, but for it all to sort of work together, we have to be thinking about making it easier for providers to provide that information and thinking about it as sort of potentially being used for research down the line in general. That's great, and I think certainly we would love to try to bring a lot of registries together to do that function. I think it's critically needed and it's not just needed for COVID. I hope it's nice sort of taking this crisis and using it so that we do that going forward, whether it's influenza or really any other issue, like being able to follow a cancer patient across the ASCO registry, the surgery registry, et cetera. Actually, a question that kind of builds on that for you, Michael, that came in through the chat boxes. So assuming we can actually do the front-end work, how can we actually pull all these data from the registries, and can companies like Google then help with the analytics and machine learning? Yeah, so let me try and tie those two questions together for a second, and maybe I'll say a little bit about what I think the key insight for machine learning compared to classical statistical techniques is, which is that today it's pretty clear that it's easier to program a computer to learn than it is to hard-code it to be smart. And so let me give the examples of that, that the fundamental thing about machine learning is that it learns from examples. It doesn't have 100,000 if-then rules in it. And if I take a study that our group published in JAMA a few years ago, it's probably a good example, then I'll tie it to cough. We had a team that was interested in diabetic retinopathy, the fastest-growing cause of preventable blindness in the world. Screening prevents blindness, but basically only an ophthalmologist or a really skilled optometrist can actually do reasonable screening. If I asked the other physicians on this call to say who feels comfortable looking in the back of the eye and saying, this is moderate diabetic retinopathy, none of us probably would raise our hands. And so the team got a bunch of pictures of retinas working with a partner and had a bunch of ophthalmologists just say on the picture, diabetic retinopathy, yes or no. It's actually a five-point scale, non-mild, moderate, severe, proliferative for that, but just that's it. And then had a bunch of ophthalmologists do that so there was a consensus group. And it turns out that if you do that, just here are retinas that have diabetic retinopathy and here are retinas that don't, that a neural network can learn what those patterns are. It doesn't need to be told, look for microaneurysms or look for little hemorrhages. It actually learns that out of the data. And so you can imagine, I don't think anyone has done these studies. It would be a great thing for one of the societies here to do. You can imagine an approach where you say, okay, we're going to have two trained abstractors say cough present or cough not present for a thousand records, 10,000 records. And then we're just going to learn what those patterns are so that for the next million of these, we don't actually have to have a human being go and look, cough or not. There's some downsides to that. There's some complexities because people write notes quite differently. But there are ways that I think things like machine learning may be able to help scale out some of the human review things that are required for registries today. But a lot of that research still needs to be done, but it seems technically plausible and companies like Microsoft or Amazon or Google are well positioned to help with those kinds of things. That's great. Did I answer the question? I think so. Yeah. I mean, I think it still comes down to, do we have the data in the right format and the way we can use it? But I think you're also describing some real opportunities for us, I think, to demonstrate some validity to help us as we kind of go down this path. I want to pull in the next question for several reasons. It's a great question. It also happens to be from Daniel Yang, who's from the Moore Foundation and the sponsor of our work and certainly Greg's work on this as well. And he raises an interesting question that given the COVID epidemic and the resulting disruptions we're seeing, unfortunately, across routine care and procedures like cancer screening and workup, we'll likely see some delayed cancer diagnoses. Is that something you think we'll be able to see across these registries? I mean, Cliff, for example, can you see time from presentation to time of surgery? Liz, can you begin to see any of that? Any reflections on that? So I can start with that. So we have had conversations at ASCO with the NCI, actually with Dr. Sharpless, the NCI director, who has a big concern about people delaying their screening. And there is a concern that we might see sort of a lessening in new visits for cancer, and then we'll see a spike and possibly later stage diagnoses because of delayed screening. So our registry at ASCO is based for the most part on medical oncology practices. So what we're looking at is for patients who have been referred to medical oncology practices, are they seeing delays in starting treatment? And those who are on treatment, are they seeing disruptions, delays, and discontinuation? So we can answer part of that. In terms of the screening and so forth, that's not something specifically that we can answer with the registry data. However, we do have our complementary practice survey that all of the practices are completing, and that is a specific question that we're asking them about. We're asking them what are the patterns of referrals to your office, et cetera. And then the other thing I will say about ASCO is we do have two other data sources that we are leveraging as well. So we have CancerLink, which is a real-world data set. And from that, we have the opportunity to have somewhat of a control group. So we have all cancer patients, and we can track for across the CancerLink practices how many new patient visits there are. We have another data set called PracticeNet, which provides very detailed real-time information from a few dozen practices about very specific information about the types of visits that they're seeing. So we've been using that to track new patient visits, the increase in telehealth, and other things that have been changing in terms of the COVID-19 pandemic. So the registry can't answer everything, but we are fortunate enough that we have other data collection efforts at ASCO that we're able to use as complementary sources. Essentially, you raised telehealth as well. That's come up a lot of how much we can actually capture the data that emerge from those virtual visits and incorporate them into registries. Cliff, any thoughts to that question from where you sit? Yeah, so I think that it's actually really interesting. So to the point of not screening, potentially coming in with more severe disease or more advanced disease, I think that there's a lot of reports that MIs are coming in later on in that stage. I know from the surgical standpoint, we see a lot of late appendicitis, much more so than early appendicitis, and people are wanting to stay away probably from the hospital and sometimes really for good reasons. So I think that across all types of specialties in healthcare, we're seeing more advanced disease. I also think that others are taking advantage of it. So as you know, we have a geriatric surgery quality program. And so a lot of folks there have said, all right, if we only had another month to get you prehabbed and get ready, like, oh, hey, we have another month. And so some patients are actually more prepared for surgery when they undergo it because they've been able to nutrition or exercise and function or whatnot. And so we're actually also seeing that with some of our work in the geriatric surgery as well. And so I think that as kind of what Elizabeth said, that our registries are allowing us to kind of see both things. And so there are good lessons and lessons where we probably could do better. Really interesting to take advantage of those natural experiments. A great question came in that I'll turf to you first, Greg. What barriers do you see with your organizations participating in joint or collaborative efforts? Kind of getting back to the socio part of the socio-technical issues here. Greg, do you want to start? Yeah, I'd be happy to. So that's, it's a great question because you can imagine, particularly in the midst of a pandemic or really anything else, as we've thought about this over years, there's natural disasters, there's other events that come along that you'd really need to plan for in advance. But even when you get to that point, asking people to collect additional information, you can only imagine trying to go to the city of New York a month or so ago and say, hey, we want you to, by the way, collect all this information on the patients who are in your ICUs or in your hospitals and tell us how your staffing is going and tell us the outcomes of your patients and the treatments you use. It's simply not feasible. And that's an extreme example because obviously there's a lot of health system stress that was occurring at the time, but you have to anticipate that if you're really thinking about how to capture data in the midst of a disaster or a pandemic. So that's one reason why we've tried to figure out ways that we can harmonize data elements and use electronic data capture. Just like Cliff and others are describing, that's perhaps maybe one of the key elements. The reason I mention all this is that the workload associated with populating a registry is a real issue. And if you're asking people to do it manually, particularly in the midst of a pandemic, it's just uniquely challenging and it creates some challenges that are going to probably cause failure. You're going to have incomplete data or biased data that's not going to give you the information you need. Absolutely. No, that makes sense. Actually, a question for you, Michael, that kind of comes on that same theme of automating data. One of the folks put in a question to comment, for example, what Google has been able to do with Ascension Health. And if you're getting all the EHR record data, how does that fit into then what's the value of the specialty society registries or clinically oriented registries? And I'll add in, and then how do we really think about how those come together? Where do we bring our unique skillsets to the table along with the data you might already be collecting? Yeah, it's a great question. So let me say sort of the two or three main things that we've been working on with Ascension. So first is, and for folks who don't know, Ascension is a very large US health system. So for one of the things that Ascension decided it wanted to do was migrate its data infrastructure to a cloud provider, which is Google Cloud. And that's what they're doing. So the keys are held by them. It's all in there. So it's an off-prem data center. The second piece is they decided to use a bunch of collaboration tools like Google Meet and Sheets and Slides, which having that in place before COVID turns out to have been a really good thing to do, to be able to communicate amongst and collaborate amongst all your employees. And the third thing, which I think probably the question is asking about is our group in Google Health has worked on building tools that let you basically take a bunch of stuff that works in your regular life, like the way Google search works to find stuff that is actually what you want to know out of a record. And then to build a tool that sits on top of and around the EHR to let providers actually find things. And there's a, for folks who want to see it, there's a video on YouTube about like how it, what it actually does. But it's, it's a pretty, as somebody who worked in, you know, several different EHRs over many years, it's pretty amazing to have that set of tooling kind of help you find things that are squirreled away in faxed records or in handwritten notes or in things like that. But you know, there are things, there are things that they sort of work in your regular life. So the, to go back to the, so that's sort of the three things that we've been doing with Ascension. And then the, what the question is asking though is, could you take all those data and magically make the registries like not be needed? And there are two reasons that's a, the answer to that is not really. The first is every health system already has access to their EHR and it hasn't fixed it. So the data are there already and there have been many places in one place. And so having all the data together in a more secure, more disaster resilient data center doesn't fix that problem. But the second is that patients move between places. So if I take my mom's gastric cancer experience, she got diagnosed at one place. She ended up having a procedure for further diagnosis in a second place, which was Notling. She decided to get care at one of the major referral centers at MD Anderson because we had lived in Houston for many years and she lived in Florida. And then when she made the decision to transition to hospice, she stopped first in an Ascension hospital in a little tiny small town in Florida. So her data is in there somewhere. And then she went home and died. And so one health systems data don't solve that problem because of the reality of the way that care is provided in the United States. Yeah, I know that makes sense. We've got about five minutes left and about, I don't know, seven or eight wonderful questions in the chat box, but I'm going to pick a couple. One question that just came in from Dr. Thorwarth, who's the head of the American College of Radiology really emphasized, I think what you were talking about as well, Michael, around how difficult it is to collect some of these data. And he's asked the question with so much of the emphasis being on productivity and clinical practice. Comment on the need for a more fully automated data extraction and entry into registries and what it will take to get them used. Is that really a rate limiting step for us going forward? Can you imagine continuing to push new registries that require that level of hand entry or the sort of the sweat and tears of clinicians as the folks putting the data in? Anyone want to start, Cliff? Liz, please. Go ahead, Liz. Michael, were you going to say something? Michael? Yeah, I'll say sort of not from Google perspective, but from the perspective of someone who at a health system had financial and operational responsibility for our quality related registries. We absolutely made tradeoffs. We had a certain amount of budget that we could spend. And we absolutely made tradeoffs between registries that required a lot of FTE time to be able to put the data in. I suspect that experience is shared across different places. So one thing I can add in our experience with the ASCO registry, I noted in my slides, there are a number of different groups that have created registries, many of which could have overlapping patient populations. We've had several groups that are doing their best to practices that are pulling our data dictionaries together and trying to come up with really like a data mark, both for registries and also for their own internal use for projects, research projects, and tracking. So I think there's definitely some opportunities instead of hand entering in this registry and that registry. If you make a coordinated effort in advance, you can actually make it so that a lot of the shared elements are efficiently being populated without having to duplicate efforts. Yeah, I hope so. And I think a lot of what we talked about in our pre-call as well, even being able to sort of standardize the forms, for example, that you mentioned, Greg, so that if you're doing one form from CDC, then it can be harmonized. Final reflections there, Greg or Cliff anyone on that question? I guess I would add to the point and thinking a little bit of what Michael was talking about too, sort of these patient experience and what happens across health systems and geographically in time. It links also to the question of what is a cough and how do you standardize data definitions? Because one of the things that you can imagine is Liz's group has a lot of expertise in cancer and as Cliff said, in surgery and we may in critical care, but those are domain expertise. And one of the keys may be to actually link those registries. Instead of trying to have one massive registry, the key may be to have identifiable information or key indicators that allow you to translate information from one registry to another so that when a rheumatology patient ends up in surgery or in the ICU, we know that and we can capture that information. And the other thing that comes to mind as part of what Liz was saying is, is it's having these data marks and actually putting registries together has a greater purpose as well. So you can do internal benchmarking, you can do QI work, but in some ways it's the basis of doing the learning healthcare system. So it allows you to actually take this information. It's, this may be disease specific, but actually the greater perspective of having harmonized data elements that you can capture and use has a greater value, both internally and for the world. Yeah, those are great, great comments. And it goes back to Michael's earlier point about if we actually could get to a universal patient identifier, that would certainly be sweet. Maybe again, a crisis like COVID might actually help be a driving force towards making that happen. There are a whole series of other questions we're not going to be able to get to, questions about, for example, patient report outcomes. How can we use registries as a way to help us update clinical guidelines? And we'll share them with the panelists and we'll post them on our website. There's just some great questions we won't get to, but I just want to thank everybody for their incredible presentations. Thank you, Michael, for your reflections. I think we really had a great opportunity to talk about what we could do on our end in terms of what specialty societies can help in terms of harmonizing our data elements, moving towards that vision of being able to follow a patient over time and get a more complete picture. So, and thanks to all of you. We'll, again, as I mentioned, have all that information available on our website, both the recording of this, as well as the slides and the slides from the past ones. And reminder, you can actually download those slides right now on your right. And then lastly, if you could just, of course, Julie reminds me, please do your summary and evaluation. That'll come up for you shortly. And then just in the very last minute, just queuing up our next one, which continues on this theme of moving towards what we can help, hope to help our technology help us with. So, you've got three lined up, three more. Next one is on cloud-based platforms and analytic tools, followed by prioritizing patient engagement and inclusion of patient-generated data. So, Janice, who asked the PRO question, please come back for that one. And towards the end of August, we're having one on how clinical registries can help identify and reduce disparities related to COVID. So, last slide. We'll tee up the final. This is our next webinar coming up on August 6th. Give you a couple weeks this time on how we can deploy cloud-based platforms. Again, we're really happy to have some of our friends from the technology world, from Verily, as well as academia, kind of logically why we've got our double AMC colleagues. And then finally, also American College of Radiology. So, I think it'll be another great webinar. Thank you again all for your wonderful presentations and just a great discussion. We clearly could have done this for many hours based on the chat questions that came in. So, thank you all so much. And we certainly at CMSS are committed to working with our societies to make that front-end piece happen so that Michael can help us with the back end. Thank you all. Take care. Bye-bye. Bye. Thank you.

Helen Burstyn

CMSS

clinical registries

pandemic response

COVID-19

Gordon and Betty Moore Foundation

AAMC

surgical outcomes

cancer treatment

critical care

data standardization

automated data extraction

machine learning