Senate PSI Majority Staff Report - Part 1

— Page 1 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Su, John (CDC/NCEZID/DHQP/ISO)" To: "Nair, Narayan (FDA/CBER)" , "Shimabukuro, Tom (CDC/NCEZID/DHQP/ISO)" Ce: "Duffy, Jonathan M. (CDC/NCEZID/DHQP/1S0)" i Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper Date: Fri, 19 Jan 2024 13:56:03 +0000 Importance: Normal Attachments: tinnitus_after_COVID_vaccination_17_Jan_2024.docx CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. You’re awesome --- thanks! Please see enclosed. « John From: Nair, Narayan Sent: Friday, January 19, 2024 8:45 AM To: Su, John (CDC/NCEZID/DHQP/ISO) |; Shimabukuro, Tom (CDC/NCEzID/OHQP/\sO) i Cc: Duffy, Jonathan M. (CDC/NCEZID/DHQP/ISO)| Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper Sure, | think we can do that. Do you have the latest draft? Narayan From: Su, John (CDC/NCEZID/DHQP/iSO) i Sent: Thursday, January 18, 2024 3:32 PM To: Nair, Narayan Shimabukuro, Tom (cc) iy Cc: Duffy, Jonathan M (CDC) Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Narayan, Adding Jon Duffy to this thread, for his awareness. Also, while I’d indicated Jan 31 as a target date, there’s interest in moving this paper forward with haste. Would the end of next week (Jan 26) be doable? Any priority you could put on this ask would be greatly appreciated. Thanks! « John From: Nair, Narayan Sent: Wednesday, January 17, 2024 11:27 PM To: Su, John (CDC/NCEZID/OHQP/'sO) i ; Shimabukuro, Tom (CDC/NCEzID/OHQP/\sO) i Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper PSI-HHS-000001136458 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 2 of 199 —

Thanks John. Good to hear from you. We will try and get back to you by the due date. Narayan From: Su, John (CDC/NCEZID/DHQP/ISO) Sent: Wednesday, January 17, 2024 6:09 PM To: Nair, Narayan ; Shimabukuro, Tom (CDC) Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Narayan, I hope you’re keeping warm! It’s frosty out there. ❄ I think this paper fell off a lot of radars. We’re trying to move this paper forward. I’m working on updating the most current draft with data from VAERS by dose number; I’ll send later tonight under separate cover. Please amend with EB data mining language (methods, results). I don’t know of a due date per se – would Jan 31 be reasonable? Of course, sooner would be greatly appreciated. Thanks! John From: Nair, Narayan Sent: Friday, December 1, 2023 11:03 AM To: Shimabukuro, Tom (CDC/NCEZID/DHQP/ISO) ; Su, John (CDC/NCEZID/DHQP/ISO) Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper Hi Tom and John, This fell off my radar with competing priorities. Did you have a due date for us to provide comments/edits on this paper? Narayan From: Shimabukuro, Tom (CDC/NCEZID/DHQP) Sent: Friday, October 27, 2023 9:25 PM To: Su, John (CDC) ; Nair, Narayan ; Bazel, Samaneh Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Agreed. I think we should use the FDA EB data mining and describe the limitations that might be unique to COVID-19 vaccines. From: Su, John (CDC/NCEZID/DHQP) Sent: Friday, October 27, 2023 3:18 PM To: Nair, Narayan (FDA/CBER) ; Bazel, Samaneh (FDA/CBER) Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper PSI-HHS-000001136459 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 3 of 199 —

Hi Narayan, Thanks for the feedback! I’ll discuss with Katherine and company. My inclination is to either use existing EB data mining data from FDA, or not – novel methodologies make me uncomfortable if they haven’t been vetted or otherwise validated. John From: Nair, Narayan Sent: Friday, October 27, 2023 3:04 PM To: Su, John (CDC/NCEZID/DHQP) ; Bazel, Samaneh (FDA/CBER) Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) Subject: RE: [EXTERNAL] RE: FDA coauthor for tinnitus paper Hi John, She does bring up a good point. As you know, data mining has all the limitations of passive surveillance as well as others. However, during the COVID vaccine era there is an additional limitation. Since most reports received involve COVID-19 vaccines, disproportionately scores (which are adjusted by year to control for time-dependent, potentially confounding, exposure and outcome variables) can be driven towards the null by COVID-19 vaccine reports contributing substantially to the comparator group. This would could occur in the setting if there was some type of class-effect (e.g., if both mRNA COVID-19 vaccines are associated with the same adverse event). We were aware of this limitation before and during the pandemic. There are many data mining tools and there was some discussion about utilizing a novel tool to adjust for this. However, we thought it would be problematic to use a brand new, possibly unvalidated tool in the context of an EUA. We ended up using the same EBGM data mining we use for all vaccines and has a long history of use rather than take an experimental approach. As new non-COVID vaccine reports are added we think this limitation will be mitigated to some degree. As far as the paper goes there are several options to address this: We could report our data mining findings and just acknowledge this as a limitation (this is what we have done in other papers) We could not include any data mining findings You could develop another tool that would compensate for the greater number of COVID vaccine reports. I am not sure how to do this but you would need a statistician with DM experience. This would be beyond our capabilities at FDA. Narayan From: Su, John (CDC/NCEZID/DHQP) Sent: Thursday, October 26, 2023 9:32 AM To: Nair, Narayan ; Bazel, Samaneh Cc: Shimabukuro, Tom (CDC) Subject: [EXTERNAL] RE: FDA coauthor for tinnitus paper CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi folks, Please see below email (it was late, and I got a bit confused). I know EB data mining looks at vaccine-event pairs between the vaccine of interest and an adverse event, and compares against all other vaccines in the VAERS database and the same adverse event, to see if a disproportionality beyond an established threshold is present. However, I don’t know the methods well enough to address Judy’s comments. How do the methods FDA uses address these points? Thanks! PSI-HHS-000001136460 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 4 of 199 —

John From: Su, John (CDC/NCEZID/DHQP) Sent: Thursday, October 26, 2023 9:10 AM To: Maro, Judy ; Yih, Katherine Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) ; Nair, Narayan (FDA/CBER) Subject: RE: FDA coauthor for tinnitus paper Hi Judy, Sorry, I’ve been juggling a bit and got my coauthors crossed. FDA performs EB data mining for VAERS, and throughout postauthorization safety monitoring for COVID-19 vaccines, has shared with CDC the results. While I’m familiar conceptually with EB data mining, I’ll need to discuss with FDA to better understand how the methods they use address the concerns you’ve raised. Thanks! John From: Su, John (CDC/NCEZID/DHQP) Sent: Thursday, October 26, 2023 8:59 AM To: Maro, Judy ; Yih, Katherine Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) ; Nair, Narayan (FDA/CBER) Subject: RE: FDA coauthor for tinnitus paper Hi Judy, Thanks for the feedback. CCing Narayan for awareness. We’ll get back to you. John From: Maro, Judy Sent: Wednesday, October 25, 2023 11:43 PM To: Su, John (CDC/NCEZID/DHQP) ; Yih, Katherine Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) Subject: RE: FDA coauthor for tinnitus paper Hi John – To do a disproportionality analysis of any kind (EBGM is just one version but they are statistically similar), you need 4 quantities or a typical 2 x 2 contingency table. So, one would need Exposure Yes, Disease yes – any specific vaccine + tinnitus reports Exposure Yes, Disease no – any specific vaccine + non-tinnitus reports Exposure no, Disease yes – all exposures but for the specific vaccine. In the covid era, this means basically COVID + tinnitus Exposure no, Disease no – all exposures but for the specific vaccine + non-tinnitus reports. Again, in this era, that means COVID + non-tinnitus PSI-HHS-000001136461 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 5 of 199 —

WARNING: This email originated from outside of the organization. Do not click links or attachments unless you recognize the sender and know the content is safe. So, for the 17,859, it’s important to know how these are spread among what vaccines and to choose the vaccines that you want to examine for a signal. It will be mostly useless to try to make statements about the COVID vaccines because the database will have so many COVID reports that you can’t create a comparator. You also need to know what the capture is for the period you are examining of the non-tinnitus reports. Best Judy From: Su, John (CDC/NCEZID/DHQP) Sent: Wednesday, October 25, 2023 11:15 PM To: Maro, Judy ; Yih, Katherine Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) Subject: RE: FDA coauthor for tinnitus paper Hi Judy, Glad you’re able to help. We’re hoping for an analysis of reports to VAERS with the MedDRA Preferred Term (PT) “tinnitus” received during Dec 14, 2020 through May 4, 2023. Specifically, if vaccine-pairs for this PT exceed thresholds for statistical significance. If having counts or a line list would help, we can put you in touch with our senior data manager. We identified 17,859 reports during the analytic period. I can share the latest draft of the manuscript (confidentially, of course) if that would help. Please let me know if you have any other questions. Thanks! John From: Maro, Judy Sent: Wednesday, October 25, 2023 10:19 PM To: Su, John (CDC/NCEZID/DHQP) ; Yih, Katherine Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) Subject: RE: FDA coauthor for tinnitus paper Folks, I’m fairly familiar with EBGM – do you have the numbers that were used? On including the FDA, I have no objections but want to note that it will involve another clearance chain which will add probably a good amount of time into the timeline. Happy to help in any way I can, Best Judy From: Su, John (CDC/NCEZID/DHQP) Sent: Wednesday, October 25, 2023 11:15 AM To: Yih, Katherine PSI-HHS-000001136462 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 6 of 199 —

WARNING: This email originated from outside of the organization. Do not click links or attachments unless you recognize the sender and know the content is safe. WARNING: This email originated from outside of the organization. Do not click links or attachments unless you recognize the sender and know the content is safe. Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) ; Maro, Judy Subject: RE: FDA coauthor for tinnitus paper Sounds great – thanks! John From: Yih, Katherine Sent: Wednesday, October 25, 2023 11:11 AM To: Su, John (CDC/NCEZID/DHQP) Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) ; Maro, Judy Subject: RE: FDA coauthor for tinnitus paper Hi John, If you all think it’s important to include this analysis, then it’s fine with me to include a couple of co-authors from FDA. (I’m expecting some or all of the VSD sites to propose a co-author, too, so wouldn’t want the number of co-authors to get too high (for logistical reasons).) Thanks for checking. Cc-ing Judy Maro, in case she has comments about this plan. Katherine From: Su, John (CDC/NCEZID/DHQP) Sent: Wednesday, October 25, 2023 10:30 AM To: Yih, Katherine Cc: Shimabukuro, Tom (CDC/NCEZID/DHQP) ; Moro, Pedro (CDC/NCEZID/DHQP) Subject: FDA coauthor for tinnitus paper Hi Katherine, We appreciate your continued patience as we work on this paper! Desire has been expressed to include Empirical Bayesian data mining of the VAERS data, which is performed by our colleagues at FDA. If we include those data, we’ll need to include coauthors from FDA. Are you okay with this approach? If so, I’ll reach out and get them involved. Thanks! John SMR1.Point32Health.org made the following annotations --------------------------------------------------------------------- Confidential, Proprietary, Exempt or Privileged: This email message with any attachments contains information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is considered privileged, proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any review, disclosure, reproduction, distribution or other use of this communication is strictly prohibited. If you received this email in error, please notify the sender by reply or telephone and delete the message, including any attachments, without saving, copying or PSI-HHS-000001136463 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 7 of 199 —

disclosing it. --------------------------------------------------------------------- SMR1.Point32Health.org made the following annotations --------------------------------------------------------------------- Confidential, Proprietary, Exempt or Privileged: This email message with any attachments contains information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is considered privileged, proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any review, disclosure, reproduction, distribution or other use of this communication is strictly prohibited. If you received this email in error, please notify the sender by reply or telephone and delete the message, including any attachments, without saving, copying or disclosing it. --------------------------------------------------------------------- SMR1.Point32Health.org made the following annotations --------------------------------------------------------------------- Confidential, Proprietary, Exempt or Privileged: This email message with any attachments contains information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is considered privileged, proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any review, disclosure, reproduction, distribution or other use of this communication is strictly prohibited. If you received this email in error, please notify the sender by reply or telephone and delete the message, including any attachments, without saving, copying or disclosing it. --------------------------------------------------------------------- PSI-HHS-000001136464 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 8 of 199 —

From: "Nair, Narayan" To: "Zinderman, Craig E" , "Menschik, David" Subject: RE: suggested edits as discussed... Date: Fri, 07 May 2021 19:40:02 -0000 Importance: Normal Attachments: Data_Mining_Note_to_CDER_2021_0507_dm_cz_nn_(2).docx Hi Craig and David, Thanks for working on this email. I think it is excellent. I had a couple of suggestions but definitely nothing I feel strong about so feel free to ignore if they don’t work for you. I plan to talk to Karen about this issue early next week. Narayan From: Zinderman, Craig E Sent: Friday, May 7, 2021 2:59 PM To: Menschik, David Nair, Narayan Subject: RE: suggested edits as discussed... Narayan: Sorry to send a second version, but David and I made a tweak to the last paragraph that we think works better. Thanks, Craig From: Zinderman, Craig E Sent: Friday, May 07, 2021 2:49 PM To: Menschik, David ; Narayan Nair ( ) Subject: RE: suggested edits as discussed... Narayan: I drafted, and David edited, the attached message to Ana as discussed. Please feel free to edits as you see fit. It’s pretty long so feel to shorten if you can see any opportunity for that. I put it in Word for ease of tracked changes. Thanks, Craig From: Menschik, David Sent: Friday, May 07, 2021 12:33 PM To: Zinderman, Craig E Subject: suggested edits as discussed... …attached… PSI-HHS-000001148712 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 9 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Good AfternoonMerning Ana, ‘Thank you again for talking with us back in March about your work exploring new data mining approaches and discussing your interest in CBER’s COVID-19 vaccine products. We are writing to kindly ask that you to please hold off pn step-creating and sending data mining reports andanalyses using COVID-19 vaccine AE data. In CBER OBE, we have been reviewing the various COVID-19 vaccine data mining results that you have been forwarding. While we appreciate your interestin sharing your results, they have not contributed to our alreadywe-already-have-a robust process for continuous- monitoring ofreviewing incoming and aggregated VAERS (and other vaccine safety) data. | will describe a little bit below:including: -AESIs: CBER and CDC have established setsof AESIs (sets of PTs representing Adverse Events of Special Interest) for key events. Incoming reports captured by these AESIs are highlighted for FDA reviewer screening, as well as medical record follow-up and chart abstraction by CDC reviewers. Many of the alerts that you have been sending relate to AESIs for which we are already screening and reviewing reports, such as AMI, TTS, Thromboembolic events, and other forms of coagulopathy. Having our staff examine your alerts creates an extra, and somewhat redundant workstream for them, since these AESIs are already under close observation. AESIs for which we are seeing substantial or notable reporting are further evaluated via comparisons to background rates (using known COVID vaccine administration data tracked by CDC and provided weekly to FDA) as well as in population-based data sources both at FDA and CDC (e.g., BEST, CMS, Vaccine Safety Datalink (VSD)). Serious report screening: FDA MOs review sSerious reports coming into VAERS daily until meeting pre-specified milestones (i.e., certain time since authorization and doses administered) and then for the first couple months with the mRNA vaccines, and still currently for dal, DA MOs review: weekly review of aggregated PT counts weekly,by seriousness, AESI, Lot number, most frequent PTs, weekly changes in PT rankings, and other metrics. -Pre-screening: for a couple of notable issues, the VAERS program contractor flags reports when they hit the door: these pre-screened events will have expedited gathering of medical records and CDC review and abstraction. TTS and anaphylaxis both have fallen into this category. Of note, data mining alerts, which are designed to generate hypotheses of possible safety issues, are no longer particularly useful for our pharmacovigilance purposes when a signal has already been identified, such as for TTS, or when an issue (e.g., an AES! such as AMI)is being worked up in a more robust (e.g., activesurveillance)system:3 1 that-yout: tike-for AMI te thorough review of VAERS data Also-as-you-knowFurther, we have a standard process for data mining screening in place for VAERS data; this screening was in place at the start of, and throughout, the COVID vaccine campaign; the frequency/nature of the calculations, stratifications and other parameters, are known and understood by us and our stakeholders. We understand that exploring new approaches and varying the stratifications or other parameters might improve the methodology and is of interestto you. However, from our perspective, the approach employed duringa period of intense, high profile surveillance should be standard, and-predictable, and road-tested. Results from Aadjusting parameters thate raise or lowerer sensitivity of the alerts as the vaccination campaign is+ underway could lead to oe a : -: safety results—confusion and have unintendedconsequences(elBeregarding vvaccineconfidence). ‘Commented [NN1]: Agree with not using “stop”. But| \was worried she may misread the sentence as “please hold ‘on” to doing what she is doing. Nair, Narayan 2021-05-07 15:35:00 PSI-HHS-000001148713 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 10 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON {We would be happy to engage with you should you wish to explore your new approaches on CDER regulated products. | So, while we appreciate your work and interestat CDER on the COVID vaccine VAERS data,-for-all-of the above-reasons,in the above context, we have found that- examining the analyses youhave been sending has largely reflected-events otherwise under evaluationno indication for action based on your findings, which have been consuming a lot of resources at a time when resources are stretched across 2-preexisting robust pharmacovigilance infrastructureactivities. Wehaven‘tseen2 So, we are asking that you please hold off on creating and sending data mining results for COVID-19 vaccine AE data. Thanks much for your time and attentionunderstanding, and; sorry for the long email. Kind regards, Thanks, Craig Zinderman, MD, MPH Associate Director for Medical Policy Office of Biostatistics and Epidemiology FDA/Center for Biologics Evaluation and Research ‘Commented [NN2]: This is my clumsy attemptto redirect her. Feel free to delete this sentence if not desirable or feasible Nair, Narayan 2021-05-07 15:33:00 PSI-HHS-000001148714 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 11 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Nair, Narayan" To: "Menschik, David" , "Zinderman, Craig" , "Baer, Bethany" Subject: RE: Data mining question Date: Fri, 15 Mar 2024 19:21:26 -0000 Importance: Normal Inline-Images: image001.png; image002 jpg; image003.jpg; image004.jpg; image005 jpg; image006.jpg Ok, thank you for checking. Narayan Sent: Friday, March 15, 2024 1:30 PM To: Nair, Narayan |; Zinderman, Cri is : Baer, Bethany Subject: RE: Data mining question | understood we provided CDC language for this limitation for the 6 month safety review of mRNA vaccines (and | could share that language if helpful) but in looking at the published article, it now appears that they took it out before publication. I’m not aware of such language included in another publication. From: Nair, Narayan Sent: Friday, March 15, 2024 1:04 PM To: Menschik, David! |; Zinderman, Cries Baer, Bethany Subject: Data mining question Good afternoon, | know in the past we have discussed one of the possible limitations of data mining currently is the vast number of VAERS reports from the COVID vaccines may limit our ability to detect statistical alerts because disproportionality scores may be driven towards the null. Do you know if there is a public reference that discusses this limitation? | have found some references that discuss general limitations for data mining but not sure if there is one that talks about how a large volume of reports from a single class of products could masks results. Narayan Nair, MD (he/him/his) Division Director Division of Pharmacovigilance Office of Biostatistics and Pharmacovigilance Center for Biologics Evaluation and Research U.S. Food and Drug Administration a iy U.S. FOOD & DRUG ADMINISTRATION PSI-HHS-000001152913 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 12 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Nair, Narayan" To: "Sly, Elizabeth" ‘inderman, Craig E" Ce: "Burk, Suzann" Subject: RE: [EXTERNAL] Empirical Bayesian (EB) Data Mining Date: Fri, 30 Sep 2022 14:21:04 -0000 Importance: Normal Thanks — | hope you have a nice weekend also (I think it will be a rainy one!) Narayan From: hy, €icebet Sent: Friday, September 30, 2022 10:16 AM To: Zinderman, Craig E |; Alimchandani, Meghna ee ; Nair, Narayan Ce: Burk, Suzann Subject: RE: [EXTERNAL] Empirical Bayesian (EB) Data Mining Hi All, Thank you all so much. |’ll simply let them know that this information is inaccurate and FDA continues to regularly conduct data mining for all approved/authorized vaccines (and I'll direct them to the link you provided). I'll also encourage them to have their CDC SME’s speak with you directly if they need any further information as | know you communicate regularly. Thanks again for the prompt reply and have a wonderful weekend. -Liz From: Zinderman, Craig E| Sent: Friday, September 30, 2022 10:10 AM To: Alimchandani, Meghna ; Nair, Narayan i ; Sly, Elizabeth Ce: Burk, Suzann Subject: RE: [EXTERNAL] Empirical Bayesian (EB) Data Mining Nothing for me either. Thanks, Craig Sent: Friday, September 30, 2022 9:53 AM To: Nair, Narayan ; Sly, Elizabeth Ee: Zinderman, Craig E Ce: Burk, Suzann Subject: RE: [EXTERNAL] Empirical Bayesian (EB) Data Mining PSI-HHS-000001160285 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 13 of 199 —

I did not have anything to add; thank you. Best regards, Meghna From: Nair, Narayan Sent: Friday, September 30, 2022 9:44 AM To: Sly, Elizabeth ; Zinderman, Craig E ; Alimchandani, Meghna Cc: Burk, Suzann Subject: RE: [EXTERNAL] Empirical Bayesian (EB) Data Mining Hi Liz, Seem my responses below in red. Craig/Meghna, please let me know if you have anything to add. Happy to discuss further if needed. 1) Please let me know if information they have summarized is accurate and/or if there is any further clarification you can provide. The statement is not accurate. FDA is still regularly conducting data mining for all approved/authorized vaccines. 2) Is there any cleared language/statement available on this topic that we could share with CDC that’s publicly releasable? I am not quite sure what they are looking for but here is a link that describes FDA datamining efforts. https://www.fda.gov/science-research/data-mining Some additional background with regard to Question #1 – we would regularly send CDC the results of our datamining for the COVID vaccines by email. Earlier, this summer I suggested to CDC ISO that we discontinue the routine regular emails. I told them that DPV would continue to conduct data mining and would notify CDC if we found any datamining alerts that were clinical relevant and required further action. This was intended as a time saving measure and to reduce email traffic. Since it had been some time that we had a datamining alert that required further evaluation, CDC agreed with this approach. It appears someone at CDC mistook the discontinuation of the weekly emails to mean that we had stopped conducting data mining Narayan From: Sly, Elizabeth Sent: Thursday, September 29, 2022 4:55 PM To: Nair, Narayan ; Zinderman, Craig E ; Alimchandani, Meghna Cc: Burk, Suzann Subject: FW: [EXTERNAL] Empirical Bayesian (EB) Data Mining Good Afternoon, Please see below for an e-mail from CDC seeking clarification related to data mining. They’ve let me know that right now they are just seeking clarification for background purposes but they also mentioned that they would appreciate a statement/quote (if available) which could be used to respond to a FOIA request, if necessary. 1. Please let me know if information they have summarized is accurate and/or if there is any further clarification you can provide. PSI-HHS-000001160286 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 14 of 199 —

2. Is there any cleared language/statement available on this topic that we could share with CDC that’s publicly releasable? Thank you, Liz From: Mitchell, Elnetta (CDC/DDID/NCEZID/DHQP) (CTR) Sent: Monday, September 26, 2022 3:58 PM To: Sly, Elizabeth Cc: Thompson, Perstephanie (CDC) Subject: [EXTERNAL] Empirical Bayesian (EB) Data Mining CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon Elizabeth, One of our SMEs stated that it is their understanding that FDA discontinued Empirical Bayesian (EB) data mining for COVID-19 vaccines after July 5, 2022, on the basis that after over 600 million doses administered, and the stability of observations of the weekly data, the data were mature and no additional potential safety signals would be found. Is this accurate? Please advise. Thanks. Elnetta Mitchell, MBA Goldbelt C6, LLC DHQP/NCEZID/CDC Centers for Disease Control and Prevention PSI-HHS-000001160287 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 15 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Nair, Narayan" To: "Sly, Elizabeth" , "Zinderman, Craig E" Ce: "Burk, Suzann" Subject: RE: [EXTERNAL] Empirical Bayesian (EB) Data Mining Date: Fri, 30 Sep 2022 13:43:32 -0000 Importance: Normal Hi Liz, Seem my responses below in red. Craig/Meghna, please let me know if you have anything to add. Happy to discuss further if needed. 1) Please let me know if information they have summarized is accurate and/or if there is any further clarification you can provide. The statement is not accurate. FDA is still regularly conducting data mining for all approved/authorized vaccines. 2) Is there any cleared language/statement available on this topic that we could share with CDC that’s publicly releasable? | am not quite sure what they are looking for but here is a link that describes FDA datamining efforts. htt www. fda.gov/science-research/dat Some additional background with regard to Question #1 — we would regularly send CDC the results of our datamining for the COVID vaccines by email. Earlier, this summer | suggested to CDC ISO that we discontinue the routine regular emails. | told them that DPV would continue to conduct data mining and would notify CDC if we found any datamining alerts that were clinical relevant and required further action. This was intended as a time saving measure and to reduce email traffic. Since it had been some time that we had a datamining alert that required further evaluation, CDC agreed with this approach. It appears someone at CDC mistook the discontinuation of the weekly emails to mean that we had stopped conducting data mining Narayan Sent: Thursday, September 29, 2022 4:55 PM |; Zinderman, Craig yt 00tti‘“‘COCSCSCSC‘*@S Alimchandani, To: Nair, Narayan Meghna Ce: Burk, Suzann Subject: FW: [EXTERNAL] Empirical Bayesian (EB) Data Mining Good Afternoon, Please see below for an e-mail from CDC seeking clarification related to data mining. They’ve let me know that right now they are just seeking clarification for background purposes but they also mentioned that they would appreciate a statement/quote (if available) which could be used to respond to a FOIA request, if necessary. 1. Please let me know if information they have summarized is accurate and/or if there is any further clarification you can provide. PSI-HHS-000001161947 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 16 of 199 —

2. Is there any cleared language/statement available on this topic that we could share with CDC that’s publicly releasable? Thank you, Liz From: Mitchell, Elnetta (CDC/DDID/NCEZID/DHQP) (CTR) Sent: Monday, September 26, 2022 3:58 PM To: Sly, Elizabeth Cc: Thompson, Perstephanie (CDC) Subject: [EXTERNAL] Empirical Bayesian (EB) Data Mining CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Good Afternoon Elizabeth, One of our SMEs stated that it is their understanding that FDA discontinued Empirical Bayesian (EB) data mining for COVID-19 vaccines after July 5, 2022, on the basis that after over 600 million doses administered, and the stability of observations of the weekly data, the data were mature and no additional potential safety signals would be found. Is this accurate? Please advise. Thanks. Elnetta Mitchell, MBA Goldbelt C6, LLC DHQP/NCEZID/CDC Centers for Disease Control and Prevention PSI-HHS-000001161948 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 17 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Zinderman, Craig E" To: "Menschik, David" |, "Nair, Narayan" Subject: RE: suggested edits as discussed... Date: Fri, 7 May 2021 18:58:58 +0000 Importance: Normal Attachments: Data_Mining Note_to CDER_2021_0507_dm_cz_(2).docx Narayan: Sorry to send a second version, but David and | made a tweak to the last paragraph that we think works better. Thanks, Craig From: Zinderman, Craig E Sent: Friday, May 07, 2021 2:49 PM To: Menschik, David |; Narayan Nair (Narayan.Nair@fda.hhs.gov) Subject: RE: suggested edits as discussed... Narayan: | drafted, and David edited, the attached message to Ana as discussed. Please feel free to edits as you see fit. It’s pretty long so feel to shorten if you can see any opportunity for that. | put it in Word for ease of tracked changes. Thanks, Craig From: Menschik, David Sent: Friday, May 07, 2021 12:33 PM To: Zinderman, Craig E Subject: suggested edits as discussed... .-attached... PSI-HHS-000001175745 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 18 of 199 —

Good AfternoonMorning Ana, : Thank you again for talking with us back in March about your work exploring new data mining approaches and discussing your interest in CBER’s COVID-19 vaccine products. We are writing to kindly ask that you to please hold on stop creating and sending data mining reports and analyses using COVID-19 vaccine AE data. In CBER OBE, we have been reviewing the various COVID-19 vaccine data mining results that you have been forwarding. While we appreciate your interest in sharing your results, they have not contributed to our alreadywe already have a robust process for continuous monitoring ofreviewing incoming and aggregated VAERS (and other vaccine safety) data. I will describe a little bit below:including: -AESIs: CBER and CDC have established sets of AESIs (sets of PTs representing Adverse Events of Special Interest) for key events. Incoming reports captured by these AESIs are highlighted for FDA reviewer screening, as well as medical record follow-up and chart abstraction by CDC reviewers. Many of the alerts that you have been sending relate to AESIs for which we are already screening and reviewing reports, such as AMI, TTS, Thromboembolic events, and other forms of coagulopathy. Having our staff examine your alerts creates an extra, and somewhat redundant workstream for them, since these AESIs are already under close observation. AESIs for which we are seeing substantial or notable reporting are further evaluated via comparisons to background rates (using known COVID vaccine administration data tracked by CDC and provided weekly to FDA) as well as in population-based data sources both at FDA and CDC (e.g., BEST, CMS, Vaccine Safety Datalink (VSD)). -Serious report screening: FDA MOs review sSerious reports coming into VAERS daily until meeting specified milestones (i.e., certain time since authorization and doses administered) and then ; for the first couple months with the mRNA vaccines, and still currently for JnJ, FDA MOs review each serious report as it is processed into VAERS. For the older mRNA vaccines, MOs now conduct weekly review of aggregated PT counts weekly, by seriousness, AESI, Lot number, most frequent PTs, weekly changes in PT rankings, and other metrics. -Pre-screening: for a couple of notable issues, the VAERS program contractor flags reports when they hit the door: these pre-screened events will have expedited gathering of medical records and CDC review and abstraction. TTS and anaphylaxis both have fallen into this category. Of note, data mining alerts, which are designed to generate hypotheses of possible safety issues, are no longer particularly useful for our pharmacovigilance purposes when a signal has already been identified, such as for TTS, or when an issue (e.g., an AESI such as AMI) is being worked up in a more robust (e.g., active surveillance) system.. Some alerts that you have sent, like for AMI, have already temporarily signaled in population-based surveillance, which has already triggered a more thorough review of VAERS data. Also as you knowFurther, we have a standard process for data mining screening in place for VAERS data; this screening was in place at the start of, and throughout, the COVID vaccine campaign; the frequency/nature of the calculations, stratifications and other parameters, are known and understood by us and our stakeholders. We understand that exploring new approaches and varying the stratifications or other parameters might improve the methodology and is of interest to you. However, from our perspective, the approach employed during a period of intense, high profile surveillance should be standard, and predictable, and road-tested. Results from Aadjusting parameters thato raise or lower sensitivity of the alerts as the vaccination campaign is underway could lead to some artificial creation of alerts and an apparent, but not real, sudden change in safety results. confusion and have unintended consequences (e.g., regarding vaccine confidence). PSI-HHS-000001175746 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 19 of 199 —

So, while we appreciate your work and interest at CDER on the COVID vaccine VAERS data, for all of the above reasons, in the above context, we have found that examining the analyses you have been sending has largely reflected events otherwise under evaluationno indication for action based on your findings, which have been consuming a lot of resources at a time when resources are stretched across a preexisting robust pharmacovigilance infrastructureactivities. We haven’t seen a proportionate increase in efficiency or yield, given the robust screening and scrutiny of VAERS data already in place as described above. We’d hate for you to be wasting your time and efforts, so we thought we should suggest that it might be a better use of resources for you to refocus your data mining efforts on other product types. Perhaps there are CDER products and drug related data in your Center that could benefit more from your continued data mining explorations and analyses. So, we are asking that you please hold on creating and sending data mining results for COVID-19 vaccine AE data. Thanks much for your time and attentionunderstanding, and; sorry for the long email. Kind regards,Thanks, Craig Zinderman, MD, MPH Associate Director for Medical Policy Office of Biostatistics and Epidemiology FDA/Center for Biologics Evaluation and Research PSI-HHS-000001175747 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 20 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Menschik, David" To: "Su, John (CDC)" , "Shimabukuro, Tom (CDC)" [i Ce: "Zinderman, Craig E" , "Nair, Narayan" |, "Alimchan Meghna" | "Marquez, Paige L (CDC)" Ly , "Harrington, Theresa (CDC)" Subject: RE: Weekly data mining Date: Tue, 30 Mar 2021 11:02:25 +0000 Importance: Normal Attachments: DE_VAERS data_mining methods_and_limitations_2021_03_DRAFT.pptx; USST_20210326.xls Hi John and Tom, Attached please find a list of all (i.e., unvetted and regardless of notability) PTs with data mining alerts (i.e., EBOS >2) for all EUA SARS-CoV-2 vaccine VAERS reports from our ‘US Signals Summary Table’ (‘as of date’ 3/26/21) along with 3 slides providing contextual information including caveats and limitations. Please feel free to share this hypothesis generating output with your team/command chain, though this is not intended to be shared more broadly. Thanks, David PSI-HHS-000001187885 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 21 of 199 —

1 Data Mining Introduction* • Statistical method for identifying disproportionality (excess of reported AE for product relative to other products) in large database • Can be useful for screening and hypothesis generating only – Evaluate findings in clinical and epidemiological context (e.g., unexpected?) – Compelling hypotheses should be explored (e.g., via case series analyses) – Statistical signal of disproportionality ≠ safety signal • Absence of disproportionality does not confirm absence of safety signal nor negate a signal otherwise detected DRAFT - DO NOT DISTRIBUTE *Best Practices in Drug and Biological Product Postmarket Safety Surveillance for FDA Staff (November 2019; Draft). Available at https://www.fda.gov/media/130216/download PSI-HHS-000001187886 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 22 of 199 —

2 DE VAERS Data Mining Methods • Empirica™ Signal software (Oracle) • Calculates Empiric Bayes Geometric Mean (EBGM) using observed to expected (O/E) vaccine-PT pair ratios – EBGM derived from statistical model (Multi-item Gamma Poisson Shrinker; MGPS) that accounts for instability from small numbers* • adjusted by age, gender and year received • Vaccine-PT pairs ranked by lower 5% bound of EBGM CI (EB05) • Standard alert threshold: EB05 >2 • Weekly US summary table includes subset alerts for age (0-1, 2-8, 9-18, 19-44, 45-64, and >65 years), gender, and serious/fatal DRAFT - DO NOT DISTRIBUTE *Szarfman A, Tonning JM, Doraiswamy PM. Pharmacovigilance in the 21st century: new systematic tools for an old problem. Pharmacotherapy. 2004 Sep;24(9):1099-104. doi: 10.1592/phco.24.13.1099.38090. PMID: 15460169. PSI-HHS-000001187887 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 23 of 199 —

3 Limitations of Data Mining Include: • Impacted by stimulated reporting (e.g., V-safe, media reports) • False alerts from statistical interaction (e.g., If vaccines X and Y often given concomitantly, statistical signal for vaccine X and AE Z may be driven by vaccine Y) • MedDRA constraints (e.g., Signal X can be reflected in multiple PTs that individually do not reach alert threshold) • Confounding (e.g., by indication) • Other VAERS limitations (e.g., underreporting, variable reporting by source, incomplete reporting, duplicate reporting) DRAFT - DO NOT DISTRIBUTE PSI-HHS-000001187888 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 24 of 199 —

From: "Zinderman, Craig E" To: "Menschik, David" , "Nair, Narayan" Subject: RE: suggested edits as discussed... Date: Fri, 7 May 2021 18:49:06 +0000 Importance: Normal Attachments: Data_Mining_Note_to_CDER_2021_0507_dm_cz.docx Narayan: I drafted, and David edited, the attached message to Ana as discussed. Please feel free to edits as you see fit. It’s pretty long so feel to shorten if you can see any opportunity for that. I put it in Word for ease of tracked changes. Thanks, Craig From: Menschik, David Sent: Friday, May 07, 2021 12:33 PM To: Zinderman, Craig E Subject: suggested edits as discussed... …attached… PSI-HHS-000001195617 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 25 of 199 —

Good AfternoonMorning Ana, : Thank you again for talking with us back in March about your work exploring new data mining approaches and discussing your interest in CBER’s COVID-19 vaccine products. We are writing to kindly ask that you to please hold on stop creating and sending data mining reports and analyses using COVID-19 vaccine AE data. In CBER OBE, we have been reviewing the various COVID-19 vaccine data mining results that you have been forwarding. While we appreciate your interest in sharing your results, they have not contributed to our alreadywe already have a robust process for continuous monitoring ofreviewing incoming and aggregated VAERS (and other vaccine safety) data. I will describe a little bit below:including: -AESIs: CBER and CDC have established sets of AESIs (sets of PTs representing Adverse Events of Special Interest) for key events. Incoming reports captured by these AESIs are highlighted for FDA reviewer screening, as well as medical record follow-up and chart abstraction by CDC reviewers. Many of the alerts that you have been sending relate to AESIs for which we are already screening and reviewing reports, such as AMI, TTS, Thromboembolic events, and other forms of coagulopathy. Having our staff examine your alerts creates an extra, and somewhat redundant workstream for them, since these AESIs are already under close observation. AESIs for which we are seeing substantial or notable reporting are further evaluated via comparisons to background rates (using known COVID vaccine administration data tracked by CDC and provided weekly to FDA) as well as in population-based data sources both at FDA and CDC (e.g., BEST, CMS, Vaccine Safety Datalink (VSD)). -Serious report screening: FDA MOs review sSerious reports coming into VAERS daily until meeting specified milestones (i.e., certain time since authorization and doses administered) and then ; for the first couple months with the mRNA vaccines, and still currently for JnJ, FDA MOs review each serious report as it is processed into VAERS. For the older mRNA vaccines, MOs now conduct weekly review of aggregated PT counts weekly, by seriousness, AESI, Lot number, most frequent PTs, weekly changes in PT rankings, and other metrics. -Pre-screening: for a couple of notable issues, the VAERS program contractor flags reports when they hit the door: these pre-screened events will have expedited gathering of medical records and CDC review and abstraction. TTS and anaphylaxis both have fallen into this category. Of note, data mining alerts, which are designed to generate hypotheses of possible safety issues, are no longer particularly useful for our pharmacovigilance purposes when a signal has already been identified, such as for TTS, or when an issue (e.g., an AESI such as AMI) is being worked up in a more robust (e.g., active surveillance) system.. Some alerts that you have sent, like for AMI, have already temporarily signaled in population based surveillance, which has already triggered a more thorough review of VAERS data. Also as you knowFurther, we have a standard process for data mining screening in place for VAERS data; this screening was in place at the start of, and throughout, the COVID vaccine campaign; the frequency/nature of the calculations, stratifications and other parameters, are known and understood by us and our stakeholders. We understand that exploring new approaches and varying the stratifications or other parameters might improve the methodology and is of interest to you. However, from our perspective, the approach employed during a period of intense, high profile surveillance should be standard, and predictable, and road-tested. Results from Aadjusting parameters thato raise or lower sensitivity of the alerts as the vaccination campaign is underway could lead to some artificial creation of alerts and an apparent, but not real, sudden change in safety results. confusion and have unintended consequences (e.g., regarding vaccine confidence). PSI-HHS-000001195618 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 26 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON So, while we appreciate your work and interestat CDER on the COVID vaccine VAERS data,-feralef the above reasons; in the above context, we have found that examining the analyses youhave been sending -has largely reflected events otherwise under-evaluationno indication for action based on your findings, which have been consuming a lot of resources at a time when resources are stretched across e-preexisting robust pharmacovigilance infrastructureactivities. Wehaven‘tseera Kind regards, thanks, Craig Zinderman, MD, MPH Associate Director for Medical Policy Office of Biostatistics and Epidemiology as for ae Evaluation and Research ‘Commented [ZCE1]: | would lean towards keeping this language, but happy to go with Narayan's thoughts, ‘Zinderman, Craig E 2021-05-07 14:42:00 PSI-HHS-000001195619 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 27 of 199 —

From: "Zinderman, Craig E" To: "Menschik, David" , "Shimabukuro, Tom (CDC)" , "Su, John (CDC)" , "Moro, Pedro L (CDC)" Cc: "Nair, Narayan" , "Alimchandani, Meghna" , "Broder, Karen R (CDC)" , "Mcneil, Michael M (CDC)" , "Lale, Allison (CDC)" Subject: RE: Weekly data mining Date: Tue, 12 Jul 2022 18:26:05 +0000 Importance: Normal Attachments: USST_20220708.xls Good Afternoon : Attached please find a list of all (i.e., unvetted and regardless of notability) PTs with data mining alerts (i.e., EB05 >2) for all SARS-CoV-2 vaccine VAERS reports from our weekly ‘US Signals Summary Table’ (‘as of date’ 7/8/22). Please feel free to share this hypothesis generating output with your team/command chain, though this is not intended to be shared more broadly. Thanks, Craig THIS MESSAGE, INCLUDING ANY ATTACHMENTS, IS INTENDED ONLY FOR THE USE OF THE PARTY TO WHOM IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL, AND PROTECTED FROM DISCLOSURE UNDER LAW If you are not the addressee, or a person authorized to deliver the document to the addressee, you are hereby notified that any review, disclosure, dissemination, copying, or other action based on the content of this communication is not authorized If you have received this document in error, please immediately notify the sender immediately by e mail or phone PSI-HHS-000001217046 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 28 of 199 —

From: "Niu, Manette" To: "Nair, Narayan" , "Ahima, Ohenewaa" Subject: FW: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports Date: Thu, 15 Apr 2021 13:36:47 +0000 Importance: Normal Inline-Images: image001.png fyi From: Niu, Manette Sent: Thursday, April 15, 2021 9:27 AM To: Baer, Bethany ; Zinderman, Craig E Menschik, David Subject: RE: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports I’ll forward you Ana’s email with the attachment. The best person to ask would be Ana as she has close ties with Bill Dumouchel. Thank you! Manette From: Baer, Bethany Sent: Thursday, April 15, 2021 9:00 AM To: Zinderman, Craig E ; Menschik, David Cc: Niu, Manette Subject: RE: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports Thanks for forwarding this on. I agree that we should consider different approaches as the underlying database is changing significantly due to the high volume of COVID vaccine reports. I think we should welcome any expert input. The spreadsheet that Bill mentioned in the first email is not attached so I can’t look at it, but David and I have been discussing and are concerned about the effect of so many COVID reports on the standard system we use. Is there a way that Bill can be more involved in our data mining process and interpretation during this unprecedented reporting time? Thanks, Bethany From: Zinderman, Craig E Sent: Wednesday, April 14, 2021 2:02 PM To: Menschik, David Baer, Bethany Cc: Niu, Manette Subject: FW: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports David, Bethany: Might be worth considering the below? I don’t pretend to understand it, but sounds like they are suggesting an analysis not stratified by year. Thoughts? PSI-HHS-000001257995 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 29 of 199 —

Thanks, Craig From: Niu, Manette Sent: Wednesday, April 14, 2021 6:24 AM To: Zinderman, Craig E Subject: FW: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports fyi From: Szarfman, Ana Sent: Tuesday, April 13, 2021 9:17 PM To: Niu, Manette Cc: Stockbridge, Norman L Subject: RE: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports Thanks Manette. Exactly. As DuMouchel pinpointed, there is a need to extend the stratification brackets by the fact that 99% of the results for FY2021 are for COVID-19 vaccines this indeed affects the results. From: Niu, Manette Sent: Monday, April 12, 2021 7:01 AM To: Szarfman, Ana Subject: FW: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports Ana, Does this effect the data mining results we are receiving in 2021? As you know, there is a backlog in VAERS reports with the contractor due to the high volume of reports we are receiving for the COVID-19 vaccines and the prioritization of those vaccine reports. Thank you! Manette From: Szarfman, Ana Sent: Saturday, April 10, 2021 1:22 PM To: Niu, Manette Cc: Vega, Amarilys ; Stockbridge, Norman L ; Quinn, John ; bill.dumouchel Rave Harpaz ; Pease-Fye, Meg Weichold, Frank ; Callahan, Lawrence ; Paredes, Antonio ; Temple, Robert ; Blum, Michael ; Dal Pan, Gerald ; Zander, Judith ; Munoz, Monica Diak, Ida-Lina > Subject: Important analysis by DuMouchel -> Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports Hello all, Please refer to the message from Bill DuMouchel that I am forwarding and to his attached spreadsheet. PSI-HHS-000001257996 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 30 of 199 —

Notice how Bill discovered the need to eliminate the stratification by year when the reports for the COVID-19 vaccine in VAERS are 99% of all reports for a year (2021). I think that we need to invite him to talk with us about the effect of adjustment factors, given the data, so we can all learn from his knowledge. Warmest regards to all, --Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration From: Bill DuMouchel Sent: Saturday, April 10, 2021 2:25 AM To: Rave Harpaz ; Steve Bright ; Rob Van Manen Cc: Szarfman, Ana ; Mohammad Al-Ansari ; Robert Weber Bruce Palsulich Subject: [EXTERNAL] Peculiarities of disproportionality statistics when the product of interest is in almost all of the reports CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. The attached spreadsheet shows some COVID19 results for the three-year period 2019-2021 2019 has no COVID19 reports 2020 has a few 2021 consists of almost all (33929/34256 > 99%) COVID99 reports Look at the values of A, B, C, D ... A+C is much greater than B+D in 2021. The years 2020 and 2021 are shown as separate analyses. Note that RR as well as the Bayesian estimates are almost equal to 1. They stay almost equal to one if the run is stratified by year, because the 2021 results dominate. The next two sets of results show the full 3-year estimates with and without including year as one of the stratification covariates. Only if you mix in more non-covid reports within each stratum can you get enough diversity to allow larger disproportionalities. PSI-HHS-000001257997 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 31 of 199 —

-Bill PSI-HHS-000001257998 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 32 of 199 —

From: "Lale, Allison (CDC/DDID/NCEZID/DHQP)" To: "Moro, Pedro (CDC/DDID/NCEZID/DHQP)" Cc: "Broder, Karen (CDC/DDID/NCEZID/DHQP)" Subject: RE: Weekly data mining Date: Mon, 28 Nov 2022 23:09:10 +0000 Importance: Normal Oh interesting. Thank you Pedro. Like I said, we used to just verbally mention on CISA calls that X, Y, Z preferred term had not signaled in VAERS – but we also could leave it out if that this creates more hassle. Thanks! Allison From: Moro, Pedro (CDC/DDID/NCEZID/DHQP) Sent: Saturday, November 26, 2022 7:49 PM To: Lale, Allison (CDC/DDID/NCEZID/DHQP) Cc: Broder, Karen (CDC/DDID/NCEZID/DHQP) Subject: RE: Weekly data mining Hi Allison, I think that because of the FOIAs we may have asked FDA to stop sending these weekly data mining outputs. I’ll reach out to David Menschik and ask for the latest weekly report Thanks Pedro From: Lale, Allison (CDC/DDID/NCEZID/DHQP) Sent: Saturday, November 26, 2022 6:28 PM To: Moro, Pedro (CDC/DDID/NCEZID/DHQP) Cc: Broder, Karen (CDC/DDID/NCEZID/DHQP) Subject: FW: Weekly data mining Hi Pedro, I was just wondering if we still get these data-mining alerts from FDA? In the past, we have checked this list for our own verification before presenting a CISA consult. For example, we have an upcoming case of Parsonage Turner syndrome (PT: Neuralgic Amytrophy) following COVID-19 vaccine. We performed a VAERS search with Elaine’s help, but it could be nice to say this event has not preliminarily signaled in VAERS. Thanks, Allison p.s. Hope you had a good holiday! PSI-HHS-000002480132 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 33 of 199 —

From: Menschik, David Sent: Tuesday, July 5, 2022 7:42 AM To: Shimabukuro, Tom (CDC/DDID/NCEZID/DHQP) ; Su, John (CDC/DDID/NCEZID/DHQP) ; Moro, Pedro (CDC/DDID/NCEZID/DHQP) Cc: Zinderman, Craig E (FDA/CBER) Nair, Narayan (FDA/CBER) Alimchandani, Meghna (FDA/CBER) Broder, Karen (CDC/DDID/NCEZID/DHQP) ; McNeil, Michael (CDC/DDID/NCEZID/DHQP) Lale, Allison (CDC/DDID/NCEZID/DHQP) Subject: Weekly data mining Good morning all, Attached please find a list of all (i.e., unvetted and regardless of notability) PTs with data mining alerts (i.e., EB05 >2) for all SARS-CoV-2 vaccine VAERS reports from our weekly ‘US Signals Summary Table’ (‘as of date’ 7/1/22). Please feel free to share this hypothesis generating output with your team/command chain, though this is not intended to be shared more broadly. Thanks, David THIS MESSAGE, INCLUDING ANY ATTACHMENTS, IS INTENDED ONLY FOR THE USE OF THE PARTY TO WHOM IT IS ADDRESSED AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, CONFIDENTIAL, AND PROTECTED FROM DISCLOSURE UNDER LAW. If you are not the addressee, or a person authorized to deliver the document to the addressee, you are hereby notified that any review, disclosure, dissemination, copying, or other action based on the content of this communication is not authorized. If you have received this document in error, please immediately notify the sender immediately by e-mail or phone. PSI-HHS-000002480133 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 34 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: To: Subject: Date: Importance: Attachments: Embedded: Inline-Images: "Edmonds, Amanda" "Madni, Rubina" "Osterman, Rachel" [i "Marks, Peter" “Hussain, Sana" [i "Devore, Nicolette" 'Agnihothr: udhakar" "Fink, Doran" Moderna adolescent declination Fri, 18 Feb 2022 22:14:49 +0000 Normal unnamed; 0191-cover.pdf; PONE-S-21-47379_(002).pdf; Response_to_Moderna.docx RE:_Modema_adolescent_EUA_request_response; [EXTERNAL] Clarification_request image00001.png; image00001(1).png; image00001(2).png; image00001(3).png Walinsky, Sarah has shared a OneDrive for Business file with you. To view it, click the link below. wi Response to Moderna.docx Hopefully this works! Discussing best way forward on the Moderna adolescent response letter. Hi there, Sarah Walinsky as inviting you to a scheduled ZoomGov meeting. Join Zoom Meeting =“ [ Meeting URL: Meeting ID: Passcode: Join by Telephone For higher quality, dial a number based on your current location. Dial: PSI-HHS-000001617192 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 35 of 199 —

mae AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Mecing 0: a Passcode: ; | International numbers Join from an H.323/SIP room system H.323: 161.199.138.10 (US West) 161.199.136.10 (US East) Meeting ID: | Passcode: SIP: Passcode: ; | PSI-HHS-000001617193 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 36 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: To: , "Osterman, Rachel" Cr Devore, Nicolette" , Sudhakar" Cc: "Hussain, Sana" Subject: RE: Moderna adolescent EUA request response Date: Fri, 18 Feb 2022 14:44:42 +0000 Importance: Normal Inline-Images: image001.png Sarah, We recommend that this letter be redrafted so that it is more clear that FDA is applying the factors (or at least some of the factors) laid out in the guidance for when the Agency will exercise discretion to decline to issue an EUA for a product (or here, for an amendment to expand use of the product to a new population). The factors are laid out in section V of this guidance on EUAs for COVID vaccines, https://www.fda.gov/media/142749/download, and are also discussed at a high level in the overarching guidance on EUAs for medical products, https://www.fda.g.ov/media/97321/download. It seems the reasoning here is that there is not an emergency need for the Moderna vaccine for this pediatric populati especially given the data suggesting an increased myocarditis risk compared to the currently available vaccine for thi: population (Pfizer). | don’t think this was intentional at all, but as written, the current draft reads as if CBER is suggesting the company would be better off submitting a BLA supplement because CBER might not need to take a supplement to the VRBPAC, whereas this step would be needed for an EUA. We haven’t provided line edits because this would involve substantial redrafting. Amanda From: Walinsy, rch Sent: Thursday, February 17, 2022 12:22 PM To: Edmonds, Amanda |; Osterman, Rache! i ; Madni, Subject: Moderna adolescent EUA request response Hi all, Moderna is pushing for action on their adolescent EUA and we are hoping to respond with the attached letter pushing them towards a BLA submission instead of an EUA submission. Could you please review and provide us with edits/comments/thoughts? We are hoping to get this out before the long weekend, if possible. Also — as a heads-up we are working on Covaxin/Ocugen decline to issue memo and letter and are hoping to get that to you soon for review and clearance before the long weekend as well. Thanks, Sarah PSI-HHS-000001626344 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 37 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Sarah Walinsky, JD Acting Chief of Staff Center for Biologics Ev: U.S. Food and Drug Adi ition and Research istration iy U.S. FOOD & DRUG ADMINISTRATION This e-mail message is intended for the exclusive use of the recipient(s) named above. It may contain information that is protected, privileged, or confidential, and it should not be disseminated, distributed, or copied to persons not authorized to receive such information. If you are not the intended recipient, any dissemination, distribution or co; ited. If you think you have received this e-mail message in error, please e-mail the sender immediately at PSI-HHS-000001626345 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 38 of 199 —

From: Stephen Hoge To: "Marks, Peter" , "Agnihothram, Sudhakar" Cc: Charbel Haber , Carla Vinals Subject: [EXTERNAL] Clarification request Date: Wed, 9 Feb 2022 02:11:39 +0000 Importance: Normal Inline-Images: image002.jpg CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Dear Dr. Marks, Moderna has appreciated the guidance and oversight we have had from the FDA throughout the pandemic. We have worked with the agency to follow the science where it leads us in the interests of both individual and public health. However, we disagree with FDA’s decision on Friday evening not to reconsider authorization 100 mcg of mRNA-1273 in adolescents 12-17 in light of recent relevant data and long-awaited analysis clearly demonstrating that the benefits of mRNA-1273 far outweighs the risks, even in adolescents. At this point, Moderna does not understand the scientific basis for the FDA’s position. We request a meeting with the FDA to discuss the evidence FDA is relying on to support the agency’s position that a 50 mcg primary series dose represents a more favorable benefit-risk profile than 100 mcg for adolescents, and also align on a path forward regarding the adolescent and pediatric indications. Our current understanding of the facts are thus: 1. A quantitative benefit-risk comparison of Moderna and Pfizer by the CDC working group (presented to ACIP last Friday) has concluded that per 1 million second doses in 18-39 yo males, mRNA-1273 (vs. BNT162b2) will result in 104 fewer hospitalizations for COVID-19, and 21 incremental cases of myocarditis. For both genders combined, the comparison mRNA-1273 (vs. BNT162b2) would result in 162 fewer hospitalizations for COVID-19 and 9 incremental cases of myocarditis per million second doses (see figure below). The net incremental benefit-risk ratio between the two vaccines is 18 COVID-19 hospitalizations prevented for everyone 1 case of myocarditis in 18-39 year-olds. The risk and benefit are consistent with multiple prior reports, including peer-reviewed articles from other agencies (US VAMC NEJM, UK Nature Medicine). 2. On February 4th, CDC ACIP also reviewed analyses (VSD, MOVING) on the clinical course of cases of myocarditis, which were generally noted to be mild and recover rapidly. The VSD summary concluded “Among 18 to 39 year-olds there were no noticeable clinical differences between cases after Moderna and those after Pfizer”. Moderna is not aware of any data or reports that concludes differently. 3. The CDC working group concluded that: "Benefits for both mRNA COVID-19 vaccines far outweigh risk of myocarditis”. The ACIP unanimously endorsed mRNA-1273 for adults 18+. 4. FDA scientists recently submitted the updated BEST analysis for peer reviewed publication. The conclusion of the FDA analysis is: "An increased risk of myocarditis/pericarditis was observed following COVID-19 mRNA vaccination, being highest in males aged 18–25 years following second dose. These results do not indicate a risk difference between mRNA-1273 and BNT162b2." The BEST analysis reported a non-significant trend towards higher rates in 18-25 yo males for mRNA-1273 that was consistent with the numbers used in the CDC’s favorable benefit-risk assessment. 5. FDA had previously indicated to Moderna that the rates of myocarditis reported in younger adults in other geographies (Scandinavia, Canada, France) were the basis of waiting for additional data to accrue in the more relevant US population. Of note, no regulatory action has been taken outside of the US as a consequence of these international observational studies. The recently completed VSD and BEST analyses cited above now include populations that are collectively larger than the previously cited international analyses, in the relevant US population, and following a more relevant dosing schema (i.e., homologous vaccination with 4-week interval for mRNA-1273). Moderna believes these larger, methodologically more relevant, US-based populations should be preferably weighed. The findings summarized above have been repeatedly validated by international agencies or in published peer-reviewed articles. Moderna believe the conclusions are approaching broad scientific consensus. Using a 50 mcg dose in the primary series introduces uncertainty concerning the magnitude and durability of the strong efficacy of the vaccine as demonstrated with 100 mcg, while at the same time we question the presumption that the 50 mcg dose will reduce the rate of myocarditis. Moderna is not aware of any data that suggest a lower (50 mcg) dose will provide a more optimal benefit-risk profile. Based on the CDC analysis, assuming the effectiveness and myocarditis rates of a 50 mcg dose of mRNA-1273 were to approach levels comparable to BNT162b2, then lowering the dose will result in 153 more preventable hospitalizations per million doses (162 hospitalizations for COVID-19 less 9 incremental cases of myocarditis). In the current public health emergency the FDA has seen fit to authorize multiple drugs, vaccines and diagnostics for the same indication as long as the known and potential benefits outweighs the known and potential risks, without regard to each product’s relative efficacy or safety to other products. We believe the incremental difference between BNT162b2 and mRNA-1273 in a rare AE without significant sequalae, such as myocarditis, or the PSI-HHS-000001626448 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 39 of 199 —

incremental superiority of efficacy of mRNA-1273 over BNT162b2 in prevention of hospitalization, do not materially alter the overwhelmingly positive benefit/risk balance of both vaccines. Given this scientific consensus regarding the positive benefit-risk profile, Moderna is concerned that FDA’s decision not to consider authorization of mRNA-1273 for adolescents is inconsistent with its previous actions to authorize alternative products to address the same indications. We do not understand the scientific basis for this decision, with unclear implications, if any, for BLS’s for the pediatric indication which we intend to file. We respectfully request a meeting with the FDA to understand what evidence the FDA is relying upon to supports its determination that (1) the known and potential benefits for vaccination with mRNA-1273 at 100 mcg do not exceed the known and potential risks for adolescents (age 12-17), and (2) a lower dose of mRNA vaccine represents a more favorable benefit-risk profile for adolescents. We believe a better understanding of the agency’s position is required if Moderna is to respond correctly. Moderna believes this discussion is urgent as there are currently over 11 million unvaccinated adolescents in the US. In addition, should SARS-CoV-2 evolve to require boosting, a highly effective and durable vaccine may be needed for adolescents, particularly those at higher risk. We will make ourselves available at any time that is convenient for the FDA. Very respectfully, Stephen Hoge, M.D. President Moderna, Inc. Slide 11. Oliver, Summary of Working Group Interpretations. CDC ACIP, February 4th, 2022 Confidentiality notice and disclaimer: The information in this message and any attachments is intended for the exclusive use of the addressee(s), is confidential and may be privileged or otherwise protected from disclosure. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, of any such information by persons or entities other than the intended addressee(s) is prohibited. If you have received this message in error and are not the intended addressee, please notify the sender immediately and delete this message and any attachments from your system without reading or disclosing them. If you are not the intended addressee, be advised that any use of the information in this message and any attachment is prohibited and may be unlawful, and you must not copy this message or attachment or disclose the contents to any other person. PSI-HHS-000001626449 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 40 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Event: Moderna adolescent declination Start Date: 2022-02-22 19:30:00 +0000 End Date: 2022-02-22 20:00:00 +0000 Organizer: Walinsky, Sarah Location: https://fda.zoomgov.com/j/1616041093?pwd=bzd4c1 dub3ppNFQxTkoOZXpjdEJZUTO9 Class: X-PERSONAL Date Created: 2022-07-02 00:27:23 +0000 Date Modified: 2024-09-05 09:58:46 +0000 Priority: 5 DTSTAMP: 2022-02-18 22:14:26 +0000 Attendee: Edmonds, Amanda ; Madni, Rubina ; Osterman, Rachel ; Marks, Peter ; Hussain, Sana ; Devore, Nicolette ; Agnihothram, Sudhakar ; Fink, Doran ; Finn, Theresa ; Farizo, Karen Alarm: Display the following message 15m before start Reminder Walinsky, Sarah has shared a OneDrive for Business file with you. To view it, click the link below. in Response to Moderna.docx ic on Hopefully this works! Discussing best way forward on the Moderna adolescent response letter. Hi there, Sarah.Walinsk) EE is inviting you to a scheduled ZoomGov meeting. PSI-HHS-000001626851 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 41 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Join Zoom Meeting One tap mobic: Meeting UR: Meeting ID: Passcode: Join by Telephone For higher quality, dial a number based on your current location. Dial: Meeting ID: Passcode: International numbers Join from an H.323/SIP room system H.323: 161.199.138.10 (US West) 161.199.136.10 (US East) Meeting ID: | Passcode: i sr. es Passcode: i PSI-HHS-000001626852 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 42 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Mm O d Cc FrNn e| 200 Technology Square + Cambridge, MA 02139 ci sce phone} * fax| EUA Number 27073 Sequence No. 0191 June 9, 2021 Marion Gruber, PhD Director, Office of Vaccines Research and Review Center for Biologics Evaluation and Research U.S. Food and Drug Administration Silver Spring, MD 20993-0002 Submission Type: Emergency Use Authorization (EUA) - Moderna COVID-19 VACCINE Adolescent (Aged 12 through <18 years old) Dear Dr. Gruber: Reference is made to pre-assigned submission tracking number (STN) EUA 27073 for the initial Emergency Use Authorization (EUA) for Moderna COVID-19 Vaccine (mRNA-1273, a novel lipid nanoparticle (LNP)- encapsulated messenger RNA (mRNA)-based vaccine against the 2019 novel coronavirus (CoV; SARS- CoV-2)). Further reference is made to IND 019745, submitted to FDA on 27Apr 2021, and EUA 27073 authorized on 18Dec2020 for Emergency Use for Moderna COVID-19 Vaccine under Sections 564, 564A, and 564B of the Federal Food, Drug, and Cosmetic Act as amended or added by the Pandemic and All-Hazards Preparedness Reauthorization Act of 2013 in Adults, aged 18 years and older. The purpose of this submission is to submit an EUA Amendment to the authorized indication of the Moderna COVID-19 Vaccine to extend the indication to adolescents aged 12 through <18 years old. The content of the submission package is described in Table 1. Table 1 — Contents of Submission Module Document ll Form 1571 12 Cover Letter Note to Reviewer 1.3.4 Financial Certification and Disclosure 144 Letter of Cross Reference modernatx.com PSI-HHS-000001627217 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 43 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON aan iain phone} * fax] mM O d e l Nn e| 200 Technology — + Cambridge, MA 02139 1.14.1.3 Draft Fact Sheet 1.6 Meeting Materials 1.16 RMP 1.19 EUA submission document 5.3.14 Validation Assay Reports 5.3.5.1 mRNA-1273-p203: TFLs, CBER Tables, Narratives, CRFs, SAP and Datasets 5.3.5.1 mRNA-1273-p301: TFLs, CBER Tables, Narratives, CIOMS and Datasets 54 Literature references If FDA has = — ‘_ do not hesitate to contact me directly ot or at This eCTD submission has been prepared by PPD Development, Inc. in full compliance with ICH and FDA guidance. The eCTD has been verified and confirmed to be virus and spyware free. PPD utilizes | | a. All technical questions should be directed to Mr. Eric Malamutt at PPD or email aT Sincerely, Ca r | ota Digitally signed by Carlota Vinals Date: 2021.06.09 Vinals 08:15:41 -04'00' Carla Vinals, Ph.D. VP. Regulatory Affairs — Infectious Disease ModernaTX, Inc. Phone: Fax: Email: modernatx.com PSI-HHS-000001627218 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 44 of 199 —

PLOS ONE Signaling COVID-19 Vaccine Adverse Events --Manuscript Draft-- Manuscript Number: Article Type: Research Article Full Title: Signaling COVID-19 Vaccine Adverse Events Short Title: Signaling COVID-19 Vaccine Adverse Events Corresponding Author: Rave Harpaz, Ph.D. Oracle Corp UNITED STATES Keywords: Covid-19; Adverse Events; Signal Detection Abstract: Statistical signal detection is a crucial tool for rapidly identifying potential risks associated with pharmaceutical products. The unprecedented environment created by the COVID-19 pandemic for vaccine surveillance predisposes signal detection to missed or delayed signals, which may limit our understanding of the risks associated with these vaccines. Based on data underlying the Vaccine Adverse Event Reporting System, we assess the current state and utility of signal detection for COVID-19 vaccine surveillance. To this end, we investigate the temporal evolution of signals corresponding to six largely recognized adverse events, and a newly discovered emerging adverse event. The results demonstrate that signals of adverse events related to COVID-19 vaccines may indeed be missed or delayed when generated by methodologies currently utilized by pharmacovigilance organizations . The results also suggest that a possible source of these missed signals is the notorious masking effect, and that properly identifying and addressing this effect exposes strong statistical associations that would otherwise be deemed uninteresting. Finally, the results demonstrate that a class of advanced methodologies can partially alleviate the problem of missed signals. Order of Authors: Rave Harpaz, Ph.D. William DuMouchel William Robbert Van Manen Alexander Nip Steve Bright Ana Szarfman Joseph Tonning Magnus Lerch Opposed Reviewers: Additional Information: Question Response Financial Disclosure Enter a financial disclosure statement that describes the sources of funding for the work included in this submission. Review the submission guidelines for detailed requirements. View published research articles from PLOS ONE for specific examples. The author(s) received no specific funding for this work Powered by Editorial Manager® and ProduXion Manager® from Aries Systems CorporationPSI-HHS-000001627327 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 45 of 199 —

This statement is required for submission and will appear in the published article if the submission is accepted. Please make sure it is accurate. Unfunded studies Enter: The author(s) received no specific funding for this work. Funded studies Enter a statement with the following details: Initials of the authors who received each award • Grant numbers awarded to each author • The full name of each funder • URL of each funder website • Did the sponsors or funders play any role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript? • NO - Include this sentence at the end of your statement: The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. • YES - Specify the role(s) played. • * typeset Competing Interests Use the instructions below to enter a competing interest statement for this submission. On behalf of all authors, disclose any competing interests that could be perceived to bias this work—acknowledging all financial support and any other relevant financial or non- financial competing interests. This statement is required for submission and will appear in the published article if the submission is accepted. Please make sure it is accurate and that any funding sources listed in your Funding Information later in the submission form are also declared in your Financial Disclosure statement. View published research articles from PLOS ONE for specific examples. The authors have declared that no competing interests exist Powered by Editorial Manager® and ProduXion Manager® from Aries Systems CorporationPSI-HHS-000001627328 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 46 of 199 —

NO authors have competing interests Enter: The authors have declared that no competing interests exist. Authors with competing interests Enter competing interest details beginning with this statement: I have read the journal's policy and the authors of this manuscript have the following competing interests: [insert competing interests here] * typeset Ethics Statement Enter an ethics statement for this submission. This statement is required if the study involved: Human participants • Human specimens or tissue • Vertebrate animals or cephalopods • Vertebrate embryos or tissues • Field research • Write "N/A" if the submission does not require an ethics statement. General guidance is provided below. Consult the submission guidelines for detailed instructions. Make sure that all information entered here is included in the Methods section of the manuscript. analysis of public data from the U.S. Vaccine Adverse Event Reporting System Powered by Editorial Manager® and ProduXion Manager® from Aries Systems CorporationPSI-HHS-000001627329 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 47 of 199 —

Format for specific study types Human Subject Research (involving human participants and/or tissue) Give the name of the institutional review board or ethics committee that approved the study • Include the approval number and/or a statement indicating approval of this research • Indicate the form of consent obtained (written/oral) or the reason that consent was not obtained (e.g. the data were analyzed anonymously) • Animal Research (involving vertebrate animals, embryos or tissues) Provide the name of the Institutional Animal Care and Use Committee (IACUC) or other relevant ethics board that reviewed the study protocol, and indicate whether they approved this research or granted a formal waiver of ethical approval • Include an approval number if one was obtained • If the study involved non-human primates, add additional details about animal welfare and steps taken to ameliorate suffering • If anesthesia, euthanasia, or any kind of animal sacrifice is part of the study, include briefly which substances and/or methods were applied • Field Research Include the following details if this study involves the collection of plant, animal, or other materials from a natural setting: Field permit number • Name of the institution or relevant body that granted permission • Data Availability Authors are required to make all data underlying the findings described fully available, without restriction, and from the time of publication. PLOS allows rare exceptions to address legal and ethical concerns. See the PLOS Data Policy and FAQ for detailed information. Yes - all data are fully available without restriction Powered by Editorial Manager® and ProduXion Manager® from Aries Systems CorporationPSI-HHS-000001627330 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 48 of 199 —

A Data Availability Statement describing where the data can be found is required at submission. Your answers to this question constitute the Data Availability Statement and will be published in the article, if accepted. Important: Stating ‘data available on request from the author’ is not sufficient. If your data are only available upon request, select ‘No’ for the first question and explain your exceptional situation in the text box. Do the authors confirm that all data underlying the findings described in their manuscript are fully available without restriction? Describe where the data may be found in full sentences. If you are copying our sample text, replace any instances of XXX with the appropriate details. If the data are held or will be held in a public repository, include URLs, accession numbers or DOIs. If this information will only be available after acceptance, indicate this by ticking the box below. For example: All XXX files are available from the XXX database (accession number(s) XXX, XXX.). • If the data are all contained within the manuscript and/or Supporting Information files, enter the following: All relevant data are within the manuscript and its Supporting Information files. • If neither of these applies but you are able to provide details of access elsewhere, with or without limitations, please do so. For example: Data cannot be shared publicly because of [XXX]. Data are available from the XXX Institutional Data Access / Ethics Committee (contact via XXX) for researchers who meet the criteria for access to confidential data. The data underlying the results presented in the study are available from (include the name of the third party • research based on public data from the U.S. Vaccine Adverse Event Reporting System and calculated values supplementary materials Powered by Editorial Manager® and ProduXion Manager® from Aries Systems CorporationPSI-HHS-000001627331 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 49 of 199 —

and contact information or URL). This text is appropriate if the data are owned by a third party and authors do not have permission to share the data. • * typeset Additional data availability information: Powered by Editorial Manager® and ProduXion Manager® from Aries Systems CorporationPSI-HHS-000001627332 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 50 of 199 —

Dear Editor, We are pleased to submit a manuscript entitled, “Signaling COVID-19 Vaccine Adverse Events” for consideration by PLOS ONE as a Research Article. As the world contends with ending the COVID-19 pandemic, understanding the risks associated with COVID-19 vaccines is critically urgent, and signal detection is a crucial tool for rapidly identifying such risks. Unfortunately, the unprecedented environment created by the COVID-19 pandemic predisposes traditional signal detection to missed or delayed signals, which can limit our understanding of the risks associated with COVID-19 vaccines and the timeliness of their identification. This manuscript investigates the current state and utility of signal detection for COVID-19 vaccines. We investigate the temporal evolution of signals corresponding to seven distinct and partially recognized adverse events with various degrees of evidence linking them to the vaccines. This temporal evaluation led to several findings, which we believe will be of interest to the readers of PLOS ONE and of importance to drug safety organizations. Notably, we demonstrate that signals of adverse events related to COVID-19 vaccines can be missed or delayed when generated by methodologies currently utilized by drug safety organizations. We identify some of the causes for this problem and propose a solution to partially alleviate the problem. Our manuscript does not assume readers’ familiarity with drug safety surveillance or signal detection. It introduces the topic of vaccine surveillance and signal detection, and provides the statistical concepts needed to understand one of the sources of missed signals and our proposed remediation. We confirm that this work is original and has not been published elsewhere nor is it currently under consideration for publication elsewhere. Thank you for your consideration. Sincerely, Rave Harpaz Senior Director Oracle Health Science cover letter PSI-HHS-000001627333 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 51 of 199 —

1 Signaling COVID-19 Vaccine Adverse Events Rave Harpaz1*, William DuMouchel 1, Robbert Van Manen 1, Alexander Nip1, Steve Bright1, Ana Szarfman 2, Joseph Tonning3, Magnus Lerch1,4 1 Oracle Health Sciences, Burlington, MA, United States 2 U.S. FDA, Silver Spring, MD, United States 3 U.S. Public Health Service/U.S. FDA retired 4 Lenolution GmbH, Berlin, Germany * Corresponding author E-mail: Manuscript Click here to access/download;Manuscript;Signaling COVID-19 Vaccine Adverse Events Final.docx PSI-HHS-000001627334 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 52 of 199 —

2 Abstract Statistical signal detection is a crucial tool for rapidly identifying potential risks associated with pharmaceutical products. The unprecedented environment created by the COVID-19 pandemic for vaccine surveillance predisposes signal detection to missed or delayed signals, which may limit our understanding of the risks associated with these vaccines. Based on data underlying the Vaccine Adverse Event Reporting System, we assess the current state and utility of signal detection for COVID-19 vaccine surveillance. To this end, we investigate the temporal evolution of signals corresponding to six largely recognized adverse events, and a newly discovered emerging adverse event. The results demonstrate that signals of adverse events related to COVID- 19 vaccines may indeed be missed or delayed when generated by methodologies currently utilized by pharmacovigilance organizations. The results also suggest that a possible source of these missed signals is the notorious masking effect, and that properly identifying and addressing this effect exposes strong statistical associations that would otherwise be deemed uninteresting. Finally, the results demonstrate that a class of advanced methodologies can partially alleviate the problem of missed signals. 1. Introduction As the world contends with ending the COVID-19 pandemic, understanding the risks associated with COVID-19 vaccines is critically urgent. The Vaccine Adverse Event Reporting System (VAERS), co‐administered by the U.S. Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention (CDC), is one of several systems used to monitor adverse events (AEs) that occur after vaccination, including the COVID-19 vaccines. Like other safety surveillance PSI-HHS-000001627335 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 53 of 199 —

3 systems, VAERS offers the opportunity to rapidly identify potential risks associated with vaccines–a process usually known as signal detection. Computational methodologies for signal detection have been routinely applied to safety surveillance systems for over 20 years and have become a de facto standard[1]. These methodologies are designed to compute surrogate measures of statistical association between specific pharmaceutical products and AEs that are reported into safety surveillance systems[2]. The measures are typically interpreted as signal scores, with larger values representing stronger statistical associations that are assumed more likely to represent true causal associations. A signal score threshold is often used to screen associations that warrant further attention. Signal detection methodologies currently deployed by safety surveillance organizations are largely based on disproportionality statistics. These methodologies use frequency analysis of 2x2 contingency tables to quantify the degree to which a product-AE combination co-occurs disproportionately as compared with that expected if there were no statistical association. To illustrate, we use the Relative Reporting Ratio (RRR), which is a disproportionality statistic underlying several signal detection methodologies. The relative reporting ratio is defined as the ratio of the number of reports mentioning a specific (target) product-event combination to an expected number of reports for the same combination under the assumption that the product and AE occur independently. Based on the values displayed in Table 1, the relative reporting ratio is formally given by 𝑅𝑅𝑅 = (𝑎+𝑏+𝑐+𝑑)∙𝑎 (𝑎+𝑏)∙(𝑎+𝑐) (1) and a number of enhancements, such as Bayesian smoothing and stratification, lead to several signal detection methodologies currently utilized by safety surveillance organizations[2]. PSI-HHS-000001627336 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 54 of 199 —

4 Table 1. 2x2 contingency table used to compute disproportionality statistics for signal detection reports with target AE reports without target AE reports with target product a b a+b reports without target product c d a+c a+b+c+d Given its impact on public health, signal detection is still an active area of research and, since its inception, multiple guidances[3-6] have been published with practice recommendations as well as admonitions concerning data and methodological limitations. Missed/delayed signals and false alerts are the two primary concerns with signal detection, and two objective measures with which the reliability of signal detection can be evaluated. Missed or delayed signals are especially disconcerting given their direct impact on public health, and in the context of the current pandemic can limit our understanding of the risks associated with COVID- 19 vaccines and the timeliness of their identification. Missed signals can stem from several sources. Incomplete data and the voluntary nature of reporting to surveillance systems are the primary sources of missed signals. However, missed signals can also stem from methodological limitations, and in particular a widely acknowledged problem called ‘masking’[4, 7, 8]. Masking and, similarly, confounding are artifacts of conventional disproportionality statistics used for signal detection that rely on the analysis of 2x2 contingency tables as illustrated above. By virtue of being two-dimensional, other factors that may confound, mask, or more generally bias the relationships between products and AEs cannot be properly accounted for. PSI-HHS-000001627337 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 55 of 199 —

5 A masked relationship between a target product and target AE can emerge when one or multiple products are frequently reported with the target event while making the background rate for the target event considerably large. This larger background rate can then make the relationship between the target product and the target event appear less unusual, hence masking the true relationship. A possible solution to masking, albeit practically infeasible, is to first identify the ‘offending’ products and then remove cases/reports containing those products from the calculation of disproportionality statistics. To illustrate, consider the values displayed in Tables 2-3, which build on the example provided in Table 1 and eq. 1. Tables 2-3 display values used for disproportionality analysis of 2x2 contingency tables capturing a hypothetical target AE and a hypothetical target product labeled ‘A’. Table 2 introduces a product labeled ‘B’, which serves as the ‘offending’ product that masks the true relationship between the target product ‘A’ and the target AE. Table 2 shows that most of the reports (80/93) mentioning the target AE are associated with product ‘B’, which leads to masking. Applying the relative reporting ratio (eq. 1) yields a masked 𝑅𝑅𝑅 = (393 × 3)/(93 × 13) = 0.98 indicating that there is no statistical association. However, removing the reports that mention product ‘B’, yields the counts displayed in Table 3, and an unmasked 𝑅𝑅𝑅 = (233 × 3)/(13 × 13) = 4.14 that indicates a strong statistical association between the target AE and target product ‘A’. PSI-HHS-000001627338 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 56 of 199 —

6 Table 2. contingency table used to compute disproportionality statistics with the inclusion of product ‘B’ that masks the association of product ‘A’ with the target AE reports w target AE reports without target AE reports with target product A 3 10 13 reports with product B 80 80 160 reports without product A or B 10 210 220 93 300 393 Table 3. contingency table used to compute disproportionality statistics with the exclusion of product ‘B’ that would mask the association of product ‘A’ with the target AE reports with target AE reports without target AE reports with target product A 3 10 13 reports without product A 10 210 220 13 220 233 Conditions that make signal detection especially vulnerable to masking effects include: smaller safety databases such as VAERS, relationships involving rare events, and relationships involving newer products. As such, the novelty of COVID-19 vaccines, coupled with ongoing vaccination programs, and the relatively early stages of COVID-19 vaccine surveillance makes signal detection especially susceptible to masking. The aim of this manuscript is to assess the current state and utility of signal detection for COVID- 19 vaccines with emphasis on the issue of missed or delayed signals and the potential of masking effects. To this end, we investigate the evolution of signals corresponding to seven distinct AEs with various degrees of evidence linking them to the vaccines. Six of these seven AEs are part of a list of adverse events deemed to be of special interest for COVID-19 vaccine surveillance by the CDC, FDA, and other health organizations[9-11], and the other is an emerging AE that is yet to be fully recognized but which has accumulated thousands of reports in VAERS and elsewhere. In PSI-HHS-000001627339 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 57 of 199 —

7 addition, we center the evaluation on the messenger RNA (mRNA) vaccines from Pfizer-BioNTech (BNT162b2) and Moderna (mRNA-1273), which account for the vast majority of COVID-19 vaccine reports in VAERS. 2. Materials and Methods 2.1 Data The investigation was performed using all VAERS reports available at the time of writing this manuscript (1990 to October 01, 2021). This data represents a total of 1,599,958 reports, including 39 weeks of COVID-19 vaccine reports, which are publicly released on a bi-weekly cadence from January 01, 2021 to October 01, 2021. Of those, 778,681 reports include the COVID-19 vaccine from three manufacturers: Pfizer-BioNTech (53%), Moderna (39%), and Janssen (8%). The investigation was based on AEs in VAERS coded at the MedDRA Preferred Term (PT) level, and products at the ‘manufacturer’ level, e.g., ‘COVID19_PFIZER/BIONTECH’. 2.2 Adverse Events of Interest (AEOI) The seven AEs investigated in this manuscript and their associated MedDRA PTs are listed below. The MedDRA PTs associated with each of the seven AEs were used to identify VAERS reports mentioning a given AE: 1. Bell's palsy (PT=’Facial paralysis’ or ‘Bell's palsy’) 2. Herpes zoster (PT=’Herpes zoster’) 3. Myocarditis (PT=’Myocarditis’) PSI-HHS-000001627340 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 58 of 199 —

8 4. Pericarditis (PT=’Pericarditis’) 5. Appendicitis (PT=’Appendicitis’ or ‘Appendicitis perforated’ or ‘Complicated appendicitis’) 6. Pulmonary embolism (PT=’Pulmonary embolism’) 7. Tinnitus (PT=’Tinnitus’) As noted in the Introduction, the first six of these AEs are part of a list of adverse events deemed to be of special interest for COVID-19 vaccine surveillance by the CDC, FDA, and other health organizations[9-11]. The last (tinnitus), is an emerging AE that is yet to be fully recognized and characterized. 2.3 Signal Detection Methodologies We evaluated disproportionality statistics produced by four signal detection methodologies. Three of these methodologies – MGPS[12], BCPNN[13], and PRR[14] – are well-established and are currently deployed by various organizations worldwide for routine safety surveillance. However, these three methodologies belong to the class of bi-variate signal detection methodologies, and as such were not designed to control masking and certain confounding effects. The fourth methodology, called Regression-Adjusted GPS (RGPS)[15], is a multi-variate signal detection methodology powered by regression that is designed to produce disproportionality statistics with adjusted background rates that can control masking and more extensive confounding effects. It operates by fitting separate Bayesian logistic regression models to each target AE and by automatically selecting predictors (e.g., vaccines) to be included in each PSI-HHS-000001627341 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 59 of 199 —

9 regression model. Additional details on the RGPS methodology are provided in the Supporting Information (S1), and complete details of the RGPS methodology in reference[15]. Stratification categories used for RGPS, MGPS, and BCPNN were age and gender. Stratification by ‘report year’ was not applied because the vast majority of COVID-19 VAERS reports represent a single year of reporting (2021). We applied the canonical version of PRR, which does not require stratification. For RGPS and MGPS, we generated both the point estimates, labeled ERAM and EBGM respectively, and their associated credible intervals labeled ER05-ER95 and EB05-EB95 respectively. Unless specified otherwise, signal scores are represented by the point estimates. The generation of signal scores for the three methodologies and analysis thereof was done using Oracle Empirica Signal 9.1[16]. 2.4 Capturing the Evolution of Signals The evolution of signal scores for each AE was captured by a time series of signal statistics. The time series starts from a period at which initial reports for an AE were available to the latest batch of reports available at the time of writing this manuscript. Each time point corresponds to a bi- weekly public release of VAERS reports, starting from week 3 (W3) January 22, 2021 and ending in week 39 (W39) October 01, 2021, for a total of 19 time points. The signal statistics computed for each time point include the signal score point estimate and its credible interval, e.g., ER05- ERAM-ER95 for RGPS and EB05-EBGM-EB95 for MGPS. These were computed based on all data available in VAERS, and not only the COVID 19 reports or data within the range of dates underlying the time series. PSI-HHS-000001627342 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 60 of 199 —

10 2.5 Analysis & Evaluation The comparison of signal detection methodologies for the time series centers on the RGPS and MGPS methodologies. These were chosen as representatives of the two classes of methodologies described in the Introduction and above (bi-variate vs. multi-variate signaling methodologies). The IC statistic[13] computed by the BCPNN methodology produces signal scores that are almost identical to those produced by MGPS and therefore redundant in many parts of our evaluation. The PRR signal statistic in its canonical application does not include smoothing or signal score adjustments for small counts as do the other methodologies, and therefore does not protect against false alarms as well as the other methodologies. For this reason, a direct comparison against PRR (in its canonical form) would not have allowed us to isolate and explain sources of missed signals. Nonetheless, our results include notes on both the PRR and IC signal statistics. We used the following concepts/conditions to describe our findings in the Results section: (1) The signaling threshold used to evaluate signals is defined as the value 1.0, i.e., the boundary of no statistical association. (2) For a given AE and methodology, a signal is present/detected if a positive statistical association for the AE is detected. This in turn occurs when the signal score produced by the methodology for the AE exceeds the signaling threshold, more specifically, if the lower limit of the signal score’s credible interval exceeds the signaling threshold defined in (1). For example, ER05 > 1.0 for RGPS and EB05 > 1.0 for MGPS. PSI-HHS-000001627343 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 61 of 199 —

11 (3) For a given AE and methodology, a signal is not present/detected if the signal score’s credible interval contains or falls below the signaling threshold, e.g., ER05 < 1.0 for RGPS and EB05 < 1.0 for MGPS. (4) For a given association, a signal score difference between two methodologies is statistically significant if their credible intervals do not overlap. Likewise, we say that there is no difference in signal scores between two methodologies if their credible intervals overlap, e.g., ER05 < EB95 < ER95. (5) A candidate association for masking is defined as one whose signal statistics satisfy the following condition: ER05 > EB95 and ER05 > 1 and EB05 ≤ 1 That is, an association where RGPS and MGPS disagree by producing signal scores that are statistically significant (non-overlapping credible intervals, ER05 > EB95) with RGPS’s interval above the signaling threshold (ER05 > 1) and that of MGPS below or including the threshold (EB05 ≤ 1). (6) The masking effect size is defined as the ratio of RGPS’s and MGPS’s signal scores, i.e., 𝐸𝑅𝐴𝑀 𝐸𝐵𝐺𝑀 − 1 In the following, the masking effect size will be averaged across the time series to produce a summary statistic and represented as a percentage. Having generated the time series of signal scores for each AE of interest, we investigate and attempt to validate masking sources based on the following: PSI-HHS-000001627344 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 62 of 199 —

12 (1) We select two time periods: an earlier point in the evolution of signals when masking starts to take effect, and the end period (W39). Doing so allows us to examine the origin of the masking sources and whether the sources change over time. The earlier time point corresponds to the earliest point in the time series (for both the Pfizer-BioNTech and Moderna vaccines) for which the RGPS and MGPS signals scores were significantly different, and RGPS’s signal score exceeded the signaling threshold as defined above. (2) For each time point we evaluate the predictors that are automatically selected by RGPS to be included in the regression model for the target AE. Based on the regression coefficients, we then identify the strongest predictors (vaccines) as potential sources of masking. (3) As mentioned in the Introduction, once masking sources have been identified, the conventional approach to control masking is to remove all reports containing the maskers, and re-compute signal scores. We use this conventional approach to confirm our findings. That is, we remove reports containing the potential maskers (vaccines) identified by RGPS and re-compute signal scores for the bi-variate signaling methodologies (MGPS, PRR, BCPNN). Substantial increases in these signal scores as well as their convergence toward the original RGPS signal score is a strong indication that the sources of masking have been correctly identified, and a likely explanation for missed or delayed signals. PSI-HHS-000001627345 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 63 of 199 —

13 3. Results Figures 1 and 2 depict our findings for each of the seven AEs investigated in this manuscript. The figures display the evolution of signal scores for each AE–captured as a time series of signal scores. As described in the Materials and Methods (Section 2.4), the time series ranges from week 3 (W3) to week 39 (W39) of COVID-19 reports, for a total of 19 time points in two-week intervals corresponding to the bi-weekly public release of VAERS reports. Figure 1. The evolution of signal scores for Bell's palsy, Herpes zoster, Myocarditis, and Pericarditis Figure 2. The evolution of signal scores for Appendicitis, Pulmonary embolism, and Tinnitus Rows in the figures correspond to AEs, and columns to vaccines (Pfizer/BioNTech vs Moderna). Figure 1 covers the AEs: Bell's palsy, Herpes zoster, Myocarditis, and Pericarditis, whereas Figure 2 the AEs: Appendicitis, Pulmonary embolism, and Tinnitus. Each figure displays a time series of signal scores for the RGPS and MGPS methodologies. Each point corresponds to the signal score point estimate and its credible interval (shaded region), i.e., ER05-ERAM-ER95 for RGPS and EB05-EBGM-EB95 for MGPS. Supporting information (S2) provides signal statistics, for all combinations of AE/vaccine/signaling methodology. The figures clearly show several trends: (1) The time series curves of signal scores produced by RGPS are always above those of MGPS, i.e., the RGPS signal scores are always larger than those of MGPS. This is not an expected pattern and is indicative of masking effects for the AEs of interest. This also suggests that the RGPS PSI-HHS-000001627346 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 64 of 199 —

14 methodology would have been able to detect signals missed by MGPS or identify signals at an earlier time point than MGPS. (2) Signal scores for the Pfizer-BioNTech vaccine are generally larger than those of the Moderna vaccine, regardless of the signaling methodology used. (3) For most AEs, RGPS and MGPS initially agree on their signals scores (statistically insignificant differences) and then diverge in their signal scores. The divergence is likely due to the influence of masking effects, the evolution of VAERS data, and possibly changes in reporting practices. (4) For several AEs, the time series exhibits aberrations. The aberrations are likely explained or coincide with external events, such as the availability of a vaccine to certain age groups, and the influence of publications. (5) For certain AEs at certain time points the signal scores fall below the signaling threshold. This indicates that at those time points signals would have been missed and that signaling qualification may be time-sensitive. (6) As more data accumulates signal scores expectedly stabilize. Larger fluctuations are seen for RGPS indicating that it is sensitive to masking and confounding effects, and that the data may still be evolving. The following describes our findings for each AE of interest. 3.1 Bell’s Palsy Bell's palsy is a form of acute facial paralysis with a weakening and a drooping appearance of the facial muscles usually on just one side of the face. In most cases the paralysis resolves spontaneously within several weeks. Bell's palsy is due to swelling of the facial nerve, and Type I interferons have been proposed as the potential mechanism[17]. Incidents of Bell’s palsy were PSI-HHS-000001627347 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 65 of 199 —

15 reported in clinical trials for both the Pfizer-BioNTech and Moderna vaccines, and it has also been documented with the Influenza vaccine[18, 19]. The FDA currently recommends its surveillance with larger populations globally. In addition, there have been multiple case reports of Bell's palsy associated with the mRNA vaccines[18, 20-22], and several studies that investigated the association[23-25]. As of week 39, there are 7795 reports of Bell's palsy for the mRNA vaccines (5684 Pfizer- BioNTech, 2111 Moderna). The time series in Figure 1 shows that the signal scores produced by each methodology differ by a small amount, with RGPS and MGPS diverging (non-overlapping credible intervals) around weeks 7-9. The figure also shows that a mild masking effect is present (40% averaged across the time series). Regardless of masking, all methods agree early on that the reported co-occurrence of the mRNA vaccines with Bell’s palsy is unlikely due to chance (signal scores exceeding the signaling threshold). However, towards the end period (W33) the MGPS signal scores fall below the signaling threshold for the Moderna vaccine. On average the signal scores for the Pfizer-BioNTech vaccine are 46% larger than those of the Moderna vaccine for RGPS, and 43% larger for MGPS. 3.2 Herpes zoster Herpes zoster (shingles) is a painful rash that develops on one side of the face or body. The rash consists of blisters that typically clear within 2 to 4 weeks[26]. Multiple reports of patients who developed herpes zoster shortly after COVID-19 vaccination have been recently published, suggesting a potential link with the mRNA COVID-19 vaccines[27-32]. Possible mechanisms that explain the pathogenic link are related to the stimulation of innate immunity through toll-like receptors 3,7 by mRNA-based vaccines[28]. PSI-HHS-000001627348 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 66 of 199 —

16 As of week 39, there are 8228 reports of herpes zoster for the mRNA vaccines (5637 Pfizer- BioNTech, 2591 Moderna). Figure 1 shows a substantial difference between RGPS and MGPS, with MGPS indicating that there is no statistical association between herpes zoster and the vaccines (signal scores below the signaling threshold), versus RGPS indicating the contrary (signal scores exceeding the signaling threshold) from week 13 (Pfizer-BioNTech) and week 17 (Moderna) through the remaining time periods. Although the value of the RGPS signal score is not large relative to the other AEs, it indicates that the association is unlikely due to chance. Averaged across the time series, RGPS signal scores for the Pfizer-BioNTech vaccine tend to be 31% larger than for the Moderna vaccine, and 33% larger for MGPS. Interestingly, the size of the masking effect for herpes zoster was the largest among the AEs of interest. Averaged across the time series, the size of the masking effect was 230% for both mRNA vaccines. The sources of masking were evaluated and validated based on the process described in Section 2.5. The two time periods examined were W17 and W39. RGPS automatically selected 67 (W17) and 44 (W39) vaccine predictors for the herpes zoster regression model. The strongest predictors were the Varicella (chickenpox) and the VARZOS (a combination Varicella and Zoster) vaccines, for a total of six vaccine predictors at the manufacturer level. Although the risk is low, there are documented cases and studies of herpes zoster following Varicella and VARZOS vaccination[33-35]. Upon removal of all reports containing the Varicella and VARZOS vaccines we found that the PRR, EBGM, and IC signal scores indeed reverted to larger signal scores close in magnitude to RGPS’s original signal score. For example, the PRR signal score for the Pfizer- BioNTech vaccine increased from 0.37 to 1.47 (297%) on W17, and 0.76 to 2.3 (202%) on W39. Similarly, the EBGM signal score increased from 0.35 to 1.47 (320%) on W17, and 0.66 to 1.48 PSI-HHS-000001627349 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 67 of 199 —

17 (124%) on W39. In addition, we found that these masking sources (i.e., the Varicella and VARZOS vaccines) did not change over time and remained consistent at both time periods that were evaluated. 3.3. Myocarditis and Pericarditis Myocarditis and pericarditis refer to inflammation of the heart muscle, and outermost layer of the heart respectively. Myocarditis and pericarditis are both thought to be caused by viral infections, and symptoms include chest pain, shortness of breath, and irregular heartbeat appearing within several days after the second dose of the mRNA vaccines. Several case reports of myocarditis and pericarditis developing rapidly after the first and second doses of the mRNA vaccines have been published[36-40], as well as several retrospective studies[11, 41-44] identifying it as a rare complication of the vaccines. One study on mice suggests that inadvertent intravenous injection of COVID-19 mRNA vaccines may induce myopericarditis[45]. The risk of myocarditis following vaccination has been observed to be highest among young males. The CDC has recognized the association with the COVID-19 mRNA vaccines[2], and both myocarditis and pericarditis now appear on the product labels (warning section) of the vaccines[46, 47]. As of week 39, there are 4690 reports of myocarditis for the mRNA vaccines (3515 Pfizer- BioNTech, 1175 Moderna), and 3079 reports of pericarditis for the mRNA vaccines (2408 Pfizer- BioNTech, 671 Moderna) in the VAERS system. Relative to the total number of cases for these AEs, 87% of myocarditis cases, and 83% of pericarditis cases are associated with the mRNA COVID-19 vaccines. PSI-HHS-000001627350 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 68 of 199 —

18 The changing age distribution of COVID-19 vaccine recipients can be observed in the progression of the time series. Figure 1 shows that both the RGPS and MGPS signal scores for myocarditis were initially not indicative of a safety signal, but around weeks 19-21 (week ending May 30, 2021) as the COVID-19 vaccines were made available in the U.S. to people under 65, a substantial increase in both signal scores can be observed. At this point RGPS and MGPS start diverging, with MGPS remaining on point and RGPS showing a gradual increase from a signal score of 2.3 to above 9.0 (Pfizer-BioNTech) and 1.5 to above 5.0 (Moderna). Similar trends of signal score progression are observed for pericarditis, with a slight decrease in RGPS signal scores around weeks 31-33 onwards. Averaged across the time series, signal scores for myocarditis tend to be 24% larger for the Pfizer-BioNTech vaccine than for the of Moderna vaccine, and 35%-38% larger for pericarditis. The size of the masking effect for myocarditis is ranked second for the AEs of interest with an average value around 190%. For pericarditis the effect size is 70%. Like the herpes zoster evaluation, the sources of masking for myocarditis were evaluated based on the process described in Section 2.5. The two time periods examined were W19 and W39. RGPS automatically selected 20 (W19) and 39 (W39) vaccine predictors for the myocarditis regression model. The strongest predictors for both time points were a set of three Smallpox vaccines (at the manufacturer level), which is consistent with published reports recognizing myocarditis as a rare AE of the Smallpox vaccine[48-50]. Upon removal of all reports containing the Smallpox vaccines on W19, the PRR, EBGM, and IC signal scores indeed reverted to larger signal scores close in magnitude to RGPS’s original signal score. The PRR signal score for the Pfizer-BioNTech vaccine increased from 1.44 to 2.48 (72%), PSI-HHS-000001627351 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 69 of 199 —

19 and for the Moderna vaccine from 0.8 to 1.34 (67%). Similarly, the EBGM signal score for the Pfizer-BioNTech vaccine increased from 1.44 to 2.17 (51%), and from 0.94 to 1.42 (51%) for the Moderna vaccine. As more data accumulated in VAERS, the Pfizer-BioNTech and Moderna COVID- 19 vaccines were also identified by RGPS as potential maskers. In this case masking each other for the myocarditis AE. On W39 the Pfizer-BioNTech vaccine was identified by RGPS as the strongest masker. Removing all reports containing the Pfizer-BioNTech vaccine led to a substantial increase in signal scores for the Moderna-myocarditis association. The PRR signal score increased from 1.2 to 4.98 (315%), and the EBGM score increased from 1.32 to 2.13 (61%). This demonstrates how the Pfizer-BioNTech vaccine is masking the Moderna vaccine, and how masking sources may evolve over time. In addition to the COVID-19 vaccines, the Smallpox vaccines were still identified by RGPS as strong sources of masking on W39. Removing both Smallpox and Pfizer-BioNTech vaccines led to the following additional increases for the Moderna association: PRR increased from 4.98 to 8.14 (63%), and EBGM increased from 2.13 to 2.4 (13%). Similarly, removing the Smallpox and Moderna vaccines led to the following increases for the Pfizer-BioNTech-myocarditis association: PRR increased from 5.42 to 10.96 to 17.94 (230%), and EBGM increased from 1.94 to 2.02 to 2.12 (9%). 3.4 Appendicitis Appendicitis is an inflammation of the appendix usually caused by an obstruction of the appendiceal lumen; however, the exact etiology of acute appendicitis is often unknown. Appendicitis is the most common cause of acute abdominal pain requiring surgery. If left untreated, acute appendicitis can result in serious complications, such as peritonitis or abscess formation[51, 52]. According to the Pfizer-BioNTech COVID-19 Vaccine Fact Sheet for Healthcare PSI-HHS-000001627352 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 70 of 199 —

20 Providers, appendicitis was reported as a serious AE in a clinical trial for 8 vaccine participants and 4 placebo participants (Pfizer-BioNTech COVID-19 Vaccine = 10,841; placebo = 10,851), but not during post-authorization experience[46]. The Moderna COVID-19 Vaccine Fact Sheet for Healthcare Providers does not mention appendicitis as an AE in clinical trials or in post- authorization experience[47]. However, both the Pfizer-BioNTech and Moderna Fact Sheets for Healthcare Providers mention lymphadenopathy as reported adverse events during clinical trials. Barda et al demonstrated an elevated risk ratio for appendicitis (risk ratio, 1.40; 95% CI, 1.02 to 2.01) with the Pfizer-BioNTech COVID-19 Vaccine in a mass nationwide vaccination setting[53]. As of week 39, there are 725 reports of appendicitis for the mRNA vaccines (537 Pfizer-BioNTech, 188 Moderna) in the VAERS system. As shown in Figure 2, both MGPS and RGPS showed extremely large signal scores early on that attenuated over time but remained high for RGPS with values above 3.7 for Pfizer-BioNTech and above 1.7 for Moderna. This early signaling by week 3 appeared even when the numbers of reports were small (15 Pfizer-BioNTech, 6 Moderna). RGPS and MGPS started diverging around week 11 likely due to masking. The figure shows a relatively large masking effect. Averaged across the time series, the size of the masking effect was high and around the value of 100% for both vaccines. Differences between the signal scores of the Pfizer- BioNTech and Moderna vaccines were highest among the AEs of interest with the Pfizer- BioNTech scores on average roughly 71% larger than those for the Moderna vaccine. This difference is consistent with the evidence from clinical trials and publications mentioned above. 3.5 Pulmonary embolism Pulmonary embolism (PE) is a sudden blockage in a lung artery. It usually happens when a blood clot breaks loose and travels through the bloodstream to the lungs. PE is a serious condition that PSI-HHS-000001627353 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 71 of 199 —

21 can cause permanent damage to the lungs, low oxygen levels in the blood, and damage to other organs in the body from not getting enough oxygen. PE can be life-threatening, especially if a clot is large, or if there are many clots[54]. Systematic reviews and meta-analyses showed high incidences of PE in COVID-19 patients[55, 56]. Barda et al. reported an elevated risk ratio for PE (risk ratio, 12.14; 95% CI, 6.89 to 29.20) for SARS-CoV-2-infected compared to uninfected persons [53]. Besides COVID-19 itself, it appears that COVID-19 vaccines increase the risk for PE: several authors reported the occurrence of PE, often in combination with vaccine-induced thrombotic thrombocytopaenia (VITT), following COVID-19 vaccination, mainly for adenovirus-based COVID- 19 vaccines[57-63]. Although no increased risk for PE was found by Klein et al. for mRNA vaccines[10] and by Barda et al. for Pfizer-BioNTech[53], some case reports described the occurrence of PE following vaccination with Pfizer-BioNTech[64-66]. To date, PE is not mentioned in the vaccine labels of the Pfizer-BioNTech and the Moderna COVID-19 vaccines. As of week 39, there are 5869 reports of PE for the mRNA vaccines (4394 Pfizer-BioNTech, 1475 Moderna) in the VAERS system. Figure 2 shows that both MGPS and RGPS exceed the signal threshold for PE already in W3 for both vaccines. In the following weeks, starting on week 9, RGPS departs from MGPS and stays on a value level about three-fold that of MGPS. Averaged across the time series, the size of the masking effect was high and around the value of 170% for both vaccines. The MGPS time series for Moderna decreases to below the signaling threshold in week 39, whereas RGPS remains well above the threshold. For Pfizer-BioNTech, MGPS and RGPS remain above the signaling threshold, with RGPS at times about three times the value of MGPS. PSI-HHS-000001627354 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 72 of 199 —

22 Compared to Moderna, signal scores for Pfizer-BioNTech are on average 17% (RGPS) and 12% (MGPS) larger. 3.6 Tinnitus Tinnitus is described as the sensation of hearing ringing, hissing, or other noises in one or both ears that is not caused by an external sound. Tinnitus can be intermittent or continuous and can vary in pitch and intensity. Prolonged exposure to loud sounds and a variety of other conditions can lead to tinnitus, however the mechanism responsible for tinnitus is unclear. Tinnitus has been linked to other vaccines such as hepatitis, rabies, measles, and H1N1 vaccines[67]. In COVID-19 vaccine trials prior to the release of the Pfizer-BioNTech and Moderna vaccines, no mention was made of the onset of tinnitus or worsening tinnitus for either vaccine. As early as March 2021, in a report from the United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA), 196 tinnitus cases among 33,207 vaccinated persons were recorded for the Pfizer-BioNTech vaccine[68], and since then several case reports linking tinnitus to the mRNA vaccines as well as to the Janssen and AstraZeneca vaccines have been published[68-71]. In addition, due to an apparently increased number of individuals experiencing tinnitus during the pandemic period, the connection between the vaccines and tinnitus received special attention in various media outlets and professional associations dedicated to tinnitus[72, 73]. To date, tinnitus is not mentioned in the vaccine labels. As mentioned in the Introduction, tinnitus is not contained in the set of AEs of interest recognized by various health organizations as do the previous six AEs. PSI-HHS-000001627355 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 73 of 199 —

23 As of week 39, there are 12296 reports of tinnitus for the mRNA vaccines (7649 Pfizer-BioNTech, 4647 Moderna) in the VAERS system. Interestingly, the number of reports for tinnitus is larger by a substantial amount than for any of the other AEs covered in this manuscript. Figure 2 shows that both MGPS and RGPS exceed the signal threshold early on for both vaccines and remain above the signaling threshold through the remaining time periods (excluding a brief crossing for MGPS and Moderna on weeks 9-15). On average the signal scores for the Pfizer-BioNTech vaccine are 30% larger than those for the Moderna vaccine. RGPS and MGPS start diverging on weeks 15- 17, with RGPS rapidly increasing to signal score values twice as large in a short amount of time. This appears correlated with the increase in the number of reports available throughout the period, and likely the dynamics of masking effects. Averaged across the time series, the size of the masking effect was high and around the value of 80% for both vaccines. Based on the process described in Section 2.5 we evaluated the sources of masking for tinnitus. The two time periods examined were W17 and W39. RGPS automatically selected 21 (W17) and 25 (W39) vaccine predictors for the tinnitus regression model. For W17 the strongest predictors and potential maskers identified by RGPS were the HPV4 (papilloma virus) vaccine, and the Janssen and Pfizer-BioNTech COVID-19 vaccines. Hence, on W17 two COVID-19 vaccines were already masking other associations; Janssen masking the Pfizer- BioNTech and Moderna COVID-19 vaccines, and the Janssen and Pfizer-BioNTech vaccines masking the Moderna vaccine. Removing all reports containing these three vaccines (HPV4, Janssen, and Pfizer-BioNTech) resulted in expected signal score increases for the Moderna- tinnitus association, with PRR increasing from 1.79 to 2.5 (40%), and EBGM increasing from 1.13 to 1.43 (27%). Expectedly, on W39 as more data accumulated in VAERS, the Pfizer-BioNTech and PSI-HHS-000001627356 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 74 of 199 —

24 Moderna vaccines were identified by RGPS as the strongest maskers (masking each other) in addition to the Janssen vaccine. On W39 the HPV4 vaccine was no longer identified as a strong masker. Removing reports containing the Janssen and Pfizer-BioNTech vaccines led to the following signal score changes for the Moderna-tinnitus association: PRR increasing from 1.8 to 5.5 (205%), and EBGM increasing from 1.18 to 1.69 (43%). Similarly, removing reports containing the Janssen and Moderna vaccines led to the following signal score changes for the Pfizer- BioNTech-tinnitus association: PRR increasing from 2.75 to 6.67 (143%), and EBGM modestly increasing from 1.57 to 1.71 (9%). This demonstrates that the Pfizer-BioNTech and Moderna vaccines may mask each other to varying degrees, in this case Pfizer-BioNTech having a larger effect on Moderna than vice versa. 3.7 Masking Statistics at the Database Level Table 4 displays counts for the number of potentially masked associations in VAERS categorized by vaccine type. The conditions that define a potentially masked association are provided in the Materials and Methods (Section 2.5 - candidate association for masking). The table shows that the likelihood of a masked association for the COVID-19 vaccines is 2.3%, which is roughly eight times larger than for non-COVID-19 vaccines (0.3%). This result clearly demonstrates the increased potential and susceptibility of VAERS COVID-19 vaccine surveillance to the problem of missed signals due to masking effects. PSI-HHS-000001627357 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 75 of 199 —

25 Table 4. VAERS counts of masked associations Num Associations Num Masked Associations All Vaccines 265987 1330 0.50% Non-COVID-19 Vaccines 241016 753 0.31% COVID-19 Vaccines 24971 577 2.31% Pfizer-BioNTech/Moderna 18588 458 2.46% PSI-HHS-000001627358 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 76 of 199 —

26 4. Discussion: The unprecedented dynamic and extent of reporting into VAERS for the novel class of COVID-19 vaccines, compounded by methodological limitations, predisposes signal detection to missed or delayed signals, which may limit our understanding of the risks associated with COVID-19 vaccines and delay their identification. We investigated seven AEs with various degrees of reported and statistical evidence that link them to the Pfizer-BioNTech and Moderna vaccines. Six of the AEs are largely recognized by various health authorities. The investigation enabled us to discover a potentially new AE (tinnitus), which is yet to be recognized by health authorities, but which has overwhelming statistical support in VAERS, as well as external supporting materials. In the investigation, we traced the evolution of signals related to the seven AEs during the course of the initial year of COVID-19 vaccination and the accompanying availability of COVID-19 vaccine AE reports made public in VAERS. This temporal evaluation led to several findings regarding the current state of signal detection for COVID-19 vaccine surveillance discussed in the following. We surmise that these findings are important not only for the COVID-19 vaccines currently approved PSI-HHS-000001627359 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 77 of 199 —

27 and investigated in this manuscript but are also important for any new COVID-19 vaccines which might be approved in the future, and likewise should also apply to any new vaccine (or drug) approved for use in the future. The results overall suggest that signals for COVID-19 vaccines may indeed be missed when generated by conventional signal detection methodologies that are currently utilized by pharmacovigilance organizations. For example, the tinnitus signal may have been overlooked partly due to the low signal scores produced for it by conventional methodologies. Similarly, due to inaccurate lower signal scores produced by conventional methodologies, statistical signals for the other six AEs might have been delayed. Fortunately, these other six AEs had already been well characterized by the FDA, CDC, and other sources. The findings demonstrate that one source of these missed or delayed signals is the notorious masking effect. Our investigation reveals that statistical masking is present, with varying degrees of strength (up to 300%), and that properly identifying and addressing this effect exposes strong statistical associations that would otherwise be deemed uninteresting. We found that masking is roughly eight times more likely to occur with COVID-19 vaccines than with other vaccines, which PSI-HHS-000001627360 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 78 of 199 —

28 can be explained by the novelty of these vaccines. We also found that masking sources may change over time. Expectedly, in earlier time periods of surveillance, other vaccines could mask signals associated with COVID-19 vaccines, whereas in later time periods, as more COVID-19 reports accumulate, the Pfizer-BioNTech and Moderna vaccines can mask each other and likely other vaccines. Masking effects have been traditionally addressed by removing cases containing the ‘offending’ product, by using stratification, or by employing regression techniques. However, each of these approaches requires to some extent identifying masking sources prior to signaling, which may limit the utility of signal detection in scenarios where masking and confounding is present. This investigation was made possible by using a methodology that automatically identifies and adjusts masking effects. The masking sources for three AEs identified by this methodology were verified using the traditional approach to address masking, i.e., by re-applying the conventional signaling approaches on data that excludes the masking sources. The results suggest that different signaling approaches may lead to drastically different results— a conclusion that is especially disconcerting in the context of COVID-19 surveillance. Unfortunately, in the absence of an ultimate benchmark, the question of which methodology to PSI-HHS-000001627361 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 79 of 199 —

29 rely on is still in debate. Nonetheless, the findings highlight the utility of a more advanced class of signal detection methodologies for COVID-19 vaccine surveillance. Given present-day computational power and recognized analytic approaches such as regression, there are few reasons to avoid the utilization of these approaches, at the very least to address acknowledged problems with conventional approaches, such as masking. The results also show that signal scores for the Pfizer-BioNTech vaccine are generally larger than those for the Moderna vaccine. These differences cannot be exclusively explained by the larger volume of reports available for the Pfizer-BioNTech vaccine. Neither do these larger signal scores suggest or provide evidence that the risk of AEs for the Pfizer-BioNTech vaccine is higher. However, it appears that the two vaccines mask each other and that the masking effect is larger in one direction (Pfizer-BioNTech) than the other (Moderna). It also appears that the VAERS data for COVID-19 vaccine surveillance is still evolving and susceptible to external influences, such as vaccination policies, publication influences, reporting practices, and updates to the MedDRA terminology. This in turn could contribute to signal score fluctuations resulting in time-dependent signaling uncertainty. PSI-HHS-000001627362 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 80 of 199 —

30 The mRNA Pfizer-BioNTech and Moderna vaccines have demonstrated to be highly effective in preventing infection and severe illness from COVID-19. They also appear to have acceptable safety profiles, suggesting that the benefits of COVID-19 vaccination outweigh the potential risk of AEs. Consequently, AEs such as those highlighted in this manuscript, which are also rare as far as we know, cannot be used to argue against vaccination. Moreover, signal detection is inherently an exploratory process. Therefore, associations flagged by signaling approaches do not imply causal relationships and always warrant further scrutiny, including those named in this manuscript. Notwithstanding, signal detection has the advantage of being fast and performed in “real time”. Analyses can be easily “tailored” to a specific age group or gender, time frame, and product type. Signal detection also has the advantage of casting a much wider net for AE reporting from millions or hundreds of millions of people and may identify rare AEs not seen in clinical trials. These advantages are critical in the “real time” and the “real world” environment of COVID-19 vaccine surveillance. PSI-HHS-000001627363 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 81 of 199 —

31 Acknowledgments Disclaimer The findings and conclusions expressed in this report are those of the authors and do not necessarily represent the views of the U.S. FDA or the federal government. Competing financial interests The authors declare no competing financial interests. DuMouchel William, Harpaz Rave, Van Manen Robert, Nip Alexander, Bright Steve are employed by Oracle Health Sciences. References 1. Szarfman, A., S.G. Machado, and R.T. O'Neill, Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA's spontaneous reports database. Drug Saf, 2002. 25(6): p. 381-392. 2. Harpaz, R., et al., Novel data-mining methodologies for adverse drug event discovery and analysis. Clin Pharmacol Ther, 2012. 91(6): p. 1010-21. 3. Wisniewski, A.F.Z., et al., Good Signal Detection Practices: Evidence from IMI PROTECT. Drug Safety, 2016. 39(6): p. 469-490. 4. Almenoff, J., et al., Perspectives on the Use of Data Mining in Pharmacovigilance. Drug Safety, 2005. 28(11): p. 981-1007. PSI-HHS-000001627364 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 82 of 199 —

32 5. CIOMS Working Group VIII. Practical Aspects of Signal Detection in Pharmacovigilance. 2010: CIOMS. 6. Guidance for Industry: Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment U.S. Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research (CDER)Center for Biologics Evaluation and Research (CBER) March 2005 https://www.fda.gov/media/71546/download. 7. Maignen, F., et al., Assessing the extent and impact of the masking effect of disproportionality analyses on two spontaneous reporting systems databases. Pharmacoepidemiol Drug Saf, 2014. 23(2): p. 195-207. 8. Juba, K.M., R.P. van Manen, and S.E. Fellows, A Review of the Food and Drug Administration Adverse Event Reporting System for Tramadol-Related Hypoglycemia. Ann Pharmacother, 2020. 54(3): p. 247-253. 9. Li, X., et al., Characterizing the incidence of adverse events of special interest for COVID- 19 vaccines across eight countries: a multinational network cohort study. medRxiv : the preprint server for health sciences, 2021: p. 2021.03.25.21254315. 10. Klein, N.P., et al., Surveillance for Adverse Events After COVID-19 mRNA Vaccination. JAMA, 2021. 11. Li, X., et al., Characterising the background incidence rates of adverse events of special interest for covid-19 vaccines in eight countries: multinational network cohort study. BMJ, 2021. 373: p. n1435. PSI-HHS-000001627365 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 83 of 199 —

33 12. Dumouchel, W., Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System. The American Statistician, 1999. 53(3): p. 177- 190. 13. Bate, A., et al., A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol, 1998. 54(4): p. 315-21. 14. Evans, S.J.W., P.C. Waller, and S. Davis, Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and Drug Safety, 2001. 10(6): p. 483-486. 15. DuMouchel, W. and R. Harpaz, Regression-adjusted GPS algorithm (RGPS), in Oracle White Paper, https://docs.oracle.com/health-sciences/empirica-signal- 811/ESIUG/Regression-Adjusted GPS Algorithm.pdf. 2012. 16. Oracle Empirica Signal. https://docs.oracle.com/en/industries/health-sciences/empirica- signal/9.1/index.html. 2021. 17. Iftikhar, H., et al., Bell's Palsy After 24 Hours of mRNA-1273 SARS-CoV-2 Vaccine. Cureus, 2021. 13(6): p. e15935-e15935. 18. Burrows, A., et al., Sequential contralateral facial nerve palsies following COVID-19 vaccination first and second doses. BMJ Case Reports, 2021. 14(7): p. e243829. 19. Ozonoff, A., E. Nanishi, and O. Levy, Bell's palsy and SARS-CoV-2 vaccines. The Lancet Infectious Diseases, 2021. 21(4): p. 450-452. 20. Cirillo, N. and R. Doan, The association between COVID-19 vaccination and Bell's palsy. The Lancet. Infectious diseases, 2021: p. S1473-3099(21)00467-9. PSI-HHS-000001627366 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 84 of 199 —

34 21. Wan, E.Y.F., et al., Bell's palsy following vaccination with mRNA (BNT162b2) and inactivated (CoronaVac) SARS-CoV-2 vaccines: a case series and nested case-control study. The Lancet Infectious Diseases. 22. Repajic, M., et al., Bell's Palsy after second dose of Pfizer COVID-19 vaccination in a patient with history of recurrent Bell's palsy. Brain, behavior, & immunity - health, 2021. 13: p. 100217-100217. 23. Tamaki, A., et al., Incidence of Bell Palsy in Patients With COVID-19. JAMA Otolaryngology–Head & Neck Surgery, 2021. 147(8): p. 767-768. 24. Renoud, L., et al., Association of Facial Paralysis With mRNA COVID-19 Vaccines: A Disproportionality Analysis Using the World Health Organization Pharmacovigilance Database. JAMA Internal Medicine, 2021. 181(9): p. 1243-1245. 25. Shemer, A., et al., Association of COVID-19 Vaccination and Facial Nerve Palsy: A Case- Control Study. JAMA Otolaryngology–Head & Neck Surgery, 2021. 147(8): p. 739-743. 26. Herpes Zoster. https://www.cdc.gov/shingles/about/symptoms.html. 27. Mohta, A., et al., Recurrent herpes zoster after COVID-19 vaccination in patients with chronic urticaria being treated with cyclosporine—A report of 3 cases. Journal of Cosmetic Dermatology. n/a(n/a). 28. Furer, V., et al., Herpes zoster following BNT162b2 mRNA Covid-19 vaccination in patients with autoimmune inflammatory rheumatic diseases: a case series. Rheumatology, 2021. 29. Chiu, H.-H., et al., Herpes zoster following COVID-19 vaccine: a report of three cases. QJM: An International Journal of Medicine, 2021. PSI-HHS-000001627367 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 85 of 199 —

35 30. Eid, E., et al., Herpes zoster emergence following mRNA COVID-19 vaccine. Journal of Medical Virology, 2021. 93(9): p. 5231-5232. 31. Arora, P., et al., Herpes zoster after inactivated COVID-19 vaccine: A cutaneous adverse effect of the vaccine. Journal of Cosmetic Dermatology. n/a(n/a). 32. Thimmanagari, K., et al., Ipsilateral Zoster Ophthalmicus Post COVID-19 Vaccine in Healthy Young Adults. Cureus, 2021. 13(7): p. e16725. 33. Moodley, A., et al., Severe Herpes Zoster Following Varicella Vaccination in Immunocompetent Young Children. Journal of Child Neurology, 2019. 34(4): p. 184-188. 34. Rueda, J.M., et al., Disseminated Herpes Zoster After Varicella Vaccination in a Healthy Boy. Actas Dermo-Sifiliográficas (English Edition), 2017. 108(6): p. 587-588. 35. Liang, M.G., et al., Herpes zoster after varicella immunization. Journal of the American Academy of Dermatology, 1998. 38(5): p. 761-763. 36. Kaul, R., et al., Myocarditis following COVID-19 vaccination. IJC Heart & Vasculature, 2021. 36: p. 100872. 37. Koizumi, T., et al., Myocarditis after COVID-19 mRNA vaccines. QJM: An International Journal of Medicine, 2021. 38. Alania-Torres, E., et al., Case Report: Probable Myocarditis After Covid-19 mRNA Vaccine in a Patient With Arrhythmogenic Left Ventricular Cardiomyopathy. Frontiers in Cardiovascular Medicine, 2021. 8(1351). 39. Ashaari, S., H.A. Sohaib, and K. Bolger, A case report: symptomatic pericarditis post- COVID-19 vaccination. European Heart Journal - Case Reports, 2021. 5(10). PSI-HHS-000001627368 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 86 of 199 —

36 40. Myocarditis and Pericarditis after Receipt of mRNA COVID-19 Vaccines. https://content.govdelivery.com/attachments/MDMBP/2021/05/28/file_attachments/1 839782/Clinician%20Letter Myocarditis 5.28.21.pdf. 2021. 41. Diaz, G.A., et al., Myocarditis and Pericarditis After Vaccination for COVID-19. JAMA, 2021. 326(12): p. 1210-1212. 42. Aye, Y.N., et al., Acute myocardial infarction and myocarditis following COVID-19 vaccination. QJM: An International Journal of Medicine, 2021. 43. Witberg, G., et al., Myocarditis after Covid-19 Vaccination in a Large Health Care Organization. New England Journal of Medicine, 2021. 44. Chelala, L., et al., Cardiac MRI Findings of Myocarditis After COVID-19 mRNA Vaccination in Adolescents. American Journal of Roentgenology, 2021. 45. Li, C., et al., Intravenous Injection of Coronavirus Disease 2019 (COVID-19) mRNA Vaccine Can Induce Acute Myopericarditis in Mouse Model. Clinical Infectious Diseases, 2021. 46. Pfizer-BioNTech Product Label. https://dailymed.nlm.nih.gov/dailymed/drugInfo.cfm?setid=908ecbe7-2f1b-42dd-94bf- f917ec3c5af8. 47. Moderna Product Label. https://dailymed.nlm.nih.gov/dailymed/drugInfo.cfm?setid=e0651c7a-2fe2-459d-a766- 0d59e919f058. 48. Halsell, J.S., et al., Myopericarditis Following Smallpox Vaccination Among Vaccinia- Naive US Military Personnel. JAMA, 2003. 289(24): p. 3283-3289. PSI-HHS-000001627369 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 87 of 199 —

37 49. Keinath, K., et al., Myocarditis secondary to smallpox vaccination. BMJ Case Reports, 2018. 2018: p. bcr-2017-223523. 50. Cardiac Adverse Events Following Smallpox Vaccination --- United States, 2003. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm5212a2.htm. 51. Jameson, J.L., et al., Harrison's Principles of Internal Medicine, 20th edition. 2018: McGraw Hill Medical. 52. Medlineplus.gov accessed September 27, 2021. 53. Barda, N., et al., Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting. New England Journal of Medicine, 2021. 385(12): p. 1078-1090. 54. MedlinePlus. https://medlineplus.gov/pulmonaryembolism.html. 55. Roncon, L., et al., Incidence of acute pulmonary embolism in COVID-19 patients: Systematic review and meta-analysis. European Journal of Internal Medicine, 2020. 82: p. 29-37. 56. Suh, Y.J., et al., Pulmonary Embolism and Deep Vein Thrombosis in COVID-19: A Systematic Review and Meta-Analysis. Radiology, 2021. 298(2): p. E70-E80. 57. Islam, A., et al., An Update on COVID-19 Vaccine Induced Thrombotic Thrombocytopenia Syndrome and Some Management Recommendations. Molecules, 2021. 26(16): p. 5004. 58. Asmat, H., et al., A rare case of COVID-19 vaccine-induced thrombotic thrombocytopaenia (VITT) involving the veno-splanchnic and pulmonary arterial circulation, from a UK district general hospital. BMJ Case Reports, 2021. 14(9): p. e244223. PSI-HHS-000001627370 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 88 of 199 —

38 59. Muster, V., et al., Pulmonary embolism and thrombocytopenia following ChAdOx1 vaccination. The Lancet, 2021. 397(10287): p. 1842. 60. Greinacher, A., et al., Thrombotic Thrombocytopenia after ChAdOx1 nCov-19 Vaccination. New England Journal of Medicine, 2021. 384(22): p. 2092-2101. 61. Bersinger, S., et al., Using Nonheparin Anticoagulant to Treat a Near-Fatal Case With Multiple Venous Thrombotic Lesions During ChAdOx1 nCoV-19 Vaccination-Related Vaccine-Induced Immune Thrombotic Thrombocytopenia. Critical Care Medicine, 2021. 49(9): p. e870-e873. 62. Malik, B., et al., Pulmonary embolism, transient ischaemic attack and thrombocytopenia after the Johnson &amp; Johnson COVID-19 vaccine. BMJ Case Reports, 2021. 14(7): p. e243975. 63. Clark, R.T., et al., Early Outcomes of Bivalirudin Therapy for Thrombotic Thrombocytopenia and Cerebral Venous Sinus Thrombosis After Ad26.COV2.S Vaccination. Annals of Emergency Medicine, 2021. 78(4): p. 511-514. 64. Al-Maqbali, J.S., et al., A 59-Year-Old Woman with Extensive Deep Vein Thrombosis and Pulmonary Thromboembolism 7 Days Following a First Dose of the Pfizer-BioNTech BNT162b2 mRNA COVID-19 Vaccine. Am J Case Rep, 2021. 22: p. e932946. 65. Esba, L.C.A. and M. Al Jeraisy, Reported adverse effects following COVID-19 vaccination at a tertiary care hospital, focus on cerebral venous sinus thrombosis (CVST). Expert Review of Vaccines, 2021. 20(8): p. 1037-1042. 66. Nune, A., et al., Multisystem inflammatory syndrome in an adult following the SARS-CoV- 2 vaccine (MIS-V). BMJ Case Reports, 2021. 14(7): p. e243888. PSI-HHS-000001627371 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 89 of 199 —

39 67. Okhovat, S., et al., Sudden onset unilateral sensorineural hearing loss after rabies vaccination. BMJ Case Reports, 2015. 2015: p. bcr2015211977. 68. Parrino, D., et al., Tinnitus following COVID-19 vaccination: report of three cases. International Journal of Audiology, 2021: p. 1-4. 69. Tseng, P.-T., et al., The reversible tinnitus and cochleopathy followed first-dose AstraZeneca COVID-19 vaccination. QJM: An International Journal of Medicine, 2021. 70. Buntz, B., “Is JJS COVID-19 Vaccine Linked to Tinnitus?”. https://www.drugdiscoverytrends.com/is-jjs-covid-19-vaccine-linked-to-tinnitus/. 2021. 71. Wichova, H., M.E. Miller, and M.J. Derebery, Otologic Manifestations After COVID-19 Vaccination: The House Ear Clinic Experience. Otology & Neurotology, 2021. 42(9). 72. American Tinnitus Association. https://www.ata.org/tinnitus-and-coronavirus. 73. British Tinnitus Association. https://www.tinnitus.org.uk/coronavirus-vaccines-and- tinnitus. PSI-HHS-000001627372 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 90 of 199 —

40 Supporting information S1 File. Description of the Regression-Adjusted GPS (RGPS) signal detection methodology S2 File. Excel table with all signal statistics referenced and evaluated in the manuscript. This includes the time series of signal statistics used in Figures 1 and 2, and other statistics underlying additional combinations of AE/vaccine/signaling methodology referenced in the manuscript. PSI-HHS-000001627373 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 91 of 199 —

Figure Click here to access/download;Figure;Fig 1 signal stats time series.png PSI-HHS-000001627374 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 92 of 199 —

Figure Click here to access/download;Figure;Fig 2 signal stats time series.png PSI-HHS-000001627375 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 93 of 199 —

Supporting Information Click here to access/download Supporting Information S1 RGPS Method Description.docx PSI-HHS-000001627376 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 94 of 199 —

Supporting Information Click here to access/download Supporting Information S2_VAERS_COVID19_TS_W39.xlsx PSI-HHS-000001627377 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 95 of 199 —

Attachment by reference: https://fda−my.sharepoint.com/personal/sarah_walinsky_fda_gov/Documents/CBE R/Response%20to%20Moderna.docx?web=1 PSI-HHS-000001628654 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 96 of 199 —

From: "Unger, Ellis" To: "Farizo, Karen" Subject: FW: COVID-19 paper Date: Mon, 7 Jun 2021 17:12:31 +0000 Importance: Normal Attachments: PDS_-_COVID19_safety_surveillance_and_masking.docx; CDER_Clearance_Request_for_Articles_Speeches_and_Other_Publications__Masking_Ass ociated_with_Early_COVID-19_Vaccine_Safety_Surveillance.pdf Inline-Images: image001.png Hi Karen, I hope all things are improving for you and yours. I received this paper to clear for a woman who works in one of the divisions in my office. We have two types of FDA disclaimers—the usual one where we say that the views expressed are not those of the FDA, etc., and an unusual disclaimer that says that the principles may not be consistent with FDA policy. This paper is about surveillance of adverse events after COVID vaccines. Is there someone in CBER you could recommend to provide a quick general comfort level on this paper? This is not what we do in CBER. Ellis From: Szarfman, Ana Sent: Friday, June 4, 2021 11:24 AM To: Unger, Ellis Subject: FW: COVID-19 paper Hi Ellis, Please refer to the attached paper and form for your clearance. Many thanks. --Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration PSI-HHS-000001622854 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 97 of 199 —

From: Stockbridge, Norman L Sent: Friday, June 4, 2021 7:14 AM To: Szarfman, Ana Subject: RE: COVID-19 paper I marked a few suggested edits. Good luck, Norman From: Szarfman, Ana Sent: Thursday, June 3, 2021 2:30 PM To: Stockbridge, Norman L Subject: RE: COVID-19 paper Many thanks Norman. Please refer to the attached paper and OND clearance form. From: Stockbridge, Norman L Sent: Tuesday, May 25, 2021 9:00 AM To: Szarfman, Ana Subject: RE: COVID-19 paper Fine with me. When it is ready to submit, send it to me with the OND clearance form. Thanks, Norman From: Szarfman, Ana Sent: Tuesday, May 25, 2021 7:48 AM To: Stockbridge, Norman L Subject: COVID-19 paper Hi Norman, Will it be OK with you if I am a co-author of the attached paper? Many thanks, Ana PSI-HHS-000001622855 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 98 of 199 —

Masking Associated with Early COVID-19 Vaccine Safety Surveillance DuMouchel W.1, Harpaz R.1*, Szarfman A.2, Van Manen R.1, Nip A.1, Bright S.1, Al-Ansari M1. 1 Oracle Health Sciences, Bedford, MA, United States 2 U.S. FDA, Silver Spring, MD, United States * Corresponding author Abstract: Purpose: raise awareness to the problem of masking associated with early stage VAERS COVID-19 vaccine surveillance, which can lead to missed signals. Provide a preliminary investigation of these masking effects. Methods: three signal detection methodologies: MGPS, PRR, and a new methodology called RGPS that can control masking were applied to six years of VAERS reports, consisting of 17 weeks of COVID-19 vaccine reports. Results: several statistically masked associations are identified, some of which are also listed in the product labels and reported in clinical trials. The most extreme case of masking is statistically verified by removing reports containing the ‘offending’ masker. RGPS appears to provide a reasonable middle ground between MGPS and PRR with respect to the veracity and number of signals produced by each methodology. Conclusions: statistical masking is present and should be considered in the context of early stage COVID- 19 signal detection. RGPS can address masking and confounding effects, which cannot be properly controlled by conventional signal detection methodologies. Purpose: As the world contends with rolling out massive scale vaccination programs to end the COVID-19 pandemic, identifying and studying adverse events related to these vaccines is critically urgent. The Vaccine Adverse Event Reporting System (VAERS), co-administered by the US Food and Drug Administration and the Centers for Disease Control and Prevention (CDC), is one of several systems used to monitor adverse events that occur after vaccination, including the COVID-19 vaccine. Like other safety surveillance systems, VAERS offers the opportunity to rapidly flag potential safety issues related to vaccines–a process usually known as signal detection. Computational methodologies for signal detection have been routinely applied to safety surveillance systems for over 20 years and have become a de facto standard[1]. Given its impact on public health, signal detection is still an active area of research, and since its conception multiple guidances[2-5] have been published with practice recommendations as well as admonitions concerning data and methodological limitations. In particular, ‘masking’ is a problem that may result in missed signals. Masking[3, 6, 7] is an artifact of conventional disproportionality statistics used for signal detection that are based on 2x2 contingency tables. A masked relationship between a target product and target adverse event can emerge when another product/s is frequently reported with the target event while making the background rate for the target event considerably large. This larger background rate can then make the relationship between the PSI-HHS-000001639129 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 99 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON target product and the target event appear less unusual, hence masking the true relationship. Conditions that make signal detection especially vulnerable to masking effects include: smaller volume of cases such as in VAERS, relationships involving rare events, and relationships involving newer products (i.e., emerging signals). As such, the early stages of COVID-19 vaccine surveillance make signal detection especially susceptible to masking. Masking effects can be ameliorated by removing cases containing the ‘offending’ product, by using stratification, and by employing regression techniques, all of which require to some extent identifying masking sources prior to signaling. Regression-Adjusted GPS (RGPS)[8] is a signal detection methodology that is designed to combine the application simplicity of conventional signal detection methodologies with the power of regression to produce disproportionately statistics with adjusted background rates. [Among other, these adjusted Commented [SNLI]: Among other things? background rates can control masking effects. ‘Stockbridge, Norman L 2021-06-04 07:04:00 The purpose of this report to raise awareness to masking in the context of early stage VAERS COVID-19 vaccine surveillance and provide a preliminary investigation of these masking effects using the RGPS methodology. Methods: Data: The investigation was done using VAERS reports from Jan. 01, 2015 to May 01, 2021-the latest Commented [SNL2]: These vaccine reports, and 6 years of other vaccine reports used as background. In total 466,401 reports were sete Cone used. Of those 145,300 reports included the COVID-19 vaccine from three manufacturers: 2021-06-04 07:04:00 Pfizer/BionTech (39.6%), Moderna (45.9%), and Janssen (14.5%). Events were represented at the MedDRA. Preferred Term (PT) level. To investigate adverse events commonly reported across the three manufacturers, products were represented at the ‘vaccine type’, e.g., ‘COVID19’ rather than at the ‘manufacturer’ level, e.g., ‘COVID19_PFIZER/BIONTECH’. Methods: The RGPS methodology operates by fitting separate Bayesian logistic regression models to each target adverse event. RGPS automatically selects two types of predictors to be included in each regression model: (1) products that are statistically associated with the target event, which are represented as indicator variables, and (2) stratification categories grouped by target event rates, which are represented as multiple regression intercepts. Rather than using the fitted regression coefficients to compute signal scores (disproportionalities), RGPS computes observed to expected ratios of counts similar to conventional methodologies. The expected counts are computed by summing the regression predicted probabilities of the target event across all reports mentioning the target product under the null hypothesis of no association between the target product and target event. The null hypothesis probabilities are computed by setting the coefficient of the target product to zero if selected as a model predictor. This results in adjusted expected counts (background rates) that can address masking. The final signal score {and its intervals) are computed using Bayesian shrinkage of observed to expected counts similar to MGPS[9]. Complete details of the RGPS methodology are presented in reference[8].. The results of RGPS were compared against those of the MGPS and the PRR[10] methodologies. Stratification categories used for RGPS and MGPS were: age (10 levels) and gender (3 values). Stratification by ‘report year’ could not be applied because COVID-19 VAERS reports dominate reporting from PSI-HHS-000001639130 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 100 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON December 2020. We applied the canonical version of PRR, which does not require stratification. For RGPS and MGPS we generated both the point estimates, labeled ERAM and EBGM respectively, and their associated credible intervals labeled EROS-ER95 and EBOS-EB95 respectively. Unless specified otherwise, signal scores are represented by the point estimates. The generation of signal scores for the three methodologies and analysis thereof was done using Oracle Empirica Signal 9.1[11]. Results: Figure 1 provides a high-level comparison of signal scores generated by RGPS, MGPS, and PRR for COVID- 19 vaccine-related adverse events. The comparison is illustrated by means of sector (heat) maps. The figure positions RGPS as a compromise between PRR and MGPS with respect to the number of signals produced by each methodology. Setting the signal score cutoffto 2, PRR produced 3,695 signals, whereas RGPS 467 signals, and MGPS 74. Setting the cutoff to 5, PRR produced 1,492 signals, whereas RGPS 74 signals, and MGPS only 7. The canonical version of PRR is generally known to produce more signals at the expense of possibly a larger number of false alerts compared to MGPS, but as a result may miss fewer signals than MGPS. Which of the methods providesa better tradeoffis still a subject of debate, and beyond the scope of this report. We conjecture that the larger number of signals produced by RGPS compared to MGPS is due to masking. Both RG*PS and PRR show a ‘hot’ zone of cardiac events some of which were recently highlighted by the cDC[12]. PRR RGPS Figure 1. Sector-map comparison of signal scores generated by RGPS, MGPS, and PRR for COVID-19 vaccine-related adverse events. The major rectangular areas represent system-organ-classes (SOCs) and the smaller interior rectangles represent preferred terms (PTs). The box sizes represent the number of cases, and colors represent the size/scale of signal scores, ranging {from green-smaller scores to red-larger scores. Table 1 provides signal statistics for the top 20 masked COVID-19 vaccine associations flagged by RGPS. A candidate association for masking is defined as one whose signal statistics satisfy the following condition: EROS > EB95 and EROS > 1 and EBO5 <1 That is, an association where RGPS and MGPS disagree by producing non-overlapping credible intervals (EROS > EB95) with RGPS| interval above the boundary of no association (ERO5>1) and thatofMGPS below ‘Commented [SNL3]: RGPS's Stockbridge, Norman L 2021-06-04 07:08:00 PSI-HHS-000001639131 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 101 of 199 —

or including the boundary of no association (EB05 < 1). The associations are ranked by the magnitude of the masking effect, which we define as the ratio of RGPS’ and MGPS’ signal scores, i.e., ERAM/EBGM. The table also provides signal scores for the IC statistic[13], which were close to those of MGPS. Several of the adverse events listed in Table 1 have been listed in the product labels and reported in clinical trials[14], and others are known issues associated with vaccine administration. While the size of the signal scores in Table 1 would normally not be considered large enough to warrant immediate action, they nonetheless clearly illustrate the masking effect. Herpes zoster (shingles, 903 reports) was ranked highest in terms of the masking ratio equal to 1.20/0.31=3.91. PRR produced a signal score of 0.17 for the same association. RGPS automatically selected 39 product predictors and 16 stratification groups for the Herpes zoster regression model (the ‘COVID19’ vaccine type was not one of the predictors). The strongest predictor and likely the culprit for masking was the VARZOS vaccine--a combination vaccine of Varicella and Zoster. Upon removal of all reports containing the VARZOS vaccine (the alternative approach for controlling masking) the PRR and EBGM signal scores reverted to 1.76 and 1.12 respectively, supporting RGPS’ finding of the offending masker. Herpes zoster has also been reported in an observation study[15]. Table 1.Top 20 masked COVID-19 vaccine associations flagged by RGPS. N: number of reports with the adverse event and COVID-19. E_RGPS: adjusted expected counts produced by RGPS. ERAM: signal score produced by RGPS. ER05/ER95: lower and upper bounds of RGPS’ credible intervals. E_MGPS: adjusted expected counts produced by MGPS. EBGM: signal score produced by MGPS. EB05/EB95: lower and upper bounds of MGPS’ credible intervals. PRR: signal score produced by PRR. 2^IC: 2 to the power of the signal score produced by the IC calculation. Masking ratio: ERAM/EBGM. Event (MedDRA PT) N E_RGPS ER05 ERAM ER95 E_MGPS EB05 EBGM EB95 PRR 2^IC Masking Ratio Herpes zoster 903 752.2 1.14 1.20 1.27 2940.2 0.29 0.31 0.33 0.17 0.31 3.91 Numb chin syndrome 10 1.8 2.63 4.79 7.46 5.2 0.96 1.58 2.49 41.28 1.85 3.03 Cholecystitis acute 10 1.8 2.64 4.80 7.48 5.1 0.96 1.58 2.50 41.28 1.86 3.03 Influenza virus test negative 354 303.6 1.07 1.17 1.27 497.4 0.65 0.71 0.78 0.52 0.71 1.64 Pneumonia 612 462.3 1.24 1.32 1.41 712.6 0.80 0.86 0.92 0.60 0.86 1.54 White blood cell count increased 491 432.6 1.05 1.14 1.22 619.3 0.74 0.79 0.85 0.62 0.79 1.43 Underdose 468 345.8 1.25 1.35 1.46 488.1 0.89 0.96 1.03 0.70 0.96 1.41 Syringe issue 313 209.9 1.35 1.49 1.63 289.0 0.98 1.08 1.18 1.11 1.08 1.38 Oral herpes 247 188.0 1.18 1.31 1.45 259.1 0.86 0.95 1.06 1.28 0.95 1.38 Sepsis 192 127.5 1.33 1.50 1.68 175.3 0.97 1.09 1.23 0.82 1.10 1.37 Product administered to patient of inappropriate age 1132 908.9 1.19 1.25 1.31 1158.6 0.93 0.98 1.03 0.38 0.98 1.27 Abdominal discomfort 1262 1038.9 1.16 1.21 1.27 1272.1 0.95 0.99 1.04 1.41 0.99 1.22 Head injury 344 274.0 1.15 1.25 1.37 333.3 0.94 1.03 1.13 0.83 1.03 1.22 Seizure 884 818.0 1.02 1.08 1.14 986.4 0.85 0.90 0.95 0.42 0.90 1.21 Disturbance in attention 431 382.2 1.04 1.13 1.22 454.8 0.87 0.95 1.02 0.95 0.95 1.19 Pallor 1171 1029.9 1.08 1.14 1.19 1219.7 0.92 0.96 1.01 0.58 0.96 1.18 Vaccination site swelling 569 516.3 1.03 1.10 1.18 609.5 0.87 0.93 1.00 0.96 0.93 1.18 Syncope 2666 2292.0 1.13 1.16 1.20 2681.5 0.96 0.99 1.03 0.87 0.99 1.17 Rash pruritic 1838 1640.8 1.08 1.12 1.16 1880.6 0.94 0.98 1.02 1.39 0.98 1.15 Loss of consciousness 2259 1947.6 1.12 1.16 1.20 2213.4 0.99 1.02 1.06 0.97 1.02 1.14 PSI-HHS-000001639132 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 102 of 199 —

Conclusion: The novelty and early stages of COVID-19 vaccine surveillance, compounded by the size of the VAERS database predisposes signal detection to the notorious masking effect. Left unattended masking could lead to costly missed signals. This report demonstrates that statistical masking is indeed present and at the very least should be considered in the context of COVID-19 signal detection. The report also demonstrates the potential utility of a new signal detection methodology called RGPS that can address masking and confounding effects that cannot be properly controlled by conventional signaling methodologies. Signal detection is inherently exploratory, therefore the associations named in this report do not imply causal relationships. Disclaimer: The findings and conclusions expressed in this report are those of the authors and do not necessarily represent the views of the U.S. FDA or the federal government. Competing financial interests: The authors declare no competing financial interests. DuMouchel W., Harpaz R., Van Manen R., Nip A., Bright S., Al-Ansari M. are employed by Oracle Health Sciences. References: 1. Szarfman, A., S.G. Machado, and R.T. O'Neill, Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA's spontaneous reports database. Drug Saf, 2002. 25(6): p. 381-392. 2. Wisniewski, A.F.Z., et al., Good Signal Detection Practices: Evidence from IMI PROTECT. Drug Safety, 2016. 39(6): p. 469-490. 3. Almenoff, J., et al., Perspectives on the Use of Data Mining in Pharmacovigilance. Drug Safety, 2005. 28(11): p. 981-1007. 4. CIOMS Working Group VIII. Practical Aspects of Signal Detection in Pharmacovigilance. 2010: CIOMS. 5. Guidance for Industry: Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment U.S. Department of Health and Human Services. Food and Drug Administration. Center for Drug Evaluation and Research (CDER)Center for Biologics Evaluation and Research (CBER) March 2005 https://www.fda.gov/media/71546/download. 6. Maignen, F., et al., Assessing the extent and impact of the masking effect of disproportionality analyses on two spontaneous reporting systems databases. Pharmacoepidemiol Drug Saf, 2014. 23(2): p. 195-207. 7. Juba, K.M., R.P. van Manen, and S.E. Fellows, A Review of the Food and Drug Administration Adverse Event Reporting System for Tramadol-Related Hypoglycemia. Ann Pharmacother, 2020. 54(3): p. 247-253. 8. DuMouchel, W. and R. Harpaz, Regression-adjusted GPS algorithm (RGPS), in Oracle White Paper, https://docs.oracle.com/health-sciences/empirica-signal-811/ESIUG/Regression- Adjusted GPS Algorithm.pdf. 2012. 9. Dumouchel, W., Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System. The American Statistician, 1999. 53(3): p. 177-190. PSI-HHS-000001639133 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 103 of 199 —

10. Evans, S.J.W., P.C. Waller, and S. Davis, Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and Drug Safety, 2001. 10(6): p. 483-486. 11. Oracle Empirica Signal. https://docs.oracle.com/en/industries/health-sciences/empirica- signal/9.1/index.html. 2021. 12. Myocarditis and Pericarditis after Receipt of mRNA COVID-19 Vaccines. https://content.govdelivery.com/attachments/MDMBP/2021/05/28/file attachments/1839782/ Clinician%20Letter Myocarditis 5.28.21.pdf. 2021. 13. Bate, A., et al., A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol, 1998. 54(4): p. 315-21. 14. DailyMed, COVID-19 Vaccine Product Labels. https://dailymed.nlm.nih.gov/dailymed/search.cfm?labeltype=all&query=COVID-19+vaccine. 2021. 15. Furer, V., et al., Herpes zoster following BNT162b2 mRNA Covid-19 vaccination in patients with autoimmune inflammatory rheumatic diseases: a case series. Rheumatology, 2021. PSI-HHS-000001639134 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 104 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Date of Request If clearance is requested to meet a deadline, CDER Clearance Request for please provide deadline date. Articles, Speeches, and Other 06/03/2021 Publications 6108/2024 1. Person to Contact 2. Phone Number 3. Email address Ana Szartman — a 4. Title of Article, Speech, or Other Publication Masking Associated with Early COVID-19 Vaccine Safety Surveillance 5. Authors DuMouchel W.1, Harpaz R.1*, Szarfman A.2, Van Manen R.1, Nip A.1, Bright S.1, Al-Ansari M1 6. Author Affiliations 1. Oracle Health Sciences, Bedford, MA, United States; 2. Division of Cardiology and Nephrology, OCHEN, OND, CDER, FDA 7. Details of Article or Speech Li Regulatory Summary‘ Letter Journal Journal Article ie ti " v 1 Review Article Editorial | pnarmacoepidemiology and Drug Safety’ Peer Reviewed Research ] Chapter Title Book Title Book Chapter Meeting Abstract | Meeting Title ‘Meeting Sponsor ‘Meeting Date Meeting Location Symposium/ Workshop "i Talk Title Meeting Sponsor Meeting Date Meeting Location Submitter Assurances 8. This article or speech was completed as: Assigned Work 9. The article or speech is a result of research involving human subjects, specimens, or subject level data (e.g., NDA data)? © Yes @ No 10. If the article or speech is reporting the results of CDER research, all of the methodological details, analytical If yes — Please provide Research Involving Human Subjects Committee (RIHSC) Protocol Number procedures, and underlying source data supporting the conclusions are documented and available for inspection? © Yes O No 41. To the best of my knowledge, this article or speech NOT contains non-public information. 12. To the best of my knowledge, the statements and conclusions in this article or speech conform with FDA policy. © Yes © No 13. The following divisions and offices have reviewed this article or speech: 14. Submitter Signature (digital) Ang Szarfman -S Sr Review and Clearance 14. First Line Reviewer Name ‘Signature (digital) Date Free of non-public information YES ional - by off nn ‘ byomnce) Conforms with FDA policy YES Pea 's = 48. Second Line Reviewer ‘cu N 7” ome (optional. by office). | FTSe.f non-public information select lame ‘Signature (digital) Conforms with FDA policy Select. 46. Clearance Official | Free of non-public information select | Comments Conforms with FDA policy select. Article or Speech is: Lcieared [)Not cleared []Returned for Revisions Isa disclaimer required? Select ne (if Yes, [] Disctaimer 1 [Disclaimer 2) Name ‘Signature (digital) Date CLEAR FORM | SAVE FORM PSI-HFSt690001640809 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 105 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Szarfman, Ana" To: "Marks, Peter" , "Anderson, Stever , "Forshee, Richard" 5 , "Witten, Celia (CBER) |, "Stockbridge, Norman L" Subject: RE: NYT - As Millions Get Shots, F.D.A. Struggles to Get Safety Monitoring System Running Date: Mon, 1 Mar 2021 19:08:29 +0000 Importance: Normal Attachments: Ana Szarfman_- Briefing of Dr Peter Marks - March _1_2021_at_100_PM..pdf; IBMsRetreat_From_Watson_HighlightsBroader_AI Struggles inHealth -_WSJ.pdf Inline-Images: image003.png Hi All, | enjoyed the opportunity to have this discussion with you. | have attached an updated presentation and the WSJ article on IBM retreating from Watson. Warmest regards, --Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration TN U.S. FOOD & DRUG ADMINISTRATION -----Original Appointment-- From: Marks, Peter Sent: Wednesday, February 17, 2021 6:50 PM To: Marks, Peter; Anderson, Steven; Forshee, Richard; Szarfman, Ana; Dal Pan, Gerald; Ball, Robert; Witten, Celia (CBER); Stockbridge, Norman L Subject: NYT - As Millions Get Shots, F.D.A. Struggles to Get Safety Monitoring System Running When: Monday, March 1, 2021 1:00 PM-2:00 PM (UTC-05:00) Eastern Time (US & Canada). Where: WebEX Importance: High PSI-HHS-000002038542 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 106 of 199 —

-- Do not delete or change any of the following text. -- When it's time, join your Webex meeting here. From: Szarfman, Ana Sent: Sunday, February 14, 2021 9:31 AM To: Marks, Peter Cc: Stockbridge, Norman L Subject: NYT - As Millions Get Shots, F.D.A. Struggles to Get Safety Monitoring System Running Hi Peter. Thanks so much for your work. Regarding the NYT article, I am quite concerned that the distributed network of EHRs that the Sentinel System uses does not have mortality data, and that they (and we) are not clamoring to fix this problem. It still takes several years to collect mortality data. Indeed there are no incentives to update electronic health records with mortality data. https://www.nytimes.com/2021/02/12/health/covid-vaccine-how-safe.html? referringSource=articleShare  Let me know if you want Bill DuMouchel and I to discuss our proposal for more effective monitoring, including the need to use an updated algorithm by DuMouchel for data mining spontaneous reports at CDER and CBER. Many thanks, --Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration PSI-HHS-000002038543 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 107 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON More ways to join: Join from the meeting link Join by meeting number Tap to join from a mobile device (attendees only) Join by phone lobal call-in numbers oll-free calling restrictions If you are a host, click here to view host information. Need help? Go to hitps://help.webex.com PSI-HHS-000002038544 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 108 of 199 —

I am humbled and thankful for the tremendously difficult and amazingly hard work you are all doing and for all your successes Many thanks for your invitation to exchange thoughts 1PSI-HHS-000002134642 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 109 of 199 —

Mortality data to address the COVID-19 public health and analytical needs of the users of the information More timely detection of adverse events and associated risk factors that we may not know how to formulate a priori Ana Szarfman, MD, PhD, FAMIA, Medical Officer, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), Safety Data Mining Developer and Medical Informatics Analyst, Division of Cardiology and Nephrology, CDER, FDA Dr. Peter Marks Briefing, February 29, 2021 2PSI-HHS-000002134643 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 110 of 199 —

Mortality data linked to EHRs and Claims data 3PSI-HHS-000002134644 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 111 of 199 —

• There is no universal tool to CENTRALLY capture mortality data in the U.S. • Multiple surveillance and analytical systems cannot easily access data on DEATHS OCCURRING IN AN OUT-OF-HEALTHCARE SETTING using EHRs or claims data. • If a patient DIES WHILE IN THE HOSPITAL, the death will be coded as such in the EHR, but not if the death occurs with patients being discharged to hospice care, to a nursing home, or to their home. • When such information is needed for RESEARCH or to be linked to CLAIMS data (such as BCBS), it is typically obtained from PRIVATE SERVICES who collect the information from various sources likeFUNERAL HOMES AND OBITUARIES IN LOCAL NEWSPAPERS. (a very time consuming and very inefficient process) 4PSI-HHS-000002134645 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 112 of 199 —

• Deaths occurring at home, on the street, or when the subject is homeless, as well as autopsies are not treated as medical clinical service events. • The death certificate information does not get back to the medical record. • Death certificates are notoriously sparse and incomplete. • They are collected by a multitude of governing localities, and then gathered by the State. The types of reportable deaths are determined by federal, state or local laws. 5PSI-HHS-000002134646 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 113 of 199 —

• The States receive the certificates and submit them to the National Death Index (NDI) where they get adjudicated and added to the NDI final file annually. • The NDI website provides death information to researchers; but the process requires funding support : https://www.cdc.gov/nchs/data/ndi/ndi_application.pdf https://www.cdc.gov/nchs/ndi/portal.htm • These requests are usually applied to 500 patients in a research project or clinical trial. We need mortality data for over 300 million individuals 6PSI-HHS-000002134647 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 114 of 199 —

• Clinical trials contain COMPREHENSIVE AUTOPSY information but not the EHRs. • Registries, like cancer registries or transplant registries systematically collect death information, but they represent relatively small siloes of information disjoined from EHRs. • CMS and DoD and VA hospitals GET FEEDBACK SEEDS from the SSA ofDEATHS THAT NEED TO BE REMOVED FROM THEIR BENEFICIARY LIST. • NOT SURE HOW OPTIMIZED THESE SYSTEMS ARE FOR INCLUDING DEATHS IN THEIR ANALYSES OF CLINICAL DATA. • Outside these Federal systems, the SSA STOPPED MAKING THIS INFORMATION AVAILABLE 5 YEARS AGO because of a potential for fraud (people applying for loans using fake Social Security codes) and would only typically provide an answer for a specific person, and inform when and how they died. 7PSI-HHS-000002134648 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 115 of 199 —

• Death information IS NOT CONSIDERED PRIVATE HEALTH INFORMATION. IT IS PUBLIC INFORMATION. • Family members, life insurance companies, and voter registrations are granted such access. • The key problem is that this information is not collected and made available in a timely way. • If there is a time, it is now for the Federal Government to act and improve the ability to centrally collect death information or to expedite the link of NDI information to address the necessary research and public health needs of all the analysts of the information. • The government can define the requirements and precautions that the receiving parties will need to put in place to avoid fraud. • The government can also monitor and prevent fraud activity. • Correction of this situation will require awareness of this problem, know-how, efforts, funding, and regulatory support. 8PSI-HHS-000002134649 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 116 of 199 —

The superior performance of the RGPS algorithm for data mining spontaneous reports, currently only available outside the FDA and CDC There is a contracting mechanism in place at the FDA to solve this problem quickly 9PSI-HHS-000002134650 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 117 of 199 —

10PSI-HHS-000002134651 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 118 of 199 —

The graph showing the signals of 3 Sector Maps next to each other is quite interesting • The PRR on the left highlights almost everything • The MGPS on the right is flat (you are not getting useful information with such low counts) • The RGPS in the middle looks more informative for follow-up evaluation • This is because RGPS can better adjust for both, masking (false negatives) and confounding (false positives). 11PSI-HHS-000002134652 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 119 of 199 —

12PSI-HHS-000002134653 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 120 of 199 —

Only to highlight an “Appendicitis, perforated” signal with the Pfizer vaccine that may require follow-up evaluation 13PSI-HHS-000002134654 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 121 of 199 —

14PSI-HHS-000002134655 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 122 of 199 —

The MGPS data mining method currently in use at the Agency and at the CDC is not the state of the art RGPS is the state of the art: • Is a regression-based extension of MGPS that incorporates more information into the signal generation process. This leads to a lower rate of missed signals and less false alerts. • Removes the effect of products whose strong signals in the background are overwhelming the signals of the product of interest and thus uncovers signals of products being masked (the false negatives.) • Adjusts for the concomitant products in the same reports having strong signals to remove the confounding that generates false positives with innocent bystander products. 15PSI-HHS-000002134656 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 123 of 199 —

VAERS doesn’t code concomitant medications, while FAERS does • If we would also adjust for the concomitant meds in the narratives of VAERS reports, we could improve the estimate of confounding (false alerts or false positives) even better. • Bill DuMouchel, who developed both, MGPS and RGPS is planning to extract and code the drugs in the narratives of VAERS reports to reduce confounding. • Note also that REPORTS FOR THE ASTRA ZENECA VACCINE IN USE IN THE U.K. arrive to the FAERS data instead of arriving to the VAERS data. 16PSI-HHS-000002134657 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 124 of 199 —

Simultaneous, automated identification of adverse events and risk factors from multiple products in multiple clinical trials, that we may not know how to specify a priori This is a new analytical approach by Dr. Bill DuMouchel that will require funding 17PSI-HHS-000002134658 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 125 of 199 —

Analysis of adverse event data (as opposed to efficacy) from studies of medical products involve several difficult problems • Pre-specification of analysis end points are rarely possible for adverse events, leading to a multiple comparisons challenge whenever many different adverse events show up. • Rare adverse event issues often show up with small counts that could be grave dangers to public health. • Data from many clinical trials and observational studies may need to be analyzed jointly, such as for many newly developed products. 18PSI-HHS-000002134659 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 126 of 199 —

Risk prediction for diseases for which we do not understand the trajectory of individual patients is low • There is a need for a statistical method capable of identifying rare, unbalanced risk factors that not only occur during or after treatment, but also, more importantly, at baseline. • Consistent efficacy and safety signals may remain hidden in disjoined clinical trial applications or by analyses that do not properly adjust for multiplicity and small counts across all the data being generated. • We need to implement a solution that will allow for display of the adverse event information and results of analyses in an interactive and user- friendly way that will not require a continuous and impossible-to- document customizations of the complex data and the analytical tools. 19PSI-HHS-000002134660 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 127 of 199 —

Advantages of having this automated, intuitive, and interactive analytical program in place • Can perform of analysis of patient level data from many clinical trials and many applications at once. • The addition or removal of some data will generate a quick analysis WITHOUT HAVING TO DO A LOT OF INTERMEDIATE ANALYSES and redo a meta-analysis based on the new summary statistics. • Can provide TRANSPARENCY to the decision-making process and enable RAPID RE-ASSESSMENT OF THE DATA FOR INCREASED COMPREHENSION of new and evolving issues of interest 20PSI-HHS-000002134661 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 128 of 199 —

Comparing different treatments that do not all appear in any one study • This approach will compare subgroups based on multiple treatment arms, covariates, and endpoints measured on each patient. • All this would usually require an analysis of the patient level data in order to fit the Bayesian shrinkage model. • T HESE ACTIVITIES NEED TO BE DONE EXPEDITIOUSLY, IN A TRANSPARENT WAY TO AVOID PUBLIC CONFUSION • WE HAVE NEVER BEEN ABLE TO CONDUCT SUCH ASSESSMENTS, AND EVEN LESS TO CONDUCT SUCH ASSESSMENTS IN AN INTUITIVE AND AUTOMATED MANNER. 21PSI-HHS-000002134662 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 129 of 199 —

Eight Studies of the ICPI Nivolumab versus other Active Comparators -- How to compare the various treatment effects? Study ARM 1 Arm 2 37 NIVOLUMAB INVESTIGATOR CHOICE 17 DOCETAXEL 25 EVEROLIMUS 57 DOCETAXEL 66 DACARBAZINE 26 INVESTIGATOR CHOICE 41 INVESTIGATOR CHOICE 27 CHEMO 22PSI-HHS-000002134663 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 130 of 199 —

Comparison: Pool All 8 Studies Into 1 Analysis For 11 Safety HLTs 23PSI-HHS-000002134664 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 131 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Representing a Pool of Studies as a Network Lines Connecting Two Arms Represent within-Study Comparisons (numbers=study multiplicity) Everolimus Docetaxel a Nivolumab LT Dacarbazine a Chemo Investigator Choice PSI-HHS-000002134665 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 132 of 199 —

Docetaxel Endpoints HLT1, HLT2, HLT3, ... Chemo AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Each Patient May Have Multiple Medically Similar Endpoint Measurements Endpoints HLT1, HLT2, HLT3, ... SO _————— Dacarbazine Nivolumab ZN— Endpoints HLT1, HLT2, HLTS3, ... Investigator Choice Everolimus —_ Endpoints HLT1, HLT2, HLTS, ... Endpoints HLT1, HLT2, HLTS3, ... Endpoints HLT1, HLT2, HLT3, ... PSI-HHS-000002134666 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 133 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Each Patient May Have Multiple Covariates Possibly Influencing Endpoints or Treatment Effects Docetaxel a Covariates Age, Gender, Race, Concom. Meds, ... Chemo ~ Covariates Age, Gender, Ra Concom. Meds, ... ce, Nivolumab a m—~ Covariates Age, Gender, Race, Concom. Meds, ... a Everolimus —_| Covariates Age, Gender, Race, Concom. Meds, ... Covariates Age, Gender, Race, Concom. Meds, ... _————— Dacarbazine NN Investigator Choice Covariates Age, Gender, Race, Concom. Meds, ... PSI-HHS-000002134667 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 134 of 199 —

Thanks to your work, there are now in place multiple approaches for passive and active surveillance for post-authorization safety signals assessments 27PSI-HHS-000002134668 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 135 of 199 —

Some critical roadblocks to consider: • Claims not collecting vaccinations in a systematic way to understand who gets vaccinated and with which vaccine. • No direct link/access to dead registries or EHRs to understand cause of death and underlying risk factors. • Viral variants need to be associated with particular clinical profiles, a task that requires the maintenance of a high level of clinical data veracity. However, • There is a lack of a universal proactive definition and application of permissible variables and values in EHRs and of unique IDs for patients (linked to providers and health facilities) that will simplify access to such data. • There are incomplete and not standardized data creation practices in place within and across systems that unnecessarily delays the analytical processing. • There are no simple ways to follow individual patient progression in EHRs and in other sources of data Potential improvements: • For passive surveillance: • RGPS will provide a big advantage over MGPS for signal detection • For active surveillance: • The simultaneous automated assessment of safety data from multiple clinical trials and OD -- and associated risk factors that we may not be able correctly specify a priori -- will provide a great analytical advantage 28PSI-HHS-000002134669 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 136 of 199 —

TBM: Rewea From Wat REET OR HE FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON _——— TS) US. 10 ¥r-23/32 Yieldwith a1.345%V_ Crude 0i160.95 points with a2.89%& Eurol.2145 points with 20.21%A WSJ RECIPES English Edition | Print Edition | Video | Podcasts | Latest Headlines Home World U.S. Politics Economy Business Tech Markets Opinion Life &Arts RealEstate WSJ. Magazine BUSINESS | HEALTH CARE | HEALTH SHARE sccey_ ~©LBM’s Retreat From Watson Highlights Broader AI Struggles in Health TWIT! une Watson Health was billed as a ‘bet the ranch’ move by Big Blue; now the company is prepared to throw in the towel COPY! EMAIL Watson Health has struggled for market share. PHOTO: COLIN MILLER/AGENCE FRANCE-PRESSEI/GETTY IMAGES Feb. 20, 2021 11:46 am ET SAVE Your browsetslaasanetsupport the audio tag. 8 minutes Ten years ago, International Business Machine Corp.'s IBM +0.66% 4 artificial intelligence systé## Watson bested humans at the quiz show “Jeopardy!” The feat was supposed to herald a shift in the way machines served up answers to questions big and small, opening up new revenue streams for Big Blue specifically and Big Tech more generally. A key target: PSI-HHS-000002136185 Itps:/neww.wsj.comvasticles/ibms-retreat-from-watson-highlights-broader-ai-struggles-in-health-116138395797mod~tech_lead_pos][2/22/2021 11:25:27 AM] AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 137 of 199 —

healthcare, a trillion-dollar industry many say is saddled with inefficiencies that some tech advocates say Al could cure. A decade later, reality has fallen short of that promise. IBM is now exploring the sale of Watson Health, a unit whose marquee product ‘was supposed to help doctors diagnose and cure cancer. IBM spent several billion dollars on acquisitions to build up Watson. Former senior IBM executive John Kelly once touted the initiative as a “bet the ranch” move. It didn’t live up to the hype. Watson Health has struggled for market share in the U.S. and abroad and currently isn’t profitable. Alphabet Inc.'s GOOG Google DeepMind unit, which famously developed a Go-playing algorithm that vanquished a champion human player in 2016, later launched several healthcare-related initiatives focused on chronic conditions. It also has lost money in recent years and run into privacy concerns over how health data was being collected. IBM computer Watson beat Ken Jennings, left, and Brad Rutter to the buzzer to answer a question during a practice round of ‘Jeopardy’ in 2011 PHOTO: SETH WENIG/ASSOCIATED PRESS. The stumbles highlight the challenges of attempting to apply Al to treating complex medical conditions, healthcare experts said. The hurdles include human, financial and technological barriers, they said. Having access to data that represents patient populations broadly has been a challenge, the experts say, as have gaps in knowledge about complex diseases whose outcomes often depend on many factors that may not be fully captured in clinical databases. Tech companies also sometimes lack deep expertise in how healthcare works, adding to the challenge of implementing Al in patient settings, according to Thomas J. Fuchs, Mount Sinai Health System’s dean of artificial intelligence and human health. “You truly have to understand the clinical workflow in the trenches,” he said. “You have to understand where you can insert Al and where it can Most Popular News J], Pope Francis Removes * Conservative African Cardinal From Vatican Post 2, Dominion Sues Mike 3, Travel’s Covid-19 Blues * Are Likely Here to Stay —People Will Go Out of Business’ A. Boeing Backs Grounding * Some 777 Jets 5, Musk Got 4.000 SpaceX * Workers to Join a Covid- 19 Study Most Popular Opinion J], Opinion: We'll Have * Herd Immunity by April 2, Opinion: The Non-Covid * Spending Blowout 3, Opinion: The Money * Boom Is Already Here 4. OPinion: A Stock- * Trading Dupe Is Bom Every Minute 5, Opinion: Why Cuomo * Should Be Worried About a Federal Probe Most Popular Videos J], Dozens of Boeing 777 * Jets Grounded After Engine Failure 2, Remote Workers * Wanted: Small Cities Offer Cash to Those Who Relocate 3, United Flight Scatters * Debris Over Colorado PSI-HHS-000002136186 ttps:/nwww.wsj. comvasticles/ibms-retreat-from-watson-highlights-broader-ai-struggles-in-health-116138395797mod-tech_lead_pos][2/22/2021 11:25:27 AM] TM + Retest From Wat ATP OR TZE PY POR PUBLIC RELEASE BY CHAIRMAN JOHNSON AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 138 of 199 —

TM+ Retest From Wat ATP OR TZE PY POR PUBLIC RELEASE BY CHAIRMAN JOHNSON be helpful” without slowing things down in the clinic. For IBM, the retreat underscores the difficulties new CEO Arvind Krishna faces in restoring growth at the iconic tech company. Mr. Krishna has said Al, along with cloud-computing, would be pivotal for IBM's prospects. Watson Health was one of IBM’s first and the largest Al efforts, said Toni Sacconaghi, an analyst at Bernstein Research. IBM initially promoted it as an engine for growth, but more recently has given it less prominence amid mounting business struggles, leadership changes and layoffs, he said. “Watson may be very emblematic of a broader issue at IBM of taking good science and finding a way to make it commercially relevant,” Mr. Sacconaghi said. STAY INFORMED Get a coronavirus briefing six days a week, and a weekly Health newsletter once the crisis abates: Sign up here. Even as Watson Health ran into problems, the company’s research arm has continued to give priority to Al and healthcare. IBM Research and Pfizer developed speech tests last year to predict the onset of Alzheimer's disease, the company said last year. IBM wouldn't comment about the sale, but said Watson Health has had successes over the years. “This work began nearly 10 years ago, at the beginning of the Al revolution, and we explored groundbreaking space in helping physicians advance healthcare through Al,” the company said. “IBM is continuing to evolve the Watson Health business, based on our decade of experience, to meet the needs of patients and physicians.” A sale would mark Mr. Krishna’s second major move to exit struggling businesses in less than a year at the helm. IBM last year said it planned to spin off its managed IT services division, which generated about $19 billion of annual revenue, or about a quarter of its total sales. By slimming IBM down, Mr. Krishna expects IBM to deliver consistent mid-single-digit growth following a decade filled with revenue declines. IBM had $73.6 billion in sales last year, down from almost $100 billion in 2010. IBM's climb down also serves as a warning to the wider tech industry that sees healthcare as a promising growth market. Watson Health and some other tech-industry Al projects that have struggled were overly ambitious, trying to answer broad, complicated health-related questions, experts said. Watson Health, for instance, was marketed broadly as finding answers to all kinds of cancer, they said. “When the notion is, ‘Well, we can answer any question in cancer care with this data,’ it's too overwhelming. We don’t have the power to do 4. 5,_ Tesla Quality Issues Biden Declares a Major Disaster in Texas, : Triggering Aid ‘Threaten Its Dominance in China EV Market WSJ MEMBER MESSAGE: EVENT Save The Date: The Future Of Everything Festival Join WS,'s iconic FOE Festival, now complimentary to all WSJ members. Reserve your spot and join the conversation. Ri ER PSI-HHS-000002136187 ttps:/nwww.wsj. comvasticles/ibms-retreat-from-watson-highlights-broader-ai-struggles-in-health-116138395797mod-tech_lead_pos][2/22/2021 11:25:27 AM] AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 139 of 199 —

IBM’s Retreat From Watson H ghlights Broader AI Struggles in Health - WSJ https://www.wsj.com/articles/ibms-retreat-from-watson-highlights-broader-ai-struggles-in-health-11613839579?mod=tech_lead_pos1[2/22/2021 11:25:27 AM] that right now,” said David Agus, the chief executive of the Ellison Institute for Transformative Medicine at the University of Southern California and an early tester of the Watson system. Another challenge is the lack of data-collection standards, which makes taking an algorithm that was developed in one setting and applying it in others difficult, experts said. “The customization problem is severe in healthcare,” said Andrew Ng, an AI expert and CEO of startup Landing AI, based in Palo Alto, Calif. The most successful applications of AI in healthcare to date have been when the technology aims to solve discrete and narrow problems, according to Cynthia Burghard, research director at IDC Health Insights, a technology market research and advisory services firm. Such applications include alert systems that warn doctors which of their patients might be at risk for readmissions or severe outcomes and chatbots that help answer basic questions. Recently, some healthcare providers and insurers also have married different data sources, including medical history and income-related information, to come up with risk scores for patients to identify those potentially more vulnerable to Covid-19 exposure to target outreach to them, she said. Such applications are easier to manage because they don’t involve diagnoses. Other areas where AI has seen some successes include radiology and pathology, disciplines where image-recognition software can be applied to answer specific questions, experts said. “It’s about incremental improvements. It’s not about solving the most complex things in healthcare,” she said. “We might get there someday, but [right now] it’s crawl, walk, run.” Another area where the technology has had inroads is in streamlining business processes, like billing and charting, rather than in making diagnoses, experts said, because the stakes are lower, and there is better data to make these systems work. There are also clear financial incentives, they said. “There’s a lot of human capital invested in these things, and a lot of that could be markedly reduced with AI support.” said Eric Topol, a cardiologist and executive vice president at Scripps Research. Despite the challenges of applying AI in healthcare, experts said they expect investments to continue. “The market size is infinite,” said USC’s Dr. Agus. “Healthcare is probably a trillion-dollar market and it’s probably 40% to 60% inefficient. So the notion that you can make it dramatically better with something as elegant as a machine-learning algorithm, or AI, which is scalable, obviously is very enticing.” Write to Daniela Hernandez at daniela.hernandez@wsj.com and Asa Fitch at asa.fitch@wsj.com PSI-HHS-000002136188 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 140 of 199 —

TBM Retreat From Wet ATP OR TEE EY POR PUBLIC RELEASE BY CHAIRMAN JOHNSON Appeared in the February 22, 2021, print edition as ‘IBM Retreat Highlights Hurdles for Health Al.’ ‘Would you like more stories like this? Join the Conversation WSJ opens select articles to reader conversation to promote thoughtful Dominion Sues MyPillow, CEO Mike Lindell dialogue. See the ‘Join the Conversation’ area to the right for stories Over Election Claims Sponsored Offers open to conversation. For more information, please reference our W community guidelines. Email feedback and questions to moderator @wsj.com. TurboTax service code: 50% off for Deluxe version Boeing Backs Grounding Some 777 Jets H&R Block Tax: After Failure During United Flight H&R Block Tax discount - Up to 60% off new tax prep software Target: 20% off entire order - Target promo code ‘Walmart: ‘Walmart promo code: $10 off all categories ‘Samsung: 5% off-Samsung promo code AT&TWireless: AT&T Wireless deal - iPhone 12 Pro Max now $700 off Travel's Covid-19 Blues Are Likely Here to “People Will Go Out of Business’ Garland Makes Case for New Era at Paycheck Protection Program to Offer Exclusive Covid-19 Loan-Application Window for Smallest Firms A Restaurants and Startups Try to Outrun Uber Eats and DoorDash PSI-HHS-000002136189 Itps:/neww.wsj.comvasticles/ibms-retreat-from-watson-highlights-broader-ai-struggles-in-health-116138395797mod~tech_lead_pos][2/22/2021 11:25:27 AM] AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 141 of 199 —

TBM Retreat From Wet ATP OR TEE EY POR PUBLIC RELEASE BY CHAIRMAN JOHNSON The Wall Street Journal _ Sign Out English Edition WSJ Membership ‘Customer Service ‘Tools & Features 'WS]+ Membership Benefits Customer Center Emails & Alerts ‘Subscription Options Contact Us Guides ‘Why Subscribe? Topics Corporate Subscriptions ‘My News Professor Journal RSS Feeds ‘Student Journal ‘Video Center WSJ High School Program Watchlist WSJ Amenity Program Podcasts WSJ Live Facebook Twitter Instagram YouTube Podcasts Snapchat Google Play App Store Historic Gains in Small Stocks Highlight Investor Exuberance a China Deploys Covid-19 Vaccine to Influence, With U.S. on Sidelines 4 \ ‘Ads ‘More “Advertise About Us ‘Commercial Real Estate Ads Commercial Partnerships Place a Classified Ad Content Partnerships Sell Your Business Corrections Sell Your Home Jobs at WST Recruitment & Career Ads News Archive ‘Coupons Register for Free Reprints Buy Issues Dow Jones Products Barron's | BigCharts Dow Jones Newswires | Factiva Financial News | Mansion Global | MarketWatch | Private Markets Risk & Compliance | WSJ Pro WSJ Video | WSJ Wine Privacy NoticeCookie NoticeDo Not Sell My Personal InformationCopyright PolicyData Policy Subscriber Agreement & Terms of Use ‘Your Ad Choices Copyright ©2021 Dow Jones & Company, Inc. All Rights Reserved. PSI-HHS-000002136190 Itps:/neww.wsj.comvasticles/ibms-retreat-from-watson-highlights-broader-ai-struggles-in-health-116138395797mod~tech_lead_pos][2/22/2021 11:25:27 AM] AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 142 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON From: "Forshee, Richard" To: "Scott, John) , "Lee, Shiowjen" Subject: FW: William DuMouchel_advanced meta-analysis (005).pptx Date: Wed, 23 Dec 2020 19:48:33 -0000 Importance: Normal Attachments: WilliamDuMouchel_advanced_meta-analysis_(005).pptx Inline-Images: image001.png Dear John and Shiowjen, Ana Szarfman invited me to a lecture by Bill DuMouchel about his implementation of Network Meta-Analysis methods. Ana thinks it could be very helpful for vaccine safety and effectiveness studies, but | have my doubts. The methods are great for certain problems, but I’m not sure they will address our biggest concerns. I’ve attached his presentation for your review. Let me know if you’d like to explore it further. Thanks and happy holidays, --Rich Sent: Wednesday, December 23, 2020 2:39 PM To: Forshee, Richard Ce: bill.dumouchel Subject: William DuMouchel_advanced meta-analysis (005).pptx Great talking with you Richard. Thanks for your participation. Please refer to the latest version of Bill DuMouchel’s presentation. --Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration ADMINISTRATION U.S. FOOD & DRUG PSI-HHS-000002185638 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 143 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Addressing the need for improving the comprehensiveness and transparency of our decision processes to address the COVID-19 pandemic Introductory words to the presentation by Dr. William DuMouchel by Ana Szarfman, MD, PhD, FAMIA Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research Food and Drug Administration PSI-HHS-000002285825 | AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 144 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON The good news is that we now have new potential interventions for containing the pandemic. These interventions are unprecedented for: *The compressed time for their development and study *The large number of potential new interventions (20 +) *The enormous size of the global population impacted (hundreds of millions of people) The usual review process for new vaccines and molecular entities at the Agency is designed at assessing a much smaller impacted population. PSI-HHS-000002285826 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 145 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Hearing the summary options of scientists that helped control in the past much smaller outbreaks would not be sufficient. * Consistent efficacy and safety signals may remain hidden in disjoined CT applications and observational studies or by analyses that do not properly adjust for multiplicity and small counts across all the data being generated. We need an automated, interactive analytical process in place that can compare the data of all interventions being continuously collected in CTs and observational studies. *The automated reanalysis of the data and identification of consistent signals by specific interventions that can be exhaustively examined by experts will provide transparency to the decision making process. PSI-HHS-000002285827 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 146 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON We have been encouraging Bill DuMouchel to implement a methodology that can assess multiple studies and applications at once (i.e., all vaccines for COVID-19, all anticoagulants) where the goal is to compare different treatments that don’t all appear in any one study, which is the “Network Meta-Analysis” paradigm. As Bill DuMouchel explained, this approach will compare subgroups based on several covariates and in addition where there are multiple endpoints measured on each patient. All this would usually require an analysis of the patient level data in order to fit the Bayesian shrinkage model. PSI-HHS-000002285828 , AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 147 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Finally, an advantage of having one program that can do an analysis of patient level data from many studies and many applications at once, is when we need to add or remove some data, we can get a quick re-analysis without having to do a lot of intermediate analyses and redo a meta-analysis based on the new summary statistics. PSI-HHS-000002285829 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 148 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Bill DuMouchel: Invented the empirical Bayesian data mining algorithm known as Gamma-Poisson Shrinker (GPS) and its successor MGPS, which have been applied to the detection of safety signals in databases of spontaneous adverse drug event reports. He was a senior member of the data mining research group at AT&T (Bell) Labs and Chief perisical Scientist at BBN Software Products, and in the faculty of several institutions, * University of California at Berkeley, * University of Michigan, = MIT, * Columbia University Was an associate editor of * — Journal of the American Statistical Association, * Statistics in Medicine, * — Statistics and Computing, and * — Journal of Computational and Graphical Statistics. PSI-HHS-000002285830 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 149 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON ORACLE Challenges Involving Models for Complicated Full-Data Meta-Analyses Analyses of pooled studies of vaccines, biologics, or drug products from prospective or observational studies and involving multiple treatments, subgroups and endpoints ‘William DuMouchel, PhD Chief Statistician Oracle Health Sciences PSI-HHS-000002285831 _ AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 150 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Many New Vaccines and Proposed Treatments for Covid-19 Are Being Tested Comparisons of Efficacy and Safety Are Complicated + Multiple Efficacy and Multiple Safety Endpoints * Within-Study Comparisons of Treatment Arms * Comparing Subsets Depending on Covariate Values (age, sex, concomitant meds, etc.) * Combining Studies (Including Prospective and Observational Studies) - Studies May Differ by Indication, Populations or Medical History - Bayesian Approaches May Give Higher Weight to Certain Study Designs PSI-HHS-000002285832 ,. AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 151 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Network Meta-Analysis https://training.cochrane.org/resource/key-concepts-network-meta-analysis-nma Several Products Need to Be Compared * Each Study Typically Only Includes Two or a Few Products * But a Chain of Within-Study Comparisons May Connect Every Pair of Products * Pooled Studies Can Create Efficient Indirect Comparisons Among Them All * Allowance Must Be Made for Population Differences Between Studies Both Observational and Prospective Studies May Be In the Pool * Bayesian Approaches May Give Higher Weight to Certain Study Designs PSI-HHS-000002285833 , AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 152 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Eight Studies of the ICPI Nivolumab versus other Active Comparators How to compare the various treatment effects? 37 INVESTIGATOR CHOICE 17 DOCETAXEL 25 EVEROLIMUS 57 DOCETAXEL 66 NIVOLUMAB DACARBAZINE 26 INVESTIGATOR CHOICE 41 INVESTIGATOR CHOICE 27 CHEMO PSI-HHS-000002285834 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 153 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Comparison: Pool All 8 Studies Into 1 Analysis For 11 Safety HLTs Results Method=MBLR PRIOR_NEAN HLT: Acute and chronic thyroiditis HLT: Adrenal conical hypofunctons HLT: Choroid and viteous structural change, depesit and deg... HLT: Coltis (exd infective) HLT: Hepatocellular damage and hepatitis NEC HLT: Hypopigmentation disorders HLT: Hypothalamic and pituitary disorders NEC HLT: HLT: Thyroid hypefundion disorders ‘Odds Ratio Estimates for Term Group="Treatment” 1.0 Confidence Interval for OR Multivariate Bayesian Logistic Regression (MBLR) MBLR (also known as BLR) is similar to meta-analysis, but uses the whole data from each study. Adjusts for potentially biasing between-study differences and multiple comparisons. The results of a BLR run may help support any of the following conclusions: An issue appears related to treatment in a screening analysis, but is not related to treatment when covariates are included in a BLR run, and these covariates show a strong relationship to issue outcome. This may indicate that a randomization error has occurred. PSI-HHS-000002285835 | AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 154 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON a NV 7. Representing a Pool of Studies as a Network Lines Connecting Two Arms Represent within-Study Comparisons (numbers=study multiplicity) Everolimus, Docetaxel ZZ Nivolumab ————_____—_Daarbazine a Ne Chemo Investigator Choice PSI-HHS-000002285836,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 155 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON a iN zs Each Patient May Have Multiple Medically Similar Endpoint Measurements — Endpoints HLT4, HLT2, HLT3, Endpoints HLT1, HLT2, HLT3, Everolimus, "ee Docetaxel | 7 Endpoints HLT1, HLT2, HLT3, Ls Nivolumabo ——_—_—————————————_ Dacarbazine a Ne Endpoints Chemo {Endpoints Investigator [HLT1, HLT2, HLT3, | HLT4, HLT2, HLT3, Choice PSI-HHS-000002285837,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 156 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Each Patient May Have Multiple Covariates Possibly Influencing Endpoints or Treatment Effects — Covariates Covariates Age, Gender, Race, Age, Gender, Race, Concom. Meds, Concom. Meds, Heo)" © Docetaxel Covariates Age, Gender, Race, Concom. Meds, ... Covariates ‘Age, Gender, Race, Concom. Meds, Nivolumabo ——_—_—————————————_ Dacarbazine aa ; Covariates Chemo _ | Covariates Investigator _| Age, Gender, Race, Age, Gender, Race, Choice ~~ Concom. Meds, Concom. Meds, PSI-HHS-000002285838,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 157 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Hierarchical Bayesian Models and Multiple Comparisons Selecting a “Best” Treatment, Covariate-Subgroup or Endpoint Post-hoc ‘while Accounting for Random Variation Standard Approach: Require Small p-Values or Wide Confidence Intervals + When there are very many comparisons, these become too conservative * True Positives Can Remain Hidden * Very Large Datasets Can Generate too many False Positives Hierarchical Bayesian Approach * Fit Models that “Shrink” the Differences among the Estimates Being Compared * Estimates typically Move toward the Average of all stand-alone Estimates * But Error Bars typically become shorter, not longer and Effect strengths are more accurately ranked PSI-HHS-000002285839,. AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 158 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON ag ONY a7 Example of Bayesian “Shrinkage”: Spontaneous Report Disproportionalities Drug-Event Combinations with large ratios of RR = N/E = Observed/Expected counts ~ RR = +---- 99.9% Cl © __Bayes Est. PSI-HHS-000002285840,- AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 159 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON How Bayesian Models Decide How Much to Shrink Shrinking Estimated Differences Provides Multiple Comparisons Protection Requires Estimation of Variance Component or Prior Variability + Compare Data Variation to that Expected by the Null Hypothesis + Excess Data Variance Allows Estimate of Prior Variability + Bayesian Calculations Produce the “Shrinkage” Estimates and Error Bars (Individual Values Move Toward Average of All Values) Next: Two examples of advanced meta-analyses where estimation of excess data variance helped evaluate prediction accuracy PSI-HHS-000002285841,_ AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 160 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Meta-Regression for Extrapolating Across Biological Systems DuMouchel W, HarrisJ (1983) Bayes methods for combining the results of cancer studies in humans and other species, J Amer Stat Assoc 78: 293-315 (w Discussion) ici cila acs Cusee WBiatom ROOF COKE GAS DESEL DIESEL DIESEL DIESEL DIESEL Chemical combinations with data for = tac ear —— Sag ee fitting a dose-response model. | 212 | 712 ] Goal is to get better estimatesof ‘US WER | @ e ;@ | | bd Human Lung Cancer Risk from —- to 4 Diesel Emissions. sxn Twa wt | @ | y = log of dose-response slope f Yar = Wet Oy + y+ Sy + ey, vin TRANsFoRM | @ Y= XB +6 +e,whered=Xp +8 | MUTAGENESIS -MA | @ | o~Tl, | (Bo) ~ Nb. V), inraceea suk (6 B, 0) ~ N(XB, 070), Lo ! (Y|6,B,0)~ N(,C) ‘ PSI-HHS-000002285842 . AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 161 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON a BNO Biological Effects of lonizing Radiation [B.E.I.R. Report IV] ___ Acadamy Press [Annex 7A, by P Groer and W DuMouchel] Making Better Use of Radium Dial Painters’ Data by Combining Studies of Bone Cancer Risk from 4 Isotopes across 4 Biological Systems ISOTOPE BIOLOGICAL SYSTEM Ra-226 Ra-228 Pu-238 Pu-239 Human e e ? ? Beagle Dog (Injection) © ° e Beagle Dog (Inhalation) e ° . Rat Plutonium Bone Cancer Risk Estimate: 300 Cancer Deaths per Million Person-Rad 95% Interval = (80, 1100) 5 to 10 times Larger than Risk from Radon 7. from: Health Risks of Radon and other Internally Deposited Alpha-Emitters, 1988, Nat PSI-HHS-000002285843,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 162 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Complicated Problems Require Combining Several Shrinkage Calculations Combining Multiple Studies Requires Estimating Random Study Effects Multiple Treatment Effects Require a Separate Shrinkage Calculation Covariate by Treatment Interactions Require More Shrinkage Parameters Evaluating Multiple Endpoints: Choose Variables that Are Probably Correlated + Ex: Safety ADRs—Choose Multiple MedDRA Preferred Terms within the Same Higher Level Grouped Term PSI-HHS-000002285844, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 163 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Rationale for Use of Covariates When Studies Are Randomized, Why Adjust for Covariates? Won’t They all Balance Out Anyway? * Depending on sample sizes, will not be perfect balance * If covariates have strong effects, adjustment for them will reduce residual variance and therefore Treatment effect uncertainty * Less focus on a single pre-specified model for safety analyses than for efficacy analyses Main Rationale—Treatment by Covariate Interactions * Estimating Treatment x Covariate interactions in a safety analysis is equivalent to searching for vulnerable subgroups * MBLR~ cross every Covariate with the Treatment effect PSI-HHS-000002285845,| AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 164 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Rationale for EB Model Across End Points Coping with Fine Granularity of Adverse Event Data or Several Efficacy Measures * Compare T vs. C on K potential AEs or Efficacy measures that are similar in meaning * Approach 1—separate analyses of all K measures ~ Small counts lead to non significant comparisons ~ Adjustment for multiple comparisons further reduces sensitivity * Approach 2—define a single measure as the union or mean of the K measures ~ Significant T vs C difference may be washed out by the pooling ~ Even if significant, little information about the original K measures Compromise Approach—EB Hierarchical Model * K individual estimates that “borrow strength” from each other * Estimate separate vector of coefficients for each response measure ~ Buta prior distribution shrinks corresponding coefficients across responses toward each other ~ The amount of shrinkage is controlled by certain prior variances that are also estimated from the data - Treatment-Covariate interaction effects, which are apriori less likely, are also shrunk toward the null hypothesis value ofO PSI-HHS-000002285846,| AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 165 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Bayesian Shrinkage Models Statistical Validity of Searching for Extreme Differences * Most significant adverse event or patient subgroup Classical Approach to Post-Hoc Interval Estimates * Maintain centers of Cl at observed differences * Expand widths of every Cl * Expansion is greater the more differences you look at ~ If you look at too many, the Cl’s are too wide to be useful Bayesian Approach * Requires a prior distribution for differences ~ Can estimate it from the multiple observed differences available * Centers of Cl’s are “shrunk” toward average or null difference ~ High-variance differences shrink the most * Widths of Cl’s usually shrink a little too ~ The more you look at, the better you can model the prior dist. PSI-HHS-000002285847,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 166 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON —Eeg, eee | Safety Study Difficulties Analysis of safety data versus studies of efficacy 1. Prespecification of end points is rarely possible for safety analyses 2. Sample sizes are typically inadequate for many safety issues 3. The optimal granularity of adverse event definitions is often uncertain 4, Subset analyses across subpopulations can rarely be prespecified 5. Pooled analyses of many studies are necessary to compare product safety profiles 6. Combining safety information from clinical trials and observational data may be necessary All of the above issues can be thought of as variations on problems of multiple comparisons Hierarchical Bayesian analysis methods analyze commonali techniques help the estimates “borrow strength’ from each other ies among the diverse effects Shrinkage W DuMouchel, “Multivariate Bayesian Logistic Regression for Analysis of Clinical Study Safety Issues”, Statistical Science, 2012, vol. 27, no. 3, 319-349). The cited pages include three invited discussions of the methodology. * JO] PSI-HHS-000002285848,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 167 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Robustness to Post-Hoc Selection Simulation Study of Bayesian Estimation from Article Cited on last Slide * Draw “true parameters” from the prior distributions 1000 times * Estimate main and interaction effects each time ~ Get both MBLR and Standard “unshrunk” estimates Focus on Estimating the “Most Significant” Interaction (Subset Difference) * 80 Interactions (8 covariates x 10 response events) * For each simulation, select By, that has largest b./Se, * Compare accuracy of estimates and confidence limits Note that Bayesian Shrinkage Eliminates Selection Bias! SIM.COEF SD.SIMC BIAS RMSE Z.SCORE CI.05 CI.95 MBLR 1.7651 0.6094 0.0005 0.2923 -0.0052 0.067 0.056 Stnd 1.7445 0.5981 0.2184 0.4330 0.5794 0.008 0.135 PSI-HHS-000002285849,. AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 168 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Proposal for Enhancement of the Current MBLR Method Use Network Meta-Analysis to Compare Multiple Interventions Across Studies Add Estimation of a Study-Level Variance Component (Shrinkage Parameter) + Allow Study-Level Variables such as Prospective vs Observational * Incorporate Extra Uncertainty for Observational Studies Allow Borrowing Strength Across Multiple Efficacy Endpoints + Example: Multiple Values of Duration Since Vaccination Explore Markov Chain Monte Carlo Computational Approach PSI-HHS-000002285850,. AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 169 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Recommended Aspects of Analysis Methodology Graphical Representation of Comparisons with Confidence Intervals * Covariate selection and interpretation of subset analyses * Comparison of results from different input assumptions—Sensitivity Analyses Collaboration with statisticians from FDA and elsewhere during program development Example pools of studies with their analysis results for training purposes Not a Replacement for Study analyses, but a uniform methodology for comparing estimates across Treatments, Studies, Endpoints and Subsets PSI-HHS-000002285851,_ AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 170 of 199 —

From: "Forshee, Richard" To: "Scott, John" , "Lee, Shiowjen" Subject: RE: William DuMouchel_advanced meta-analysis (005).pptx Date: Mon, 04 Jan 2021 19:24:12 -0000 Importance: Normal Inline-Images: image001.png Thanks, John. I won’t pursue this. Best Regards, --Rich From: Scott, John Sent: Monday, January 4, 2021 12:53 PM To: Forshee, Richard ; Lee, Shiowjen Subject: RE: William DuMouchel_advanced meta-analysis (005).pptx Hi Rich, Thanks for sharing - I really don’t see any applicability at all of this to COVID vaccines, possibly to any vaccines. Best, John From: Forshee, Richard Sent: Wednesday, December 23, 2020 2:49 PM To: Scott, John ; Lee, Shiowjen Subject: FW: William DuMouchel_advanced meta-analysis (005).pptx Dear John and Shiowjen, Ana Szarfman invited me to a lecture by Bill DuMouchel about his implementation of Network Meta-Analysis methods. Ana thinks it could be very helpful for vaccine safety and effectiveness studies, but I have my doubts. The methods are great for certain problems, but I’m not sure they will address our biggest concerns. I’ve attached his presentation for your review. Let me know if you’d like to explore it further. Thanks and happy holidays, --Rich From: Szarfman, Ana Sent: Wednesday, December 23, 2020 2:39 PM To: Forshee, Richard Cc: bill.dumouchel Subject: William DuMouchel_advanced meta-analysis (005).pptx Great talking with you Richard. Thanks for your participation. Please refer to the latest version of Bill DuMouchel’s presentation. PSI-HHS-000002185990 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 171 of 199 —

--Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration PSI-HHS-000002185991 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 172 of 199 —

From: "Szarfman, Ana" To: "Forshee, Richard" Cc: "bill.dumouchel Subject: William DuMouchel_advanced meta-analysis (005).pptx Date: Wed, 23 Dec 2020 19:38:32 +0000 Importance: Normal Attachments: William_DuMouchel_advanced_meta-analysis_(005).pptx Inline-Images: image003.png Great talking with you Richard. Thanks for your participation. Please refer to the latest version of Bill DuMouchel’s presentation. --Ana Ana Szarfman, MD, PhD, FAMIA, Diplomate by the American Board of Pathology in both, Clinical Pathology (1984) and Clinical Informatics (2017), and Fellow of the American Medical Informatics Association (2020) Medical Officer, Safety Data Mining Developer and Medical Informatics Analyst, Celebrating nearly a quarter of a century of successful implementation of safety data mining, interactive patient profiles, and other automated analytical tools. Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research, Food and Drug Administration PSI-HHS-000002194262 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 173 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Addressing the need for improving the comprehensiveness and transparency of our decision processes to address the COVID-19 pandemic Introductory words to the presentation by Dr. William DuMouchel by Ana Szarfman, MD, PhD, FAMIA Division of Cardiology and Nephrology, OCHEN, Center for Drug Evaluation and Research Food and Drug Administration PSI-HHS-000002293250 | AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 174 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON The good news is that we now have new potential interventions for containing the pandemic. These interventions are unprecedented for: *The compressed time for their development and study *The large number of potential new interventions (20 +) *The enormous size of the global population impacted (hundreds of millions of people) The usual review process for new vaccines and molecular entities at the Agency is designed at assessing a much smaller impacted population. PSI-HHS-000002293251 | AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 175 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Hearing the summary options of scientists that helped control in the past much smaller outbreaks would not be sufficient. * Consistent efficacy and safety signals may remain hidden in disjoined CT applications and observational studies or by analyses that do not properly adjust for multiplicity and small counts across all the data being generated. We need an automated, interactive analytical process in place that can compare the data of all interventions being continuously collected in CTs and observational studies. *The automated reanalysis of the data and identification of consistent signals by specific interventions that can be exhaustively examined by experts will provide transparency to the decision making process. PSI-HHS-000002293252 , AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 176 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON We have been encouraging Bill DuMouchel to implement a methodology that can assess multiple studies and applications at once (i.e., all vaccines for COVID-19, all anticoagulants) where the goal is to compare different treatments that don’t all appear in any one study, which is the “Network Meta-Analysis” paradigm. As Bill DuMouchel explained, this approach will compare subgroups based on several covariates and in addition where there are multiple endpoints measured on each patient. All this would usually require an analysis of the patient level data in order to fit the Bayesian shrinkage model. PSI-HHS-000002293253 , AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 177 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Finally, an advantage of having one program that can do an analysis of patient level data from many studies and many applications at once, is when we need to add or remove some data, we can get a quick re-analysis without having to do a lot of intermediate analyses and redo a meta-analysis based on the new summary statistics. PSI-HHS-000002293254 . AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 178 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Bill DuMouchel: Invented the empirical Bayesian data mining algorithm known as Gamma-Poisson Shrinker (GPS) and its successor MGPS, which have been applied to the detection of safety signals in databases of spontaneous adverse drug event reports. He was a senior member of the data mining research group at AT&T (Bell) Labs and Chief perisical Scientist at BBN Software Products, and in the faculty of several institutions, * University of California at Berkeley, * University of Michigan, = MIT, * Columbia University Was an associate editor of * — Journal of the American Statistical Association, * Statistics in Medicine, * — Statistics and Computing, and * — Journal of Computational and Graphical Statistics. PSI-HHS-000002293255 . AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 179 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON ORACLE Challenges Involving Models for Complicated Full-Data Meta-Analyses Analyses of pooled studies of vaccines, biologics, or drug products from prospective or observational studies and involving multiple treatments, subgroups and endpoints ‘William DuMouchel, PhD Chief Statistician Oracle Health Sciences PSI-HHS-000002293256 _ AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 180 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Many New Vaccines and Proposed Treatments for Covid-19 Are Being Tested Comparisons of Efficacy and Safety Are Complicated + Multiple Efficacy and Multiple Safety Endpoints * Within-Study Comparisons of Treatment Arms * Comparing Subsets Depending on Covariate Values (age, sex, concomitant meds, etc.) * Combining Studies (Including Prospective and Observational Studies) - Studies May Differ by Indication, Populations or Medical History - Bayesian Approaches May Give Higher Weight to Certain Study Designs PSI-HHS-000002293257 , AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 181 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Network Meta-Analysis https://training.cochrane.org/resource/key-concepts-network-meta-analysis-nma Several Products Need to Be Compared * Each Study Typically Only Includes Two or a Few Products * But a Chain of Within-Study Comparisons May Connect Every Pair of Products * Pooled Studies Can Create Efficient Indirect Comparisons Among Them All * Allowance Must Be Made for Population Differences Between Studies Both Observational and Prospective Studies May Be In the Pool * Bayesian Approaches May Give Higher Weight to Certain Study Designs PSI-HHS-000002293258 , AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 182 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Eight Studies of the ICPI Nivolumab versus other Active Comparators How to compare the various treatment effects? 37 INVESTIGATOR CHOICE 17 DOCETAXEL 25 EVEROLIMUS 57 DOCETAXEL 66 NIVOLUMAB DACARBAZINE 26 INVESTIGATOR CHOICE 41 INVESTIGATOR CHOICE 27 CHEMO PSI-HHS-000002293259 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 183 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Comparison: Pool All 8 Studies Into 1 Analysis For 11 Safety HLTs Results Method=MBLR PRIOR_NEAN HLT: Acute and chronic thyroiditis HLT: Adrenal conical hypofunctons HLT: Choroid and viteous structural change, depesit and deg... HLT: Coltis (exd infective) HLT: Hepatocellular damage and hepatitis NEC HLT: Hypopigmentation disorders HLT: Hypothalamic and pituitary disorders NEC HLT: HLT: Thyroid hypefundion disorders ‘Odds Ratio Estimates for Term Group="Treatment” 1.0 Confidence Interval for OR Multivariate Bayesian Logistic Regression (MBLR) MBLR (also known as BLR) is similar to meta-analysis, but uses the whole data from each study. Adjusts for potentially biasing between-study differences and multiple comparisons. The results of a BLR run may help support any of the following conclusions: An issue appears related to treatment in a screening analysis, but is not related to treatment when covariates are included in a BLR run, and these covariates show a strong relationship to issue outcome. This may indicate that a randomization error has occurred. PSI-HHS-000002293260 | AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 184 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON a NV 7. Representing a Pool of Studies as a Network Lines Connecting Two Arms Represent within-Study Comparisons (numbers=study multiplicity) Everolimus, Docetaxel ZZ Nivolumab ————_____—_Daarbazine a Ne Chemo Investigator Choice PSI-HHS-000002293261,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 185 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON a iN zs Each Patient May Have Multiple Medically Similar Endpoint Measurements — Endpoints HLT4, HLT2, HLT3, Endpoints HLT1, HLT2, HLT3, Everolimus, "ee Docetaxel | 7 Endpoints HLT1, HLT2, HLT3, Ls Nivolumabo ——_—_—————————————_ Dacarbazine a Ne Endpoints Chemo {Endpoints Investigator [HLT1, HLT2, HLT3, | HLT4, HLT2, HLT3, Choice PSI-HHS-000002293262 AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 186 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Each Patient May Have Multiple Covariates Possibly Influencing Endpoints or Treatment Effects — Covariates Covariates Age, Gender, Race, Age, Gender, Race, Concom. Meds, Concom. Meds, Heo)" © Docetaxel Covariates Age, Gender, Race, Concom. Meds, ... Covariates ‘Age, Gender, Race, Concom. Meds, Nivolumabo ——_—_—————————————_ Dacarbazine aa ; Covariates Chemo _ | Covariates Investigator _| Age, Gender, Race, Age, Gender, Race, Choice ~~ Concom. Meds, Concom. Meds, PSI-HHS-000002293263,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 187 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Hierarchical Bayesian Models and Multiple Comparisons Selecting a “Best” Treatment, Covariate-Subgroup or Endpoint Post-hoc ‘while Accounting for Random Variation Standard Approach: Require Small p-Values or Wide Confidence Intervals + When there are very many comparisons, these become too conservative * True Positives Can Remain Hidden * Very Large Datasets Can Generate too many False Positives Hierarchical Bayesian Approach * Fit Models that “Shrink” the Differences among the Estimates Being Compared * Estimates typically Move toward the Average of all stand-alone Estimates * But Error Bars typically become shorter, not longer and Effect strengths are more accurately ranked PSI-HHS-000002293264, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 188 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON ag ONY a7 Example of Bayesian “Shrinkage”: Spontaneous Report Disproportionalities Drug-Event Combinations with large ratios of RR = N/E = Observed/Expected counts ~ RR = +---- 99.9% Cl © __Bayes Est. PSI-HHS-000002293265,- AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 189 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON How Bayesian Models Decide How Much to Shrink Shrinking Estimated Differences Provides Multiple Comparisons Protection Requires Estimation of Variance Component or Prior Variability + Compare Data Variation to that Expected by the Null Hypothesis + Excess Data Variance Allows Estimate of Prior Variability + Bayesian Calculations Produce the “Shrinkage” Estimates and Error Bars (Individual Values Move Toward Average of All Values) Next: Two examples of advanced meta-analyses where estimation of excess data variance helped evaluate prediction accuracy PSI-HHS-000002293266,_ AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 190 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Meta-Regression for Extrapolating Across Biological Systems DuMouchel W, HarrisJ (1983) Bayes methods for combining the results of cancer studies in humans and other species, J Amer Stat Assoc 78: 293-315 (w Discussion) ici cila acs Cusee WBiatom ROOF COKE GAS DESEL DIESEL DIESEL DIESEL DIESEL Chemical combinations with data for = tac ear —— Sag ee fitting a dose-response model. | 212 | 712 ] Goal is to get better estimatesof ‘US WER | @ e ;@ | | bd Human Lung Cancer Risk from —- to 4 Diesel Emissions. sxn Twa wt | @ | y = log of dose-response slope f Yar = Wet Oy + y+ Sy + ey, vin TRANsFoRM | @ Y= XB +6 +e,whered=Xp +8 | MUTAGENESIS -MA | @ | o~Tl, | (Bo) ~ Nb. V), inraceea suk (6 B, 0) ~ N(XB, 070), Lo ! (Y|6,B,0)~ N(,C) ‘ PSI-HHS-000002293267,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 191 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON a BNO Biological Effects of lonizing Radiation [B.E.I.R. Report IV] ___ Acadamy Press [Annex 7A, by P Groer and W DuMouchel] Making Better Use of Radium Dial Painters’ Data by Combining Studies of Bone Cancer Risk from 4 Isotopes across 4 Biological Systems ISOTOPE BIOLOGICAL SYSTEM Ra-226 Ra-228 Pu-238 Pu-239 Human e e ? ? Beagle Dog (Injection) © ° e Beagle Dog (Inhalation) e ° . Rat Plutonium Bone Cancer Risk Estimate: 300 Cancer Deaths per Million Person-Rad 95% Interval = (80, 1100) 5 to 10 times Larger than Risk from Radon 7. from: Health Risks of Radon and other Internally Deposited Alpha-Emitters, 1988, Nat PSI-HHS-000002293268,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 192 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Complicated Problems Require Combining Several Shrinkage Calculations Combining Multiple Studies Requires Estimating Random Study Effects Multiple Treatment Effects Require a Separate Shrinkage Calculation Covariate by Treatment Interactions Require More Shrinkage Parameters Evaluating Multiple Endpoints: Choose Variables that Are Probably Correlated + Ex: Safety ADRs—Choose Multiple MedDRA Preferred Terms within the Same Higher Level Grouped Term PSI-HHS-000002293269, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 193 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Rationale for Use of Covariates When Studies Are Randomized, Why Adjust for Covariates? Won’t They all Balance Out Anyway? * Depending on sample sizes, will not be perfect balance * If covariates have strong effects, adjustment for them will reduce residual variance and therefore Treatment effect uncertainty * Less focus on a single pre-specified model for safety analyses than for efficacy analyses Main Rationale—Treatment by Covariate Interactions * Estimating Treatment x Covariate interactions in a safety analysis is equivalent to searching for vulnerable subgroups * MBLR~ cross every Covariate with the Treatment effect PSI-HHS-000002293270,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 194 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Rationale for EB Model Across End Points Coping with Fine Granularity of Adverse Event Data or Several Efficacy Measures * Compare T vs. C on K potential AEs or Efficacy measures that are similar in meaning * Approach 1—separate analyses of all K measures ~ Small counts lead to non significant comparisons ~ Adjustment for multiple comparisons further reduces sensitivity * Approach 2—define a single measure as the union or mean of the K measures ~ Significant T vs C difference may be washed out by the pooling ~ Even if significant, little information about the original K measures Compromise Approach—EB Hierarchical Model * K individual estimates that “borrow strength” from each other * Estimate separate vector of coefficients for each response measure ~ Buta prior distribution shrinks corresponding coefficients across responses toward each other ~ The amount of shrinkage is controlled by certain prior variances that are also estimated from the data - Treatment-Covariate interaction effects, which are apriori less likely, are also shrunk toward the null hypothesis value ofO PSI-HHS-000002293271, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 195 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Bayesian Shrinkage Models Statistical Validity of Searching for Extreme Differences * Most significant adverse event or patient subgroup Classical Approach to Post-Hoc Interval Estimates * Maintain centers of Cl at observed differences * Expand widths of every Cl * Expansion is greater the more differences you look at ~ If you look at too many, the Cl’s are too wide to be useful Bayesian Approach * Requires a prior distribution for differences ~ Can estimate it from the multiple observed differences available * Centers of Cl’s are “shrunk” toward average or null difference ~ High-variance differences shrink the most * Widths of Cl’s usually shrink a little too ~ The more you look at, the better you can model the prior dist. PSI-HHS-000002293272, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 196 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON —Eeg, eee | Safety Study Difficulties Analysis of safety data versus studies of efficacy 1. Prespecification of end points is rarely possible for safety analyses 2. Sample sizes are typically inadequate for many safety issues 3. The optimal granularity of adverse event definitions is often uncertain 4, Subset analyses across subpopulations can rarely be prespecified 5. Pooled analyses of many studies are necessary to compare product safety profiles 6. Combining safety information from clinical trials and observational data may be necessary All of the above issues can be thought of as variations on problems of multiple comparisons Hierarchical Bayesian analysis methods analyze commonali techniques help the estimates “borrow strength’ from each other ies among the diverse effects Shrinkage W DuMouchel, “Multivariate Bayesian Logistic Regression for Analysis of Clinical Study Safety Issues”, Statistical Science, 2012, vol. 27, no. 3, 319-349). The cited pages include three invited discussions of the methodology. * JO] PSI-HHS-000002293273,, AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 197 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Robustness to Post-Hoc Selection Simulation Study of Bayesian Estimation from Article Cited on last Slide * Draw “true parameters” from the prior distributions 1000 times * Estimate main and interaction effects each time ~ Get both MBLR and Standard “unshrunk” estimates Focus on Estimating the “Most Significant” Interaction (Subset Difference) * 80 Interactions (8 covariates x 10 response events) * For each simulation, select By, that has largest b./Se, * Compare accuracy of estimates and confidence limits Note that Bayesian Shrinkage Eliminates Selection Bias! SIM.COEF SD.SIMC BIAS RMSE Z.SCORE CI.05 CI.95 MBLR 1.7651 0.6094 0.0005 0.2923 -0.0052 0.067 0.056 Stnd 1.7445 0.5981 0.2184 0.4330 0.5794 0.008 0.135 PSI-HHS-000002293274,. AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 198 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON | —eg eee Proposal for Enhancement of the Current MBLR Method Use Network Meta-Analysis to Compare Multiple Interventions Across Studies Add Estimation of a Study-Level Variance Component (Shrinkage Parameter) + Allow Study-Level Variables such as Prospective vs Observational * Incorporate Extra Uncertainty for Observational Studies Allow Borrowing Strength Across Multiple Efficacy Endpoints + Example: Multiple Values of Duration Since Vaccination Explore Markov Chain Monte Carlo Computational Approach PSI-HHS-000002293275,. AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON

— Page 199 of 199 —

AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON Recommended Aspects of Analysis Methodology Graphical Representation of Comparisons with Confidence Intervals * Covariate selection and interpretation of subset analyses * Comparison of results from different input assumptions—Sensitivity Analyses Collaboration with statisticians from FDA and elsewhere during program development Example pools of studies with their analysis results for training purposes Not a Replacement for Study analyses, but a uniform methodology for comparing estimates across Treatments, Studies, Endpoints and Subsets PSI-HHS-000002293276,_ AUTHORIZED FOR PUBLIC RELEASE BY CHAIRMAN JOHNSON