What is meta-analysis?
Meta-analysis refers to a statistical process that compares, combines, and synthesizes research findings from multiple studies in a principled manner. Meta-analyses are largely applied to medicine and pharmacology, two of the fields on which we focus in this description. However, it is important to note that it is increasingly popular and widely used in many other fields, including public health, epidemiology, environmental sciences, and social sciences such as economics and finance. Those who wish to further investigate this topic in depth are referred to this related handbook.
The U.S. Food and Drug Administration (FDA) released a draft guideline in 2018 to evaluate the risks and benefits of human drugs or biological products, and thereby safeguard public health. The guideline defines meta-analysis as “the combining of evidence from relevant studies using appropriate statistical methods to allow inference(s) to be made to the population of interest”. There are two components in this definition that need clarification:
- What does it mean to combine evidence?
- What are the appropriate statistical methods?
Combining evidence implies the datasets are being merged. Depending on how each study reports its findings, the datasets could represent either summary values, such as sample averages and sample variances, or raw data. It is worth taking mental note of this distinction since a meta-analysis can vary significantly based on which category it falls into. Meta-analyses using the summary values will oftentimes include the keywords aggregate data or study-level data. Similarly, those using raw data will likely contain the keywords individual participant data, subject-level data, or, in the medical field, individual patient data.
To address the second question, it is helpful to refer to the title of the FDA guideline—Meta-Analyses of Randomized Controlled Clinical Trials to Evaluate the Safety of Human Drugs or Biological Products Guidance for Industry. Notice that it specifically stipulates randomized controlled clinical trials (RCTs). Statistical analysis consists of several steps: data collection including the study designing, data analysis including the modeling, inference, and validation. The ambiguity in the term ‘statistical methods’ arises from the fact that this breakdown could sometimes be arbitrary. But no matter how it is compartmentalized, the first two stages cannot be omitted. Assuming that the data analysis must be scientifically sound, the expression "appropriate statistical methods" should be interpreted broadly as referring to model development properly reflecting the study design. According to the title of the FDA guideline, the ‘study design’ will always be RCTs. Note that although the meta-analysis for non-randomized trials is an active field of research, the literature is relatively sparse and, more importantly, the FDA does not provide official best practices. Furthermore, without proper considerations for the sources of variation and reporting biases, the results from their meta-analysis can be specious.
Why meta-analysis?
We have yet to answer the most important question:
Why do we need meta-analysis?
The evidence produced by meta-analyses is often placed at the apex of the evidence pyramid. Section 6.B of the FDA guideline explains the “Hierarchy of Evidence for Decision-Making”:
Meta analysis of RCTs with subject-level data will take precedence over others. Between meta-analyses of RCTs and other study designs, the common practice places the meta-analysis before others with the caveat that subject-level data were used. Between subject-level and study-level meta-analyses, the evidence from a study-level meta-analysis generally takes second place to that of its subject-level counterpart as well. However, the FDA guideline does not establish a hierarchy for study-level meta-analyses in relation to other study designs. This does not imply that study-level meta-analyses are without benefits. Study-level meta-analyses still carry practical values, while limited, as an exploratory step before planning a new study. For instance, researchers could conduct a meta-analysis with existing study-level data to ’test the water’ before making investments in new trials. The abundance of such study-level summary results lends itself especially well to such practicality.
More than two treatments
Another important classification of meta-analyses is whether there are more than two treatments. In a trial, an arm refers to a group of participants who are assigned a particular treatment. For example, in the case of Human Immunodeficiency Virus (HIV) treatment, the standard of care is called antiretroviral therapy (ART). There are many drugs used for ART: efavirenz, rilpivirine, indinavir, abacavir, nelfinavir, atazanavir, ritonavir, and dolutegravir. If a study were to include three different treatments, it would, by definition, include three unique arms.
Network meta-analysis (NMA) refers to a type of meta-analysis that compares multiple treatments where each trial is allowed to have a different number of arms. Network meta-analysis delivers unique benefits since it enables the comparison of two treatments which have not been compared vis-à-vis. For example, if one trial compares efavirenz, abacavir, and rilpivirine, and another trial compares rilpivirine and indinavir, then a network meta-analysis with these studies can pick up the difference in treatment effects between efavirenz and indinavir, or abacavir and indinavir. Related organizations could save money and time by reducing the number of trials required to investigate the difference in treatment effects of interest.