Jump to content

Systematic reviews

From UBC Wiki
Cao et al (2025) paper on otto-SR, "Automation of Systematic Reviews with Large Language Models"

Compiled by

Updated

See also

Introduction

"...systematic reviews seek to identify as many potentially relevant studies as possible that meet the research question for a given topic." — Centre for Reviews and Dissemination, U of York
"...instead of ... finding papers here and there that support [our] pre-existing beliefs, [we] take a scientific, systematic approach to the very process of looking for scientific evidence, ensuring [our] evidence is as complete and representative as possible of all the research that has ever been done..." — Goldacre, 2012

A systematic review is a scientific investigation in and of itself with pre-planned methods and an assemblage of original studies as its ‘subject’. Results of multiple primary papers are thus synthesized in the SR, and strategies are used to limit bias and random error. These strategies include a comprehensive search of all potentially relevant articles and the use of explicit, reproducible criteria in the selection of studies for review. Primary research designs and study characteristics are appraised, data are synthesized, and results are interpreted. According to the Dictionary of epidemiology (by Last), 4e (2001), a systematic review is "...the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic. Meta-analysis may be, but is not necessarily, used as part of this process..." More specifically, the Health Technology Assessment (HTA) editorial board says that "...a systematic review is a synthesis that collates all empirical evidence fitting pre-specified eligibility criteria in order to answer a specific research question..."

What is a systematic review?

A systematic review is an investigation of a clearly-formulated question that uses methodical and explicit steps to identify, select, and critically appraise relevant research, and to collect and analyze data from any appropriate studies that may be found. The process includes a series of searches for papers or studies in reputable biomedical databases (such as PubMed or Embase for medical questions); a comparison of important features of each study against a list of inclusion and/or exclusion criteria (features include study design, subject or medical condition attributes at baseline, details of the exposure or intervention, time factors, etc.); and a detailed critical appraisal of each study for risk of bias and potential confounding factors. Statistical methods (meta-analysis) may or may not be applied to analyze and summarize the results of these included studies. In published reports, systematic reviews explicitly describe the database(s) and key words that were used in the search; the last date databases were searched; study inclusion/exclusion criteria; process for determining inclusion/exclusion, risk of bias/confounding, and data extraction; a list of apparently-relevant studies that were excluded, and reasons for each exclusion; key details of each included study; and summary of findings.

Systematic reviews have been described as "...papers that summarize other papers" and "overviews of primary studies that have used explicit and reproducible methods". In SRs, the systematic approach to reviewing the literature is more robust and powerful than in traditional literature reviews, which may be may prone to biases of various kinds (e.g., language, publication, geographic). Further, a systematic review strives to exhaustively search all the relevant peer-reviewed literature as well as grey literature and unpublished research findings. The process by which studies are included or excluded is entirely transparent, and is therefore repeatable for future updating.

Cook et al describe SRs as "...scientific investigations in themselves, with pre-planned methods and an assembly of original studies as their “subjects.”" SRs synthesize findings from key, high-powered trials and reports of therapies and interventions using explicit inclusion and exclusion criteria, and may or may not include a meta-analysis. As an approach to gathering, analyzing and synthesizing a body of research, SRs are very popular. SRs include clearly-defined protocols and procedures that ensure accountability and transparency and are typically collaborative in nature. Research teams work in conjunction with a group of professionals, experts and practitioners in the field to ensure that all key resources are located and evaluated. SRs are comprehensive in the way they capture relevant literature yet they often very specific. A set of criteria clearly defines which studies are to be included or excluded in the review - called "inclusion - exclusion" criteria. In the final analysis and synthesis, SRs are evaluated based on methodological rigour and a meta-analysis is conducted when possible.

What is the intent of your systematic review?

There are at least a dozen systematic review types; note that the most common types conducted in medicine address interventions, prognosis or test accuracy.

Is a systematic review appropriate for my (your) research question?

Do you have the team in place to conduct the review?

  • Systematic reviews require a multidisciplinary team.
  • Your team should consist of a topic expert, a systematic review expert, an expert in searching, an experienced statistician and perhaps others (stakeholders, knowledge users, specialist knowledge).

Is there an existing review?

Do you have sufficient time to conduct the review?

  • Systematic reviews require a significant amount of time to complete
  • Most systematic reviews take at least a year, and often two or more.

Developing a comprehensive search strategy

"... current evidence does not support GenAI use in evidence synthesis without human involvement or oversight. However, for most tasks other than searching, GenAI may have a role in assisting humans with evidence synthesis." — Clark et al (2025).
  • Examine previously-conducted systematic reviews in your discipline.
  • Identify the most effective, efficient search strategies you aim to emulate; show a librarian.
  • Consult a librarian experienced in searching bibliographic databases using OVID.
  • Provide the librarian with seed papers related to the topic of interest.
  • Discuss the limitations of any approach, and how you can account for this in your dataset.
  • Consider using one of the artificial intelligence (AI) search tools such as Open Evidence, otto-SR, Perplexity, PubMed.ai, Undermind.ai to find seed papers, but only to inform early stages of your SR until more evidence proves value. Still, more recent research of these tools suggest that they are also helpful in screening and data extraction phases of the SR. Discuss with a librarian.

References

Disclaimer

  • Note: Please use your critical reading skills while reading entries. No warranties, implied or actual, are granted for any health or medical search or AI information obtained while using these pages. Check with your librarian for more contextual, accurate information.