is a systematic review a research design

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 17 October 2022.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2022, October 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 21 May 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

Introduction to Systematic Reviews

• Reference work entry
• First Online: 20 July 2022
• pp 2159–2177
• Cite this reference work entry

• Tianjing Li 3 ,
• Ian J. Saldanha 4 &
• Karen A. Robinson 5  

228 Accesses

A systematic review identifies and synthesizes all relevant studies that fit prespecified criteria to answer a research question. Systematic review methods can be used to answer many types of research questions. The type of question most relevant to trialists is the effects of treatments and is thus the focus of this chapter. We discuss the motivation for and importance of performing systematic reviews and their relevance to trialists. We introduce the key steps in completing a systematic review, including framing the question, searching for and selecting studies, collecting data, assessing risk of bias in included studies, conducting a qualitative synthesis and a quantitative synthesis (i.e., meta-analysis), grading the certainty of evidence, and writing the systematic review report. We also describe how to identify systematic reviews and how to assess their methodological rigor. We discuss the challenges and criticisms of systematic reviews, and how technology and innovations, combined with a closer partnership between trialists and systematic reviewers, can help identify effective and safe evidence-based practices more quickly.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

AHRQ (2015) Methods guide for effectiveness and comparative effectiveness reviews. Available from https://effectivehealthcare.ahrq.gov/products/cer-methods-guide/overview . Accessed on 27 Oct 2019

Andersen MZ, Gülen S, Fonnes S, Andresen K, Rosenberg J (2020) Half of Cochrane reviews were published more than two years after the protocol. J Clin Epidemiol 124:85–93. https://doi.org/10.1016/j.jclinepi.2020.05.011

Article   Google Scholar  

Berkman ND, Lohr KN, Ansari MT, Balk EM, Kane R, McDonagh M, Morton SC, Viswanathan M, Bass EB, Butler M, Gartlehner G, Hartling L, McPheeters M, Morgan LC, Reston J, Sista P, Whitlock E, Chang S (2015) Grading the strength of a body of evidence when assessing health care interventions: an EPC update. J Clin Epidemiol 68(11):1312–1324

Borah R, Brown AW, Capers PL, Kaiser KA (2017) Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7(2):e012545. https://doi.org/10.1136/bmjopen-2016-012545

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, Howells DW, Ioannidis JP, Oliver S (2014) How to increase value and reduce waste when research priorities are set. Lancet 383(9912):156–165. https://doi.org/10.1016/S0140-6736(13)62229-1

Clarke M, Chalmers I (1998) Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 280(3):280–282

Cooper NJ, Jones DR, Sutton AJ (2005) The use of systematic reviews when designing studies. Clin Trials 2(3):260–264

Djulbegovic B, Kumar A, Magazin A, Schroen AT, Soares H, Hozo I, Clarke M, Sargent D, Schell MJ (2011) Optimism bias leads to inconclusive results-an empirical study. J Clin Epidemiol 64(6):583–593. https://doi.org/10.1016/j.jclinepi.2010.09.007

Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, Salanti G, Meerpohl J, MacLehose H, Hilton J, Tovey D, Shemilt I, Thomas J (2017) Living systematic review network. Living systematic review: 1. Introduction-the why, what, when, and how. J Clin Epidemiol 91:23–30

Equator Network. Reporting guidelines for systematic reviews. Available from https://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses&eq_guidelines_clinical_specialty=0&eq_guidelines_report_section=0&s=+ . Accessed 9 Mar 2020

Garner P, Hopewell S, Chandler J, MacLehose H, Schünemann HJ, Akl EA, Beyene J, Chang S, Churchill R, Dearness K, Guyatt G, Lefebvre C, Liles B, Marshall R, Martínez García L, Mavergames C, Nasser M, Qaseem A, Sampson M, Soares-Weiser K, Takwoingi Y, Thabane L, Trivella M, Tugwell P, Welsh E, Wilson EC, Schünemann HJ (2016) Panel for updating guidance for systematic reviews (PUGs). When and how to update systematic reviews: consensus and checklist. BMJ 354:i3507. https://doi.org/10.1136/bmj.i3507 . Erratum in: BMJ 2016 Sep 06 354:i4853

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ (2011) GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4):383–394

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds) (2019a) Cochrane handbook for systematic reviews of interventions, 2nd edn. Wiley, Chichester

Google Scholar  

Higgins JPT, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R (2019b) Standards for the conduct of new Cochrane intervention reviews. In: JPT H, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R (eds) Methodological expectations of Cochrane intervention reviews. Cochrane, London

IOM (2011) Committee on standards for systematic reviews of comparative effectiveness research, board on health care services. In: Eden J, Levit L, Berg A, Morton S (eds) Finding what works in health care: standards for systematic reviews. National Academies Press, Washington, DC

Jonnalagadda SR, Goyal P, Huffman MD (2015) Automating data extraction in systematic reviews: a systematic review. Syst Rev 4:78

Krnic Martinic M, Pieper D, Glatt A, Puljak L (2019) Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol 19(1):203. Published 4 Nov 2019. https://doi.org/10.1186/s12874-019-0855-0

Lasserson TJ, Thomas J, Higgins JPT (2019) Chapter 1: Starting a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane. Available from www.training.cochrane.org/handbook

Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC (1992) Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med 327(4):248–254

Lau J (2019) Editorial: systematic review automation thematic series. Syst Rev 8(1):70. Published 11 Mar 2019. https://doi.org/10.1186/s13643-019-0974-z

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6(7):e1000100. https://doi.org/10.1371/journal.pmed.1000100

Lund H, Brunnhuber K, Juhl C, Robinson K, Leenaars M, Dorch BF, Jamtvedt G, Nortvedt MW, Christensen R, Chalmers I (2016) Towards evidence based research. BMJ 355:i5440. https://doi.org/10.1136/bmj.i5440

Marshall IJ, Noel-Storr A, Kuiper J, Thomas J, Wallace BC (2018) Machine learning for identifying randomized controlled trials: an evaluation and practitioner’s guide. Res Synth Methods 9(4):602–614. https://doi.org/10.1002/jrsm.1287

Michelson M, Reuter K (2019) The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun 16:100443. https://doi.org/10.1016/j.conctc.2019.100443 . Erratum in: Contemp Clin Trials Commun 2019 16:100450

Moher D, Liberati A, Tetzlaff J (2009) Altman DG; PRISMA group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151(4):264–269. W64

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, Shekelle P, Stewart LA, PRISMA-P Group (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 4(1):1. https://doi.org/10.1186/2046-4053-4-1

NIHR HTA Stage 1 guidance notes. Available from https://www.nihr.ac.uk/documents/hta-stage-1-guidance-notes/11743 ; Accessed 10 Mar 2020

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, Catalá-López F, Li L, Reid EK, Sarkis-Onofre R, Moher D (2016) Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 13(5):e1002028. https://doi.org/10.1371/journal.pmed.1002028

Page MJ, Higgins JPT, Sterne JAC (2019) Chapter 13: assessing risk of bias due to missing results in a synthesis. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ et al (eds) Cochrane handbook for systematic reviews of interventions, 2nd edn. Wiley, Chichester, pp 349–374

Chapter   Google Scholar  

Robinson KA (2009) Use of prior research in the justification and interpretation of clinical trials. Johns Hopkins University

Robinson KA, Goodman SN (2011) A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med 154(1):50–55. https://doi.org/10.7326/0003-4819-154-1-201101040-00007

Rouse B, Cipriani A, Shi Q, Coleman AL, Dickersin K, Li T (2016) Network meta-analysis for clinical practice guidelines – a case study on first-line medical therapies for primary open-angle glaucoma. Ann Intern Med 164(10):674–682. https://doi.org/10.7326/M15-2367

Saldanha IJ, Lindsley K, Do DV et al (2017) Comparison of clinical trial and systematic review outcomes for the 4 most prevalent eye diseases. JAMA Ophthalmol 135(9):933–940. https://doi.org/10.1001/jamaophthalmol.2017.2583

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:10

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA (2017) AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 358:j4008. https://doi.org/10.1136/bmj.j4008

Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D (2007) How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147(4):224–233

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hróbjartsson A, Kirkham J, Jüni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schünemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JP (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Emberson JR, Hernán MA, Hopewell S, Hróbjartsson A, Junqueira DR, Jüni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

Thomas J, Kneale D, McKenzie JE, Brennan SE, Bhaumik S (2019) Chapter 2: determining the scope of the review and the questions it will address. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane. Available from www.training.cochrane.org/handbook

USPSTF U.S. Preventive Services Task Force Procedure Manual (2017). Available from: https://www.uspreventiveservicestaskforce.org/uspstf/sites/default/files/inline-files/procedure-manual2017_update.pdf . Accessed 21 May 2020

Whitaker (2015) UCSF guides: systematic review: when will i be finished? https://guides.ucsf.edu/c.php?g=375744&p=3041343 , Accessed 13 May 2020

Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J (2016) Churchill R; ROBIS group. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69:225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005

Download references

Author information

Authors and affiliations.

Department of Ophthalmology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA

Tianjing Li

Department of Health Services, Policy, and Practice and Department of Epidemiology, Brown University School of Public Health, Providence, RI, USA

Ian J. Saldanha

Department of Medicine, Johns Hopkins University, Baltimore, MD, USA

Karen A. Robinson

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Tianjing Li .

Editor information

Editors and affiliations.

Department of Surgery, Division of Surgical Oncology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

Steven Piantadosi

Department of Epidemiology, School of Public Health, Johns Hopkins University, Baltimore, MD, USA

Curtis L. Meinert

Section Editor information

Department of Epidemiology, University of Colorado Denver Anschutz Medical Campus, Aurora, CO, USA

The Johns Hopkins Center for Clinical Trials and Evidence Synthesis, Johns Hopkins University, Baltimore, MD, USA

Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Cite this entry.

Li, T., Saldanha, I.J., Robinson, K.A. (2022). Introduction to Systematic Reviews. In: Piantadosi, S., Meinert, C.L. (eds) Principles and Practice of Clinical Trials. Springer, Cham. https://doi.org/10.1007/978-3-319-52636-2_194

Download citation

DOI : https://doi.org/10.1007/978-3-319-52636-2_194

Published : 20 July 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-319-52635-5

Online ISBN : 978-3-319-52636-2

eBook Packages : Mathematics and Statistics Reference Module Computer Science and Engineering

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Study Design 101: Systematic Review

• Case Report
• Case Control Study
• Cohort Study
• Randomized Controlled Trial
• Practice Guideline
• Systematic Review
• Meta-Analysis
• Helpful Formulas
• Finding Specific Study Types

A document often written by a panel that provides a comprehensive review of all relevant studies on a particular clinical or health-related topic/question. The systematic review is created after reviewing and combining all the information from both published and unpublished studies (focusing on clinical trials of similar treatments) and then summarizing the findings.

• Exhaustive review of the current literature and other sources (unpublished studies, ongoing research)
• Less costly to review prior studies than to create a new study
• Less time required than conducting a new study
• Results can be generalized and extrapolated into the general population more broadly than individual studies
• More reliable and accurate than individual studies
• Considered an evidence-based resource

Disadvantages

• Very time-consuming
• May not be easy to combine studies

Design pitfalls to look out for

Studies included in systematic reviews may be of varying study designs, but should collectively be studying the same outcome.

Is each study included in the review studying the same variables?

Some reviews may group and analyze studies by variables such as age and gender; factors that were not allocated to participants.

Do the analyses in the systematic review fit the variables being studied in the original studies?

Fictitious Example

Does the regular wearing of ultraviolet-blocking sunscreen prevent melanoma? An exhaustive literature search was conducted, resulting in 54 studies on sunscreen and melanoma. Each study was then evaluated to determine whether the study focused specifically on ultraviolet-blocking sunscreen and melanoma prevention; 30 of the 54 studies were retained. The thirty studies were reviewed and showed a strong positive relationship between daily wearing of sunscreen and a reduced diagnosis of melanoma.

Real-life Examples

Yang, J., Chen, J., Yang, M., Yu, S., Ying, L., Liu, G., ... Liang, F. (2018). Acupuncture for hypertension. The Cochrane Database of Systematic Reviews, 11 (11), CD008821. https://doi.org/10.1002/14651858.CD008821.pub2

This systematic review analyzed twenty-two randomized controlled trials to determine whether acupuncture is a safe and effective way to lower blood pressure in adults with primary hypertension. Due to the low quality of evidence in these studies and lack of blinding, it is not possible to link any short-term decrease in blood pressure to the use of acupuncture. Additional research is needed to determine if there is an effect due to acupuncture that lasts at least seven days.

Parker, H.W. and Vadiveloo, M.K. (2019). Diet quality of vegetarian diets compared with nonvegetarian diets: a systematic review. Nutrition Reviews , https://doi.org/10.1093/nutrit/nuy067

This systematic review was interested in comparing the diet quality of vegetarian and non-vegetarian diets. Twelve studies were included. Vegetarians more closely met recommendations for total fruit, whole grains, seafood and plant protein, and sodium intake. In nine of the twelve studies, vegetarians had higher overall diet quality compared to non-vegetarians. These findings may explain better health outcomes in vegetarians, but additional research is needed to remove any possible confounding variables.

Related Terms

Cochrane Database of Systematic Reviews

A highly-regarded database of systematic reviews prepared by The Cochrane Collaboration , an international group of individuals and institutions who review and analyze the published literature.

Exclusion Criteria

The set of conditions that characterize some individuals which result in being excluded in the study (i.e. other health conditions, taking specific medications, etc.). Since systematic reviews seek to include all relevant studies, exclusion criteria are not generally utilized in this situation.

Inclusion Criteria

The set of conditions that studies must meet to be included in the review (or for individual studies - the set of conditions that participants must meet to be included in the study; often comprises age, gender, disease type and status, etc.).

Now test yourself!

1. Systematic Reviews are similar to Meta-Analyses, except they do not include a statistical analysis quantitatively combining all the studies.

a) True b) False

2. The panels writing Systematic Reviews may include which of the following publication types in their review?

a) Published studies b) Unpublished studies c) Cohort studies d) Randomized Controlled Trials e) All of the above

Evidence Pyramid - Navigation

• Meta- Analysis
• Case Reports
• << Previous: Practice Guideline
• Next: Meta-Analysis >>

• Last Updated: Sep 25, 2023 10:59 AM
• URL: https://guides.himmelfarb.gwu.edu/studydesign101

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

• Open access
• Published: 09 June 2012

Clarifying differences between review designs and methods

• David Gough 1 ,
• James Thomas 1 &
• Sandy Oliver 1  

Systematic Reviews volume  1 , Article number:  28 ( 2012 ) Cite this article

102k Accesses

397 Citations

75 Altmetric

Metrics details

This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews. Terminology is necessary for planning, describing, appraising, and using reviews, building infrastructure to enable the conduct and use of reviews, and for further developing review methodology. There is insufficient consensus on terminology for a typology of reviews to be produced and any such attempt is likely to be limited by the overlapping nature of the dimensions along which reviews vary. It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation. Three such main dimensions are proposed: (1) aims and approaches (including what the review is aiming to achieve, the theoretical and ideological assumptions, and the use of theory and logics of aggregation and configuration in synthesis); (2) structure and components (including the number and type of mapping and synthesis components and how they relate); and (3) breadth and depth and the extent of ‘work done’ in addressing a research issue (including the breadth of review questions, the detail with which they are addressed, and the amount the review progresses a research agenda). This then provides an overarching strategy to encompass more detailed descriptions of methodology and may lead in time to a more overarching system of terminology for systematic reviews.

Peer Review reports

Research studies vary in many ways including the types of research questions they are asking, the reasons these questions are being asked, the theoretical and ideological perspectives underlying these questions, and in the research methods that they employ. Systematic reviews are a form of research; they are (and the theoretical and ideological perspectives underlying these methods) a way of bringing together what is known from the research literature using explicit and accountable methods [ 1 ]. Systematic methods of review have been successfully developed particularly for questions concerning the impact of interventions; these synthesize the findings of studies which use experimental controlled designs. Yet the logic of systematic methods for reviewing the literature can be applied to all areas of research; therefore there can be as much variation in systematic reviews as is found in primary research [ 2 , 3 ]. This paper discusses some of the important conceptual and practical differences between different types of systematic review. It does not aim to provide an overall taxonomy of all types of reviews; the rate of development of new approaches to reviewing is too fast and the overlap of approaches too great for that to be helpful. Instead, the paper argues that, for the present at least, it is more useful to identify the key dimensions on which reviews differ and to examine the multitude of different combinations of those dimensions. The paper also does not aim to describe all of the myriad actual and potential differences between reviews; this would be a task too large even for a book let alone a paper. The focus instead is on three major types of dimensions of difference. The first dimension is the aims and approaches of reviews; particularly in terms of their methodologies (their ontological and epistemological foundations and methods of synthesis). The second dimension is the structure and components of reviews. The third dimension is the breadth, depth, and extent of the work done by a review in engaging with a research issue. Once these three aspects of a review are clear, consideration can be given to more specific methodological issues such as methods of searching, identifying, coding, appraising, and synthesizing evidence. The aim of this paper is to clarify some of the major conceptual distinctions between reviews to assist the selection, evaluation, and development of methods for reviewing.

Clarifying the nature of variation in reviews

As forms of research, systematic reviews are undertaken according to explicit methods. The term ‘systematic’ distinguishes them from reviews undertaken without clear and accountable methods.

The history of systematic reviews is relatively recent [ 4 , 5 ] and despite early work on meta-ethnography [ 6 ], the field has been dominated by the development and application of statistical meta-analysis of controlled trials to synthesize the evidence on the effectiveness of health and social interventions. Over the past 10 years, other methods for reviewing have been developed. Some of these methods aim to extend effectiveness reviews with information from qualitative studies [ 7 ]. The qualitative information may be used to inform decisions made in the statistical synthesis or be part of a mixed methods synthesis (discussed later). Other approaches have been developed from a perspective which, instead of the statistical aggregation of data from controlled trials, emphasize the central role that theory can play in synthesizing existing research [ 8 , 9 ], address the complexity of interventions [ 10 ], and the importance of understanding research within its social and paradigmatic context [ 11 ]. The growth in methods has not been accompanied by a clear typology of reviews. The result is a complex web of terminology [ 2 , 12 ].

The lack of clarity about the range of methods of review has consequences which can limit their development and subsequent use. Knowledge or consensus about the details of specific methods may be lacking, creating the danger of the over-generalization or inappropriate application of the terminology being used. Also, the branding of different types of review can lead to over-generalizations and simplification with assumptions being made about differences between reviews that only apply to particular stages of a review or that are matters of degree rather than absolute differences. For example, concepts of quality assurance can differ depending upon the nature of the research question being asked. Similarly, infrastructure systems developed to enable the better reporting and critical appraisal of reviews, such as PRISMA [ 13 ], and for registration of reviews, such as PROSPERO [ 14 ] currently apply predominantly to a subset of reviews, the defining criteria of which may not be fully clear.

A further problem is that systematic reviews have attracted criticism on the assumption that systematic reviewing is applicable only to empirical quantitative research [ 15 ]. In this way, polarized debates about the utility and relevance of different research paradigms may further complicate terminological issues and conceptual understandings about how reviews actually differ from one another. All of these difficulties are heightened because review methods are undergoing a period of rapid development and so the methods being described are often being updated and refined.

Knowledge about the nature and strengths of different forms of review is necessary for: appropriate choice of review methods by those undertaking reviews; consideration of the importance of different issues of quality and relevance for each stage of a review; appropriate and accurate reporting and accountability of such review methods; interpretation of reviews; commissioning of reviews; development of procedures for assessing and undertaking reviews; and development of new methods.

Clarifying the nature of the similarities and differences between reviews is a first step to avoiding these potential limitations. A typology of review methods might be a solution. There are many diverse approaches to reviews that can be easily distinguished, such as statistical meta-analysis and meta-ethnography. A more detailed examination, however, reveals that the types of review currently described often have commonalities that vary across types of review and at different stages of a review. Three of these dimensions are described here. Exploring these dimensions also reveals how reviews differ in degree along these overlapping dimensions rather than falling into clear categories.

Review aims and approaches

Primary research and research reviews vary in their ontological, epistemological, ideological, and theoretical stance, their research paradigm, and the issues that they aim to address. In reviews, this variation occurs in both the method of review and the type of primary research that they consider. As reviews will include primary studies that address the focus of the review question, it is not surprising that review methods also tend to reflect many of the approaches, assumptions, and methodological challenges of the primary research that they include.

One indication of the aim and approach of a study is the research question which the study aims to answer. Questions commonly addressed by systematic reviews include: what is the effect of this intervention (addressed by, for example, the statistical meta-analysis of experimental trials); what is the accuracy of this diagnostic tool (addressed by, for example, meta-analysis of evaluations of diagnostic tests); what is the cost of this intervention (addressed by, for example, a synthesis of cost-benefit analyses); what is the meaning or process of a phenomena (addressed by, for example, conceptual synthesis such as meta-ethnography or a critical interpretative synthesis of ethnographic studies); what is the effect of this complex intervention (addressed by, for example, multi-component mixed methods reviews); what is the effect of this approach to social policy in this context (addressed by, for example, realist synthesis of evidence of efficacy and relevance across different policy areas); and what are the attributes of this intervention or activity (addressed by, for example, framework synthesis framed by dimensions explicitly linked to particular perspectives).

Although different questions drive the review process and suggest different methods for reviewing (and methods of studies included) there is considerable overlap in the review methods that people may select to answer these questions; thus the review question alone does not provide a complete basis for generating a typology of review methods.

Role of theory

There is no agreed typology of research questions in the health and social sciences. In the absence of such a typology, one way to distinguish research is in the extent that it is concerned with generating, exploring, or testing theory [ 16 ].

In addressing an impact question using statistical meta-analysis, the approach is predominantly the empirical testing of a theory that the intervention works. The theory being tested may be based on a detailed theory of change (logic model) or be a ‘black box’ where the mechanisms by which change may be affected are not articulated. The review may, in addition to testing theory, include methods to generate hypotheses about causal relations. Testing often (though not always) wants to add up or aggregate data from large representative samples to obtain a more precise estimate of effect. In the context of such reviews, searching aims to identify a representative sample of studies, usually by attempting to include all relevant studies in order to avoid bias from study selection (sometimes called ‘exhaustive’ searching). Theoretical work in such analyses is undertaken predominantly before and after the review, not during the review, and is concerned with developing the hypothesis and interpreting the findings.

In research examining processes or meanings the approach is predominantly about developing or exploring theory. This may not require representative samples of studies (as in aggregative reviews) but does require variation to enable new conceptual understandings to be generated. Searching for studies in these reviews adopts a theoretical approach to searching to identify a sufficient and appropriate range of studies either through a rolling sampling of studies according to a framework that is developed inductively from the emerging literature (akin to theoretical sampling in primary research) [ 17 ]; or through a sampling framework based on an existing body of literature (akin to purposive sampling in primary research) [ 18 ]. In both primary research and reviews, theoretical work is undertaken during the process of the research; and, just as with the theory testing reviews, the nature of the concepts may be relatively simple or very complex.

Aggregative and configurative reviews

The distinction between research that tests and research that generates theory also equates to the distinction between review types made by Voils, Sandelowski and colleagues [ 19 , 20 ] (although we have been very influenced by these authors the detail of our use of these terms may differ in places). Reviews that are collecting empirical data to describe and test predefined concepts can be thought of as using an ‘aggregative’ logic. The primary research and reviews are adding up (aggregating) and averaging empirical observations to make empirical statements (within predefined conceptual positions). In contrast, reviews that are trying to interpret and understand the world are interpreting and arranging (configuring) information and are developing concepts (Figure 1 ). This heuristic also maps onto the way that the review is intended to inform knowledge. Aggregative research tends to be about seeking evidence to inform decisions whilst configuring research is seeking concepts to provide enlightenment through new ways of understanding.

Continua of approaches in aggregative and configurative reviews.

Aggregative reviews are often concerned with using predefined concepts and then testing these using predefined ( a priori ) methods. Configuring reviews can be more exploratory and, although the basic methodology is determined (or at least assumed) in advance, specific methods are sometimes adapted and selected (iteratively) as the research proceeds. Aggregative reviews are likely to be combining similar forms of data and so be interested in the homogeneity of studies. Configurative reviews are more likely to be interested in identifying patterns provided by heterogeneity [ 12 ].

The logic of aggregation relies on identifying studies that support one another and so give the reviewer greater certainty about the magnitude and variance of the phenomenon under investigation. As already discussed in the previous section, the approach to searching for studies to include (the search strategy) is attempting to be exhaustive or, if not exhaustive, then at least avoiding bias in the way that studies are found. Configuring reviews have the different purpose of aiming to find sufficient cases to explore patterns and so are not necessarily attempting to be exhaustive in their searching. (Most reviews contain elements of both aggregation and configuration and so some may require an unbiased set of studies as well as sufficient heterogeneity to permit the exploration of differences between them).

Aggregating and configuring reviews also vary in their approach to quality assurance. All reviews aim to avoid drawing misleading conclusions because of problems in the studies they contain. Aggregative reviews are concerned with a priori methods and their quality assurance processes assess compliance with those methods. As the basis of quality assurance is known a priori , many aspects of this can be incorporated into the inclusion criteria of the review and then can be further checked at a later quality assurance stage. The inclusion criteria may, for example, require only certain types of study with specific methodological features. There is less consensus in the practice of quality assessment in configurative reviews; some adopt a similar strategy to those employed in aggregative reviews, whereas others reject the idea that the quality of a study can be assessed through an examination of its method, and instead prioritize other issues, such as relevance to the review and the contribution the study can make in the review synthesis to testing or generating theory [ 21 – 23 ]. Some of the differences between aggregating and configuring reviews are shown in Figure 1 .

Although the logics of aggregating and configuring research findings demand different methods for reviewing, a review often includes components of both. A meta-analysis may contain a post hoc interpretation of statistical associations which may be configured to generate hypotheses for future testing. A configurative synthesis may include some components where data are aggregated (for example, framework synthesis) [ 24 , 25 ]. Examples of reviews that are predominantly aggregative, configurative, or with high degrees of both aggregation and configuring are given in Table 1 (and for a slightly different take on this heuristic see Sandelowski et al. [ 20 ]).

Similarly, the nature of a review question, the assumptions underlying the question (or conceptual framework), and whether the review aggregates or configures the results of other studies may strongly suggest which methods of review are appropriate, but this is not always the case. Several methods of review are applicable to a wide range of review approaches. Both thematic [ 26 ] and framework synthesis [ 24 , 25 ] which identify themes within narrative data can, for example, be used with both aggregative and configurative approaches to synthesis.

Reviews that are predominantly aggregative may have similar epistemological and methodological assumptions to much quantitative research and there may be similar assumptions between predominantly configurative reviews and qualitative research. However, the quantitative/qualitative distinction is not precise and does not reflect the differences in the aggregative and configurative research processes; quantitative reviews may use configurative processes and qualitative reviews can use aggregative processes. Some authors also use the terms conceptual synthesis for reviews that are predominantly configurative, but the process of configuring in a review does not have to be limited to concepts; it can also be the arrangement of numbers (as in subgroup analyses of statistical meta-analysis). The term ‘interpretative synthesis’ is also used to describe reviews where meanings are interpreted from the included studies. However, aggregative reviews also include interpretation, before inspection of the studies to develop criteria for including studies, and after synthesis of the findings to develop implications for policy, practice, and further research. Thus, the aggregate/configure framework cannot be thought of as another way of expressing the qualitative/quantitative ‘divide’; it has a more specific meaning concerning the logic of synthesis, and many reviews have elements of both aggregation and configuration.

Further ideological and theoretical assumptions

In addition to the above is a range of issues about whose questions are being asked and the implicit ideological and theoretical assumptions driving both them and the review itself. These assumptions determine the specific choices made in operationalizing the review question and thus determine the manner in which the review is undertaken, including the research studies included and how they are analyzed. Ensuring that these assumptions are transparent is therefore important both for the execution of the review and for accountability. Reviews may be undertaken to inform decision-making by non-academic users of research such as policymakers, practitioners, and other members of the public and so there may be a wide range of different perspectives that can inform a review [ 27 , 28 ]. The perspectives driving the review will also influence the findings of the review and thereby clarify what is known and not known (within those perspectives) and thus inform what further primary research is required. Both reviewer and user perspectives can thus have an ongoing influence in developing user-led research agendas. There may be many different agendas and thus a plurality of both primary research and reviews of research on any given issue.

A further fundamental issue that is related to the types of questions being asked and the ideological and theoretical assumptions underlying them is the ontological and epistemological position taken by the reviewers. Aggregative reviews tend to assume that there is (often within disciplinary specifications/boundaries) a reality about which empirical statements can be made even if this reality is socially constructed (generalizations); in other words they take a ‘realist’ philosophical position (a broader concept than the specific method of ‘realist synthesis’). Some configurative reviews may not require such realist assumptions. They take a more relativist idealist position; the interest is not in seeking a single ‘correct’ answer but in examining the variation and complexity of different conceptualizations [ 12 , 29 ]. These philosophical differences can be important in understanding the approach taken by different reviewers just as they are in understanding variation in approach (and debates about research methods) in primary research. These differences also relate to how reviews are used. Aggregative reviews are often used to make empirical statements (within agreed conceptual perspectives) to inform decision making instrumentally whilst configuring reviews are often used to develop concepts and enlightenment [ 30 ].

Structure and components of reviews

As well as varying in their questions, aims, and philosophical approach, reviews also vary in their structure. They can be single reviews that synthesize a specific literature to answer the review question. They may be maps of what research has been undertaken that are products in their own right and also a stage on the way to one or more syntheses. Reviews can also contain multiple components equating to conducting many reviews or to reviewing many reviews.

Systematic maps

To some degree, most reviews describe the studies they contain and thus provide a map or account of the research field. Some reviews go further than this and more explicitly identify aspects of the studies that help describe the research field in some detail; the focus and extent of such description varying with the aims of the map. Maps are useful products in their own right but can also be used to inform the process of synthesis and the interpretation of the synthesis [ 3 , 30 ]. Instead of automatically undertaking a synthesis of all included studies, an analysis of the map may lead to a decision to synthesize only a subset of studies, or to conduct several syntheses in different areas of the one map. A broader initial review question and a narrower subsequent review question allows the synthesis of a narrower subset of studies to be understood within the wider literature described in terms of research topics, primary research methods, or both. It also allows broader review questions to create a map for a series of reviews (Figure 2 ) or mixed methods reviews (Figure 3 ). In sum, maps have three main purposes of: (i) describing the nature of a research field; (ii) to inform the conduct of a synthesis; and (iii) to interpret the findings of a synthesis [ 3 , 31 ].The term ‘scoping review’ is also sometimes used in a number of different ways to describe (often non-systematic) maps and/or syntheses that rapidly examine the nature of the literature on a topic area [ 32 , 33 ]; sometimes as part of the planning for a systematic review.

A map leading to several syntheses.

A mixed method review with three syntheses.

• Mixed methods reviews

The inclusion criteria of a review may allow all types of primary research or only studies with specific methods that are considered most appropriate to best address the review question. Including several different methods of primary research in a review can create challenges in the synthesis stage. For example, a review asking about the impact of some life experience may examine both randomized controlled trials and large data sets on naturally occurring phenomena (such as in large scale cohort studies). Another strategy is to have sub-reviews that ask questions about different aspects of an issue and which are likely to consider different primary research [ 34 , 35 ]. For example, a statistical meta-analysis of impact studies compared with a conceptual synthesis of people’s views of the issue being evaluated [ 34 , 35 ]. The two sub-reviews can then be combined and contrasted in a third synthesis as in Figure 3 . Mixed methods reviews have many similarities with mixed methods in primary research and there are therefore numerous ways in which the products of different synthesis methods may be combined [ 35 ].

Mixed knowledge reviews use a similar approach but combine data from previous research with other forms of data; for example a survey of practice knowledge about an issue (Figure 4 ).

Mixed knowledge review.

Another example of a mixed methods review is realist synthesis [ 9 ] that examines the usefulness of mid-level policy interventions across different areas of social policy by unpacking the implicit models of change, followed by an iterative process of identifying and analyzing the evidence in support of each part of that model. This is quite similar to a theory-driven aggregative review (or series of reviews) that aggregatively test different parts of a causal model. The first part of the process is a form of configuration in clarifying the nature of the theory and what needs to be empirically tested; the second part is the aggregative testing of those subcomponents of the theory. The difference between this method and more ‘standard’ systematic review methods is that the search for empirical evidence is more of an iterative, investigative process of tracking down and interpreting evidence. Realist synthesis will also consider a broad range of empirical evidence and will assess its value in terms of its contribution rather than according to some preset criteria. The approach therefore differs from the predominantly a priori strategy used in either standard ‘black box’ or in theory driven aggregative reviews. There have also been attempts to combine aggregative ‘what works’ reviews with realist reviews [ 36 ]. These innovations are exploring how best to develop the breadth, generalizability and policy relevance of aggregative reviews without losing their methodological protection against bias.

There are also reviews that use other pre-existing reviews as their source of data. These reviews of reviews may draw on the data of previous reviews either by using the findings of previous reviews or by drilling down to using data from the primary studies in the reviews [ 37 ]. Information drawn from many reviews can also be mined to understand more about a research field or research methods in meta-epidemiology [ 38 ]. As reviews of reviews and meta-epidemiology both use reviews as their data, they are sometimes both described as types of ‘meta reviews’. This terminology may not be helpful as it links together two approaches to reviews which have little in common apart from the shared type of data source. A further term is ‘meta evaluation’. This can refer to the formative or summative evaluation of primary evaluation studies or can be a summative statement of the findings of evaluations which is a form of aggregative review (See Gough et al. in preparation, and [ 39 ]).

Breadth, depth, and ’work done’ by reviews

Primary research studies and reviews may be read as isolated products yet they are usually one step in larger or longer-term research enterprises. A research study usually addresses a macro research issue and a specific focused sub-issue that is addressed by its specific data and analysis [ 16 ]. This specific focus can be broad or narrow in scope and deep or not so deep in the detail in which it is examined.

Breadth of question

Many single component aggregative reviews aim for homogeneity in the focus and method of included studies. They select narrowly defined review questions to ensure a narrow methodological focus of research findings. Although well justified, these decisions may lead to each review providing a very narrow view of both research and the issue that is being addressed. A user of such reviews may need to take account of multiple narrow reviews in order to help them determine the most appropriate course of action.

The need for a broader view is raised by complex questions. One example is assessing the impact of complex interventions. There are often many variants of an intervention, but even within one particular highly specified intervention there may be variations in terms of the frequency, duration, degree, engagement, and fidelity of delivery [ 40 ]. All of this variation may result in different effects on different participants in different contexts. The variation may also impact differentially within the hypothesized program theory of how the intervention impacts on different causal pathways. Reviews therefore need a strategy for how they can engage with this complexity. One strategy is to achieve breadth through multi-component reviews; for example, a broad map which can provide the context for interpreting a narrower synthesis, a series of related reviews, or mixed methods reviews. Other strategies include ‘mega reviews’, where the results from very many primary studies or meta-analyses are aggregated statistically (for example, [ 41 , 42 ]) and multivariate analyses, where moderator variables are used to identify the ‘active ingredients’ of interventions (for example, [ 43 , 44 ]). Whether breadth is achieved within a single review, from a sequence of reviews, from reviews of reviews, or from relating to the primary and review work of others, the cycle of primary research production and synthesis is part of a wider circle of engagement and response to users of research [ 45 ].

Review resources and breadth and depth of review

The resources required for a systematic review are not fixed. With different amounts of resource one can achieve different types of review. Broad reviews such as mixed methods and other multi-component reviews are likely to require more resources, all else being constant, than narrow single method reviews. Thus, in addition to the breadth of review is the issue of its depth, or the detail with which it is undertaken. A broad review may not have greater resources than a narrow review in which case those resources are spread more thinly and each aspect of that breadth may be undertaken with less depth.

When time and other resources are very restricted then a rapid review may be undertaken where some aspect of the review will be limited; for example, breadth of review question, sources searched, data coded, quality and relevance assurance measures, and depth of analysis [ 46 , 47 ]. Many students, for example, undertake literature reviews that may be informed by systematic review principles of rigor and transparency of reporting; some of these maybe relatively modest exercises whilst others make up a substantial component of the thesis. If rigor of execution and reporting are reduced too far then it may be more appropriate to characterize the work as non systematic scoping than as a systematic review.

Reviews thus vary in the extent that they engage with a research issue. The enterprise may range in size from, for instance, a specific program theory to a whole field of research. The enterprise may be under study by one research team, by a broader group such as a review group in an international collaboration or be the focus of study by many researchers internationally. The enterprises may be led academic disciplines, applied review collaborations, by priority setting agendas, and by forums to enable different perspectives to be engaged in research agendas. Whatever the nature of the strategic content or process of these macro research issues, reviews vary in the extent that they plan to contribute to such more macro questions. Reviews thus vary in the extent that this research work is done within a review; rather than before and after a review (by primary studies or by other reviews).

Reviews can be undertaken with different levels of skill, efficiency, and automated tools [ 48 ] and so resources do not equate exactly with the ‘work done’ in progressing a research issue. In general, a broad review with relatively little depth (providing a systematic overview) may be comparable in work done to a detailed narrow review (as in many current statistical meta-analyses). A multi-component review addressing complex questions using both aggregative and configuring methods may be attempting to achieve more work, though there may be challenges in terms of maintaining consistency or transparency of detail in each component of the review. In contrast, a rapid review has few resources and so is attempting less than other reviews but there may be dangers that the limited scope (and limited contribution to the broader research agenda) is not understood by funders and users of the review. How best to use available resources is a strategic issue depending upon the nature of the review question, the state of the research available on that issue and the knowledge about that state of the research. It is an issue of being fit for purpose. A review doing comparatively little ‘work’ may be exactly what is needed in one situation but not in another.

Explicit accountable methods are required for primary research and reviews of research. This logic applies to all research questions and thus multiple methods for reviews of research are required, just as they are required for primary research. These differences in types of reviews reflect the richness of primary research not only in the range of variation but also in the philosophical and methodological challenges that they pose including the mixing of different types of methods. The dominance of one form of review question and review method and the branding of some other forms of review does not clearly describe the variation in review designs and methods and the similarities and differences between these methods. Clarity about the dimensions along which reviews vary provides a way to develop review methods further and to make critical judgments necessary for the commission, production, evaluation, and use of reviews. This paper has argued for the need for clarity in describing the design and methods of systematic reviews along many dimensions; and that particularly useful dimensions for planning, describing, and evaluating reviews are:

Review aims and approach: (i) approach of the review: ontological, epistemological, theoretical, and ideological assumptions of the reviewers and users of the review including any theoretical mode; (ii) review question: the type of answer that is being sought (and the type of information that would answer it); and (ii) aggregation and configuration: the relative use of these logics and strategies in the different review components (and the positioning of theory in the review process, the degree of homogeneity of data, and the iteration of review method).

Structure and components of reviews: (iv) the systematic map and synthesis components of the review; and (v) the relation between these components.

Breadth, depth, and ‘work done’ by reviews: (vi) macro research strategy: the positioning of the review (and resources and the work aimed to be done) within the state of what is already known and other research planned by the review team and others; and (vii) the resources used to achieve this.

Clarifying some of the main dimensions along which reviews vary can provide a framework within which description of more detailed aspects of methodology can occur; for example, the specific strategies used for searching, identifying, coding, and synthesizing evidence and the use of specific methods and techniques ranging from review management software to text mining to statistical and narrative methods of analysis. Such clearer descriptions may lead in time to a more overarching system of terminology for systematic reviews.

Authors’ information

DG, JT, and SO are all directors of the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) [ 49 ].

Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, Russell Sage Foundation, New York

Google Scholar  

Gough D: Dimensions of difference in evidence reviews (Overview; I. Questions, evidence and methods; II.Breadth and depth; III. Methodological approaches; IV. Quality and relevance appraisal; V. Communication, interpretation and application. Series of six posters presented at National Centre for Research Methods meeting, Manchester. January 2007, EPPI-Centre, London, http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1919 ,

Gough D, Thomas J: Commonality and diversity in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 35-65.

Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Professions. 2002, 25: 12-37. 10.1177/0163278702025001003.

Article   Google Scholar  

Bohlin I: Formalising syntheses of medical knowledge: the rise of meta-analysis and systematic reviews. Perspect Sci. in press, in press

Noblit G: Hare RD: Meta-ethnography: synthesizing qualitative studies. 1988, Sage Publications, Newbury Park NY

Noyes J, Popay J, Pearson A, Hannes K, Booth A: Qualitative research and Cochrane reviews. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. Version 5.1.0 (updated March 2011). The Cochrane Collaboration. www.cochrane-handbook.org

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.

Article   PubMed   PubMed Central   Google Scholar  

Pawson R: Evidenced-based policy: a realist perspective. 2006, Sage, London

Book   Google Scholar  

Shepperd S, Lewin S, Struas S, Clarke M, Eccles M, Fitzpatrick R, Wong G, Sheikh A: Can we systematically review studies that evaluate complex interventions?. PLoS Med. 2009, 6: 8-10.1371/journal.pmed.1000008.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R: Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005, 61: 417-430. 10.1016/j.socscimed.2004.12.001.

Article   PubMed   Google Scholar  

Barnett-Page E, Thomas J: Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009, 9: 59-10.1186/1471-2288-9-59.

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement. PLoS Med. 2009, 6: 6-10.1371/journal.pmed.1000006.

PLoS Medicine Editors: Best practice in systematic reviews: The importance of protocols and registration. PLoS Med. 2011, 8: 2-

Thomas G: Introduction: evidence and practice. Evidence-based Practice in Education. Edited by: Pring R, Thomas G. 2004, Open University Press, Buckingham, 44-62.

Gough D, Oliver S, Newman M, Bird K: Transparency in planning, warranting and interpreting research. Teaching and Learning Research Briefing 78. 2009, Teaching and Learning Research Programme, London

Strauss A, Corbin J: Basics of qualitative research, grounded theory procedures and techniques. 1990, Sage, London

Miles M, Huberman A: Qualitative Data Analysis. 1994, Sage, London

Voils CI, Sandelowski M, Barroso J, Hasselblad V: Making sense of qualitative and quantitative findings in mixed research synthesis studies. Field Methods. 2008, 20: 3-25. 10.1177/1525822X07307463.

Sandelowski M, Voils CJ, Leeman J, Crandlee JL: Mapping the Mixed Methods-Mixed Research Synthesis Terrain. Journal of Mixed Methods Research. 2011, 10.1177/1558689811427913.

Pawson R, Boaz A, Grayson L, Long A, Barnes C: Types and Quality of Knowledge in Social Care. 2003, Social Care Institute for Excellence, London

Oancea A, Furlong J: Expressions of excellence and the assessment of applied and practice-based research. Res Pap Educ. 2007, 22: 119-137. 10.1080/02671520701296056.

Harden A, Gough D: Quality and relevance appraisal. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 153-178.

Thomas J, Harden A: Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008, 8: 45-10.1186/1471-2288-8-45.

Oliver S, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, Stein K, Buchanan P, Gyte G: A multidimensional conceptual framework for analyzing public involvement in health services research. Heal Expect. 2008, 11: 72-84. 10.1111/j.1369-7625.2007.00476.x.

Carroll C, Booth A, Cooper K: A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011, 11: 29-10.1186/1471-2288-11-29.

Rees R, Oliver S: Stakeholder perspectives and participation in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 17-34.

Oliver S, Dickson K, Newman M: Getting started with a review. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 66-82.

Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: a Framework for Assessing Research Evidence. 2003, Government Chief Social Researcher’s Office, London

Weiss C: The many meanings of research utilisation. Public Adm Rev. 1979, 29: 426-431.

Peersman G: A Descriptive Mapping of Health Promotion Studies in Young People EPPI Research Report. 1996, EPI-Centre, London

Arksey H, O’Malley L: Scoping Studies: towards a methodological framework. Int J Soc Res Methodol. 2005, 8: 19-32. 10.1080/1364557032000119616.

Levac D, Colquhoun H, O’Brien KK: Scoping studies: advancing the methodology. Implement Sci. 2010, 5: 69-10.1186/1748-5908-5-69.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. Brit Med J. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.

Harden A, Thomas J: Mixed methods and systematic reviews: examples and emerging issues. Handbook of Mixed Methods in the Social and Behavioral Sciences. Edited by: Tashakkori A, Teddlie C. 2010, Sage, London, 749-774. 2

Chapter   Google Scholar  

Leontien M, van der Knaap , Leeuw FL, Bogaerts S, Laura TJ: Nijssen Combining campbell standards and the realist evaluation approach: the best of two worlds?. J Eval. 2008, 29: 48-57. 10.1177/1098214007313024.

Smith V, Devane D, Begley CM, Clarke M: Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011, 11: 15-10.1186/1471-2288-11-15.

Oliver S, Bagnall AM, Thomas J, Shepherd J, Sowden A, White I, Dinnes J, Rees R, Colquitt J, Oliver K, Garrett Z: RCTs for policy interventions: a review of reviews and meta-regression. Health Technol Assess. 2010, 14: 16-

Scriven M: An introduction to meta-evaluation. Educational Products Report. 1969, 2: 36-38.

Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S: A conceptual framework for implementation fidelity. Implement Sci. 2007, 2: 40-10.1186/1748-5908-2-40.

Smith ML, Glass GV: Meta-analysis of psychotherapy outcome studies. Am Psychol. 1977, 32: 752-760.

Article   CAS   PubMed   Google Scholar  

Hattie J: Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. 2008, Routledge, London

Cook TD, Cooper H, Cordray DS, Hartmann H, Hedges LV, Light RJ, Louis TA, Mosteller F: Meta-analysis for Explanation: A Casebook. 1992, Russell Sage Foundation, New York

Thompson SG, Sharp SJ: Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999, 18: 2693-2708. 10.1002/(SICI)1097-0258(19991030)18:20<2693::AID-SIM235>3.0.CO;2-V.

Stewart R, Oliver S: Making a difference with systematic reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 227-244.

Government Social Research Unit: Rapid Evidence Assessment Toolkit. 2008, http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment ,

Abrami PC, Borokhovski E, Bernard RM, Wade CA, Tamim R, Persson T, Surkes MA: Issues in conducting and disseminating brief reviews of evidence. Evidence & Policy: A Journal of Research, Debate and Practice. 2010, 6: 371-389. 10.1332/174426410X524866.

Brunton J, Thomas J: Information management in reviews. Introduction to Systematic Reviews. Edited by: Gough D, Oliver S, Thomas J. 2012, Sage, London, 83-106.

Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre): http://eppi.ioe.ac.uk ,

Download references

Acknowledgements

The authors wish to acknowledge the support and intellectual contribution of their previous and current colleagues at the EPPI-Centre. They also wish to acknowledge the support of their major funders as many of the ideas in this paper were developed whilst working on research supported by their grants; this includes the Economic and Social Research Council, the Department of Health, and the Department for Education. The views expressed here are those of the authors and are not necessarily those of our funders.

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, 20 Bedford Way, London, WC1H 0AL, UK

David Gough, James Thomas & Sandy Oliver

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Gough .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors’ contributions

All three authors have made substantial contributions to the conception of the ideas in this paper, have been involved in drafting or revising it critically for important intellectual content, and have given final approval of the version to be published.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, authors’ original file for figure 4, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Gough, D., Thomas, J. & Oliver, S. Clarifying differences between review designs and methods. Syst Rev 1 , 28 (2012). https://doi.org/10.1186/2046-4053-1-28

Download citation

Received : 04 January 2012

Accepted : 19 March 2012

Published : 09 June 2012

DOI : https://doi.org/10.1186/2046-4053-1-28

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Aggregation configuration
• Complex reviews
• Methodology
• Research methods
• Scoping reviews
• Systematic reviews
• Taxonomy of reviews

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Study designs: Part 7 - Systematic reviews

Affiliations.

• 1 Department of Anaesthesiology, Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, Maharashtra, India.
• 2 Director, Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry, India.
• PMID: 32670836
• PMCID: PMC7342340
• DOI: 10.4103/picr.PICR_84_20

In this series on research study designs, we have so far looked at different types of primary research designs which attempt to answer a specific question. In this segment, we discuss systematic review, which is a study design used to summarize the results of several primary research studies. Systematic reviews often also use meta-analysis, which is a statistical tool to mathematically collate the results of various research studies to obtain a pooled estimate of treatment effect; this will be discussed in the next article.

Keywords: Research design; review [publication type]; systematic review [publication type].

Copyright: © 2020 Perspectives in Clinical Research.

• Open access
• Published: 15 May 2024

Factors and management techniques in odontogenic keratocysts: a systematic review

• Mario Dioguardi 1 ,
• Cristian Quarta 1 ,
• Diego Sovereto 1 ,
• Giorgia Apollonia Caloro 2 ,
• Andrea Ballini 1 ,
• Riccardo Aiuto 3 ,
• Angelo Martella 4 ,
• Lorenzo Lo Muzio 1 &
• Michele Di Cosola 1  

European Journal of Medical Research volume  29 , Article number:  287 ( 2024 ) Cite this article

159 Accesses

Metrics details

Odontogenic keratocysts exhibit frequent recurrence, distinctive histopathological traits, a tendency towards aggressive clinical behavior, and a potential linkage to the nevoid basal cell carcinoma syndrome. The aim of this systematic review is to compile insights concerning the control of this condition and assess the effectiveness of various treatment approaches in reducing the likelihood of recurrence.

Materials and methods

The following systematic review adhered to the PRISMA guidelines. The systematic revision was registered on PROSPERO and  structured around the questions related to the population, intervention, control, outcome and study design (PICOS).

After conducting a search on the PubMed database, we initially identified 944 records. After using end-note software to remove duplicate entries, results totally with 462 distinct records. A thorough review of the titles and abstracts of these articles led to the selection of 50 papers for in-depth examination. Ultimately, following the application of our eligibility criteria, we incorporated 11 articles into our primary outcome analysis.

Among the studies examined, the most common location for these lesions was found to be in the area of the mandibular ramus and the posterior region of the mandible. In cases where the exact location wasn’t specified, the mandible emerged as the predominant site. When we considered the characteristics of these lesions in studies that mentioned locularity, most were described as unilocular in two studies, while in two other studies, the prevalence of multilocular lesions was observed. Risk factors associated with keratocyst recurrence include younger patient age, the presence of multilocular lesions, larger lesion size, and a longer anteroposterior dimension. Certain treatment methods have demonstrated a lack of relapses. These include the use of 5-fluorouracil, marsupialization, enucleation with peripheral ostectomy or resection, enucleation and curettage, as well as resection without creating continuity defects. However, it is important to note that further research is essential. Prospective studies and randomized trials are needed to collect more comprehensive evidence regarding the effectiveness of various treatment approaches and follow-up protocols for managing odontogenic keratocysts.

Clinical relevance

Odontogenic keratocysts still enter into differential diagnoses with other lesions that affect the jaw bones such as ameloblastama and other tumor forms, furthermore it is not free from recurrence, therefore the therapeutic approach to the lesion aimed at its elimination can influence both the possible recurrence and complications, knowledge of the surgical methods that offer the most predictable and clinically relevant result for the management of follow-up and recurrences.

Introduction

The odontogenic keratocyst (OKC) is a developmental cyst that originates from remnants of the dental lamina within the jawbones [ 1 ]. Several studies have reported a preference for males [ 1 , 2 , 3 ], with an incidence peak around the third decade [ 4 ] and a nearly equal distribution in other decades, with another small peak between 50 and 70 years of age [ 1 ]. It can occur in any area of the jawbones but is most commonly found in the mandible, with a particular preference for the mandibular angle extending to the mandibular ramus [ 4 ].

Diagnosis of OKC is typically  radiological. Radiographs commonly reveal well-defined radiolucent areas with  rounded or scalloped margins that are well demarcated; these areas can present as either multilocular or unilocular [ 5 ].

In the 2022 classification, OKC remains classified as a cyst; molecular studies have detected frequent mutations in the tumor suppressor gene PTCH1, a gene that activates the SHH pathway, leading to aberrant epithelial proliferation [ 1 ], sparking debates on whether OKC is a cyst or a cystic neoplasm. It was labeled as a keratocystic odontogenic tumor in 2005 [ 5 ], thus considered a cystic neoplasm, and later reclassified as a cyst in the 2017 classification [ 1 ].

Keratocysts are characterized by a high recurrence rate, specific histological features, aggressive clinical behavior, and can be associated with the nevoid basal cell carcinoma syndrome [ 6 ].

The mechanism of recurrence was proposed by Brannon [ 7 ] in 1976, suggesting it was due to three different mechanisms:

Incomplete removal of the cyst,

Growth of new keratocysts from satellite cysts,

Development of a new keratocyst in the area adjacent to the site of the primary keratocyst, interpreted as recurrence.

Odontogenic keratocysts can be treated with various surgical methods, which can be divided into conservative approaches and invasive approaches or a combination thereof [ 8 ]; in the literature, enucleation, marsupialization, resection, and the use of adjunct therapies such as Carnoy’s solution and cryotherapy are reported [ 1 , 4 , 9 ].

Despite many studies in the literature examining several therapeutic approaches in managing this lesion, it is still not clear which method provides lower recurrence rates without causing significant morbidity [ 10 ]; the purpose of this systematic review is to gather information on the management of this lesion and evaluate which treatment method results in fewer recurrences.

The following systematic review adhered to the PRISMA (Preferred Reporting Items for Systematic review and Meta-Analysis) protocol guidelines [ 11 ].

The systematic revision was registered on PROSPERO with number of: CRD42023480051.

The study was structured around the questions related to the population, intervention, control, outcome and study design (PICOS):

Population (P): individuals with non-syndromic or syndromic odontogenic keratocyst (initial cases) diagnosed histologically;

Intervention (I): surgical interventions for patients with odontogenic keratocystic, such as enucleation, enucleation coupled with curettage, enucleation with additional therapeutic measures (such as Carnoy's solution application, cryotherapy), marsupialization or decompression, with or without subsequent cystectomy and adjunctive therapy, and resection;

Control (C): not applicable;

Outcome (O): recurrence of KOT (Keratocystic Odontogenic Tumor) associated with distinct surgical treatments and characteristics of the keratocysts analyzed;

Study design (S): prospective randomized controlled clinical trials, controlled clinical investigations (either prospective or retrospective), and case series that explored and compared the diverse surgical approaches concerning recurrence over a suitable follow-up period (minimum of 1 year).

The formulation of the PICOS question can be summarized as follows: “What characteristics do the odontogenic keratocysts analyzed in the studies have? Which surgeries had the least recurrences during the follow-up?”.

Following the initial selection phase of records identified in various databases, potentially eligible articles were qualitatively assessed. This assessment aimed to investigate which surgical treatment was the most reliable in giving the least number of recurrences.

Eligibility criteria

This text discusses the process of selecting research articles for a study related to the recurrence of KOT associated with distinct surgical interventions, such as enucleation, with or without curettage and additional therapeutic measures, marsupialization or decompression, with or without subsequent cystectomy and adjunctive therapy, and resection.

The process involved initially identifying potentially eligible articles based on their abstracts. These articles were then subjected to a thorough examination of their full content to determine their suitability for both qualitative and quantitative analyses.

The criteria for including articles in the full-text analysis were studies relating to KOT treatments in which the number of recurrences and the general characteristics of the lesions are reported.

The exclusion criteria were applied to exclude the following types of studies:

Studies involving animals or conducted in a laboratory setting (in vitro)

Letters to the editor

Articles that did not adequately specify the type of surgical method used

Studies with an inadequate follow-up period (less than 1 year)

Clinical studies conducted more than 30 years ago (only studies from the last 30 years were included because classifications and surgical and therapeutic techniques have been constantly changing and improving, with generally earlier diagnoses and more suitable treatments with lower recurrence rates. Therefore, to avoid increasing the heterogeneity of the included studies and to prevent bias in the aggregated treatment results, the reviewers collectively decided to include only studies from 1989 onwards)

Review articles

Research methodology

Studies have been identified through bibliographic research on electronic databases.

The literature search was conducted on the search engines “PubMed”. The search on the providers was conducted between 02.09.2023 and 12.09.2023, and the last search for a partial update of the literature was conducted on 18.09.2023.

The following search terms were used on PubMed: “KOT” AND “Recurrence” (37 records), “odontogenic keratocyst marsupialization” (285 records), “odontogenic keratocyst enucleation” (622 records).

Screening methodology

The selection criteria and their combinations for searching were established prior to the record identification stage through mutual consensus between the two reviewers  (M.D. and M.D.C.) responsible for choosing potentially eligible articles. Following this, the records acquired were then assessed separately by the two independent reviewers, with a third reviewer  (A.B.) serving as an decision-maker in cases of uncertainty.

The screening process involved evaluating the titles and abstracts of articles, and in cases where there was uncertainty, a more in-depth examination of the article's content was conducted to remove records that were not relevant to the topics under review.

Following a search in the PubMed database, 944 records were initially located. Subsequently, after applying end-note software to eliminate duplications, 462 unique records remained. Upon reviewing the titles and abstracts of these articles, after this initial screening, a total of 50 articles were selected for a thorough examination of their full text by two reviewers. From these 50 articles, the ones that met the criteria for qualitative analysis for the outcome were identified. Finally, applying the eligibility criteria, we included 16 articles for the primary outcome analysis (Fig.  1 ).

Flowchart of the different phases of the systematic review

Study characteristics and data extraction

The included studies for the quantitative analysis were: Maurette et al. [ 12 ]; Nakamura et al. [ 13 ]; Bataineh and al Qudah [ 14 ]; Leung et al. [ 15 ]; Kolokythas et al. [ 9 ]; Berge et al. [ 16 ]; Pogrel and Jordan, [ 17 ]; Tabrizi et al. [ 18 ]; Zecha et al. [ 19 ]; Moellmann et al. [ 20 ]; Caminiti et al.[ 21 ], Stoelinga [ 4 ]; Dammer et al. [ 2 ]; Marker et.al. [ 22 ]; August et al.[ 23 ]; Brøndum and Jensen [ 24 ].

The extracted data included the journal (author, data, and reference); study design; number of patients (males/females); number of lesions; number of lesions associated with basal cell naevus syndrome (BCNS); mean age (range); site where the lesions were diagnosed; locularity (multilocular or unilocular); type of treatment; mean follow-up.

Finally, for each study, the number of relapses relating to each treatment was observed.

The data extracted are shown in Table  1 and 2 .

Risk of bias

The risk of bias was assessed using the Newcastle–Ottawa Scale (NOS) for cohort studies, assigning a value from 0 to 3 for each item, the assessment of the risk of bias was assessed by the first reviewer, and was deemed acceptable for all included studies, details are shown in Table  3

The articles included in this review analyze different types of keratocyst treatment and lesion characteristics.

Among the first to coin the term 'odontogenic keratocyst' was Philipsen in 1956, who, in a literature review, proposed the term 'odontogenic keratocyst' for all odontogenic cysts that exhibit epithelial keratinization [ 25 ].

The terminology, as adopted by Pindborg in 1962 and 1963 and also used by Toller in 1967, replaced the term ‘primordial cyst’ with ‘odontogenic keratocyst’, identifying 33 odontogenic keratocysts (study not included in this review) [ 26 , 27 , 28 , 29 ]

One of the early retrospective studies conducted on odontogenic keratocysts was performed by Pindborg, who retrospectively identified 26 keratinized cysts out of a total of 791 odontogenic cysts in 1962 [ 27 ].

The odontogenic keratocysts are often described in literature as benign cysts occurring within the bones, and they exhibit a propensity for infiltrative and aggressive growth patterns. These cysts make up an estimated 2–21.8% of all cysts affecting the jaw [ 24 , 25 ]. Moreover, there is a potential association between these cysts and genetic mutations, notably linked to nevoid basal cell carcinoma syndrome (NBCCS), a condition characterized by the presence of multiple OKCs in the jaw region [ 26 ]; this is also found in one of the articles included in this review [ 13 ], while in others the association was not specified [ 14 , 17 ] or there was no association at all [ 9 , 12 , 15 , 16 , 18 , 19 , 20 , 21 ]; many of these studies have placed the correlation with this syndrome in the exclusion criteria, as in the patients who are affected by it the probability that these cysts will reappear is high, and therefore it would be difficult to distinguish a recurrent event from the appearance of a new cyst [ 21 ]

These cysts are notorious for their tendency to grow aggressively in their immediate prossimity and for having a notably high rate of recurrence. Several contributing factors underpin this recurrence, including the use of inadequate treatment methods, incomplete elimination of the cyst, a high rate of cell division (mitotic index) within the cyst's epithelial cells, a larger cyst size, and the specific location of the cyst. The latter factor becomes especially problematic if it is challenging to access surgically [ 25 , 27 ]. Although they exhibit hostile conduct, OKC generally induce limited bone enlargement as they tend to proliferate within the intramedullary region, effectively growing within the bone [ 30 ].

Substantial lesions marked by substantial cortical plate erosion and engagement with neighboring structures may not produce symptoms in individuals, resulting in a delayed diagnosis [ 31 ].

The most frequent location of the lesions in the studies analyzed is at the level of the mandibular ramus and in the posterior mandible [ 12 , 13 , 14 , 15 , 16 , 19 ], and where the precise localization of the lesions is not specified, the mandible is the most frequent site [ 9 , 18 , 20 , 21 ]. In the studies in which locularity is specified among the characteristics of the lesions, the majority of the lesions were unilocular in two studies [ 13 , 21 ], while in two other studies the quantity of multilocular lesions was greater [ 14 , 15 ]. Younger patient age, multilocularity of the lesion, larger size, and longer anteroposterior dimension of the keratocyst have been identified as risk factors for keratocyst recurrence [ 15 ].

The treatments that have not had relapses are that with 5-fluorouracil [ 21 ], marsupialization [ 13 , 17 , 18 ], enucleation with peripheral ostectomy or resection [ 9 ], enucleation and curettage [ 12 ], and resection without continuity defects [ 14 ].

Decompression has been studied in 5 articles [ 9 , 12 , 22 , 23 , 24 ]; this method has the advantage of having minimal surgical morbidity and reduced risk to anatomical structures associated with the lesion, such as developing nerves or teeth [ 22 ]. Decompression and marsupialization techniques involve creating a communication between the cyst and the oral cavity, relieving pressure and allowing cyst shrinkage and bone apposition [ 12 ]. Clinical and radiographic resolution of OKCs after marsupialization is relatively rapid, typically within 19 months [ 17 ]. In studies where marsupialization alone was used for treatment, there were no relapses in two studies [ 17 , 18 ], while Zecha et al. [ 19 ] found four cases of relapse in ten patients treated with marsupialization.

Decompression and marsupialization are non-invasive treatment options for keratocysts, but require patient cooperation, including regular irrigation and follow-up [ 17 , 18 ].

Topical 5-fluorouracil is known for its antiproliferative effects on keratocystic epithelium and satellite cysts; furthermore, its use has some advantages, such as technical ease and the lack of neurotoxicity [ 21 ] and, in the only study of this review in which it were used in the treatment, there were no relapses [ 21 ].

Other treatment modalities used to reduce keratocyst recurrence are resection of the affected maxillary segment and enucleation with additional treatments such as curettage or ostectomy [ 9 , 14 ], which in these studies have not given recurrences, which, as regards resection, is a similar result to other studies in the literature [ 4 , 8 , 32 ]. However, despite the remarkably high success rate of this approach, resection is not widely embraced as a standard procedure, primarily due to concerns regarding its aggressiveness and associated postoperative complications, including morbidity [ 33 ]. Enucleation, often combined with curettage (the process of scraping the walls of the lesion cavity) or ostectomy (the surgical removal of bone tissue), is commonly used to treat keratocysts; although a more conservative treatment than resection, the effectiveness of this modality may be limited in cases where vital structures, such as the exposed inferior alveolar nerve, are at risk or when there is a perforation of the bony wall exposing the overlying mucosal tissue [ 15 ].

Carnoy’s solution was used in three studies [ 15 , 20 , 21 ] and of these studies one used the modified Carnoy’s solution [ 21 ]. The FDA avoid the use of Carnoy's solution containing chloroform in the United States, leading to the adoption of a modified formula. However, the modified formula has been found to have a higher relapse rate, suggesting the potential role that traditional Carnoy’s solution may have in treatment [ 34 ].

There are risk factors associated with the recurrence of odontogenic keratocyst, such as age, multilocularity, lesion size and radiographic characteristics.

The various surgical techniques used to treat keratocysts have potential benefits, including preservation of jaw function, reduction of the potential for recurrence, and eradication of the cystic lesion.

Marsupialization or decompression are advantageous conservative treatment options that aim to minimize surgical invasiveness while effectively managing keratocysts.

Long-term follow-up and monitoring of patients treated for these lesions is important to detect recurrence early.

There is a need for further research, prospective studies and randomized trials to gather more evidence on the effectiveness of different treatment methods and follow-up protocols for odontogenic keratocysts.

Availability of data and materials

All data generated or analyzed during this study are included in this published article.

Speight PM, Takata T. New tumour entities in the 4th edition of the World Health Organization Classification of Head and Neck tumours: odontogenic and maxillofacial bone tumours. Virchows Arch 2018;472:331–9. https://doi.org/10.1007/s00428-017-2182-3

Dammer R, Niederdellmann H, Dammer P, Nuebler-Moritz M. Conservative or radical treatment of keratocysts: a retrospective review. Br J Oral Maxillofac Surg. 1997;35:46–8. https://doi.org/10.1016/s0266-4356(97)90009-7 .

Article   CAS   PubMed   Google Scholar  

Ahlfors E, Larsson A, Sjögren S. The odontogenic keratocyst: a benign cystic tumor? J Oral Maxillofac Surg. 1984;42:10–9. https://doi.org/10.1016/0278-2391(84)90390-2 .

Stoelinga PJ. Long-term follow-up on keratocysts treated according to a defined protocol. Int J Oral Maxillofac Surg. 2001;30:14–25. https://doi.org/10.1054/ijom.2000.0027 .

Barnes L. Pathology and genetics of head and neck tumours; IARC.2005;9.

Soluk-Tekkesin M, Wright JM. The World Health Organization classification of odontogenic lesions: a summary of the changes of the 2022 (5th) edition. Turk Patoloji Derg. 2022;38:168–84. https://doi.org/10.5146/tjpath.2022.01573 .

Article   PubMed   PubMed Central   Google Scholar  

Brannon RB. The odontogenic keratocyst. A clinicopathologic study of 312 cases. Part I. Clinical features. Oral Surg Oral Med Oral Pathol. 1976;42:54–72. https://doi.org/10.1016/0030-4220(76)90031-1 .

Titinchi F. Protocol for management of odontogenic keratocysts considering recurrence according to treatment methods. J Korean Assoc Oral Maxillofac Surg. 2020;46:358–60. https://doi.org/10.5125/jkaoms.2020.46.5.358 .

Kolokythas A, Fernandes RP, Pazoki A, Ord RA. Odontogenic keratocyst: to decompress or not to decompress? A comparative study of decompression and enucleation versus resection/peripheral ostectomy. J Oral Maxillofac Surg. 2007;65:640–4. https://doi.org/10.1016/j.joms.2006.06.284 .

Article   PubMed   Google Scholar  

Troiano G, Dioguardi M, Cocco A, Laino L, Cervino G, Cicciu M, Ciavarella D, Lo Muzio L. Conservative vs radical approach for the treatment of solid/multicystic ameloblastoma: a systematic review and meta-analysis of the last decade. Oral Health Prev Dent. 2017;15:421–6. https://doi.org/10.3290/j.ohpd.a38732 .

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:e1-34. https://doi.org/10.1016/j.jclinepi.2009.06.006 .

Maurette PE, Jorge J, de Moraes M. Conservative treatment protocol of odontogenic keratocyst: a preliminary study. J Oral Maxillofac Surg. 2006;64:379–83. https://doi.org/10.1016/j.joms.2005.11.007 .

Nakamura N, Mitsuyasu T, Mitsuyasu Y, Taketomi T, Higuchi Y, Ohishi M. Marsupialization for odontogenic keratocysts: long-term follow-up analysis of the effects and changes in growth characteristics. Oral Surg Oral Med Oral Pathol Oral Radiol Endod. 2002;94:543–53. https://doi.org/10.1067/moe.2002.128022 .

Bataineh AB, Al Qudah M. Treatment of mandibular odontogenic keratocysts. Oral Surg Oral Med Oral Pathol Oral Radiol Endod. 1998;86:42–7. https://doi.org/10.1016/s1079-2104(98)90148-2 .

Leung YY, Lau SL, Tsoi KY, Ma HL, Ng CL. Results of the treatment of keratocystic odontogenic tumours using enucleation and treatment of the residual bony defect with carnoy’s solution. Int J Oral Maxillofac Surg. 2016;45:1154–8. https://doi.org/10.1016/j.ijom.2016.02.002 .

Berge TI, Helland SB, Sælen A, Øren M, Johannessen AC, Skartveit L, Grung B. Pattern of recurrence of nonsyndromic keratocystic odontogenic tumors. Oral Surg Oral Med Oral Pathol Oral Radiol. 2016;122:10–6. https://doi.org/10.1016/j.oooo.2016.01.004 .

Pogrel MA, Jordan RC. Marsupialization as a definitive treatment for the odontogenic keratocyst. J Oral Maxillofac Surg. 2004;62:651–65. https://doi.org/10.1016/j.joms.2003.08.029 .

Tabrizi R, Özkan BT, Dehgani A, Langner NJ. Marsupialization as a treatment option for the odontogenic keratocyst. J Craniofac Surg. 2012;23:e459-461. https://doi.org/10.1097/SCS.0b013e31825b3308 .

Zecha JA, Mendes RA, Lindeboom VB, van der Waal I. Recurrence rate of keratocystic odontogenic tumor after conservative surgical treatment without adjunctive therapies-a 35 year single institution experience. Oral Oncol. 2010;46:740–2. https://doi.org/10.1016/j.oraloncology.2010.07.004 .

Moellmann HL, Parviz A, Goldmann-Kirn M, Rana M, Rana M. Comparison of five different treatment approaches of mandibular keratocystic odontogenic keratocyst (OKC): a retrospective recurrence analysis of clinical and radiographic parameters. J Maxillofac Oral Surg. 2023. https://doi.org/10.1007/s12663-023-01929-0 .

Caminiti MF, El-Rabbany M, Jeon J, Bradley G. 5-fluorouracil is associated with a decreased recurrence risk in odontogenic keratocyst management: a retrospective cohort study. J Oral Maxillofac Surg. 2021;79:814–21. https://doi.org/10.1016/j.joms.2020.07.215 .

Marker P, Brøndum N, Clausen PP, Bastian HL. Treatment of large odontogenic keratocysts by decompression and later cystectomy: a long-term follow-up and a histologic study of 23 cases. Oral Surg Oral Med Oral Pathol Oral Radiol Endod. 1996;82:122–31. https://doi.org/10.1016/s1079-2104(96)80214-9 .

August M, Faquin WC, Troulis MJ, Kaban LB. Dedifferentiation of odontogenic keratocyst epithelium after cyst decompression. J Oral Maxillofac Surg. 2003;61:678–83. https://doi.org/10.1053/joms.2003.50137 .

Brøndum N, Jensen VJ. Recurrence of keratocysts and decompression treatment. a long-term follow-up of forty-four cases. Oral Surg Oral Med Oral Pathol. 1991;72:265–9. https://doi.org/10.1016/0030-4220(91)90211-t .

HP P. Om keratocyster (kolesten tomer) in the jaws. Tandlaegebladet. 1956;60:963–81.

Google Scholar  

Toller P. Origin and growth of cysts of the jaws. Ann R Coll Surg Engl. 1967;40:306–36.

CAS   PubMed   PubMed Central   Google Scholar  

Pindborg JJ, Hansen J. Studies on odontogenic cyst epithelium. 2. clinical and roentgenologic aspects of odontogenic keratocysts. Acta Pathol Microbiol Scand. 1963;58:283–94.

Rud J, Pindborg JJ. Odontogenic keratocysts: a follow-up study of 21 cases. J Oral Surg. 1969;27:323–30.

CAS   PubMed   Google Scholar  

Panders AK, Haddlers HN. Solitary keratocysts of the jaws. J Oral Surg. 1969;27:931–8.

Scarfe WC, Toghyani S, Azevedo B. Imaging of benign odontogenic lesions. Radiol Clin North Am. 2018;56:45–62. https://doi.org/10.1016/j.rcl.2017.08.004 .

Eryilmaz T, Ozmen S, Findikcioglu K, Kandal S, Aral M. Odontogenic keratocyst: an unusual location and review of the literature. Ann Plast Surg. 2009;62:210–2. https://doi.org/10.1097/SAP.0b013e31817dad9c .

Pitak-Arnnop P, Chaine A, Oprean N, Dhanuthai K, Bertrand JC, Bertolus C. Management of odontogenic keratocysts of the jaws: a 10 year experience with 120 consecutive lesions. J Craniomaxillofac Surg. 2010;38:358–64. https://doi.org/10.1016/j.jcms.2009.10.006 .

Kaczmarzyk T, Mojsa I, Stypulkowska J. A systematic review of the recurrence rate for keratocystic odontogenic tumour in relation to treatment modalities. Int J Oral Maxillofac Surg. 2012;41:756–67. https://doi.org/10.1016/j.ijom.2012.02.008 .

Dashow JE, McHugh JB, Braun TM, Edwards SP, Helman JI, Ward BB. Significantly decreased recurrence rates in keratocystic odontogenic tumor with simple enucleation and curettage using carnoy’s versus modified carnoy’s solution. J Oral Maxillofac Surg. 2015;73:2132–5. https://doi.org/10.1016/j.joms.2015.05.005 .

Download references

Acknowledgements

Not applicable

This research received no external funding.

Author information

Authors and affiliations.

Department of Clinical and Experimental Medicine, University of Foggia, Via Rovelli 50, 71122, Foggia, Italy

Mario Dioguardi, Cristian Quarta, Diego Sovereto, Andrea Ballini, Lorenzo Lo Muzio & Michele Di Cosola

Unità Operativa Nefrologia e Dialisi, Presidio Ospedaliero Scorrano, ASL (Azienda Sanitaria Locale) Lecce, Via Giuseppina Delli Ponti, 73020, Scorrano, Italy

Giorgia Apollonia Caloro

Department of Biomedical, Surgical, and Dental Science, University of Milan, 20122, Milan, Italy

Riccardo Aiuto

DataLab, Department of Engineering for Innovation, University of Salento, Lecce, Italy

Angelo Martella

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, M.D.and C.Q.; methodology, M.D.; software, M.D. and D.S.; validation, M.D. and A.B.; formal analysis, M.D.; investigation, M.D. and C.Q.; data curation, M.D. and D.S.; bibliographic reserach, C.Q. and R.A.; writing—original draft preparation, M.D. and C.Q.; writing—review and editing, M.D. and A.B.; visualization, D.S and M.D..; supervision L.L.M.., and M.D.C.; Critical revision of the manuscript for important intellectual content M.D., C.Q.; and A.B.; Bioinformatic analysis review, A.M.; project administration, L.L.M. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Mario Dioguardi .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Institutional Review Board Statement

Consent for publication, competing interests.

The authors declare no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Dioguardi, M., Quarta, C., Sovereto, D. et al. Factors and management techniques in odontogenic keratocysts: a systematic review. Eur J Med Res 29 , 287 (2024). https://doi.org/10.1186/s40001-024-01854-z

Download citation

Received : 26 January 2024

Accepted : 22 April 2024

Published : 15 May 2024

DOI : https://doi.org/10.1186/s40001-024-01854-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

European Journal of Medical Research

ISSN: 2047-783X

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Med Prim Care
• v.2(1); Jan-Mar 2013

Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare

S. gopalakrishnan.

Department of Community Medicine, SRM Medical College, Hospital and Research Centre, Kattankulathur, Tamil Nadu, India

P. Ganeshkumar

Healthcare decisions for individual patients and for public health policies should be informed by the best available research evidence. The practice of evidence-based medicine is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient's values and expectations. Primary care physicians need evidence for both clinical practice and for public health decision making. The evidence comes from good reviews which is a state-of-the-art synthesis of current evidence on a given research question. Given the explosion of medical literature, and the fact that time is always scarce, review articles play a vital role in decision making in evidence-based medical practice. Given that most clinicians and public health professionals do not have the time to track down all the original articles, critically read them, and obtain the evidence they need for their questions, systematic reviews and clinical practice guidelines may be their best source of evidence. Systematic reviews aim to identify, evaluate, and summarize the findings of all relevant individual studies over a health-related issue, thereby making the available evidence more accessible to decision makers. The objective of this article is to introduce the primary care physicians about the concept of systematic reviews and meta-analysis, outlining why they are important, describing their methods and terminologies used, and thereby helping them with the skills to recognize and understand a reliable review which will be helpful for their day-to-day clinical practice and research activities.

Introduction

Evidence-based healthcare is the integration of best research evidence with clinical expertise and patient values. Green denotes, “Using evidence from reliable research, to inform healthcare decisions, has the potential to ensure best practice and reduce variations in healthcare delivery.” However, incorporating research into practice is time consuming, and so we need methods of facilitating easy access to evidence for busy clinicians.[ 1 ] Ganeshkumar et al . mentioned that nearly half of the private practitioners in India were consulting more than 4 h per day in a locality,[ 2 ] which explains the difficulty of them in spending time in searching evidence during consultation. Ideally, clinical decision making ought to be based on the latest evidence available. However, to keep abreast with the continuously increasing number of publications in health research, a primary healthcare professional would need to read an insurmountable number of articles every day, covered in more than 13 million references and over 4800 biomedical and health journals in Medline alone. With the view to address this challenge, the systematic review method was developed. Systematic reviews aim to inform and facilitate this process through research synthesis of multiple studies, enabling increased and efficient access to evidence.[ 1 , 3 , 4 ]

Systematic reviews and meta-analyses have become increasingly important in healthcare settings. Clinicians read them to keep up-to-date with their field and they are often used as a starting point for developing clinical practice guidelines. Granting agencies may require a systematic review to ensure there is justification for further research and some healthcare journals are moving in this direction.[ 5 ]

This article is intended to provide an easy guide to understand the concept of systematic reviews and meta-analysis, which has been prepared with the aim of capacity building for general practitioners and other primary healthcare professionals in research methodology and day-to-day clinical practice.

The purpose of this article is to introduce readers to:

• The two approaches of evaluating all the available evidence on an issue i.e., systematic reviews and meta-analysis,
• Discuss the steps in doing a systematic review,
• Introduce the terms used in systematic reviews and meta-analysis,
• Interpret results of a meta-analysis, and
• The advantages and disadvantages of systematic review and meta-analysis.

Application

What is the effect of antiviral treatment in dengue fever? Most often a primary care physician needs to know convincing answers to questions like this in a primary care setting.

To find out the solutions or answers to a clinical question like this, one has to refer textbooks, ask a colleague, or search electronic database for reports of clinical trials. Doctors need reliable information on such problems and on the effectiveness of large number of therapeutic interventions, but the information sources are too many, i.e., nearly 20,000 journals publishing 2 million articles per year with unclear or confusing results. Because no study, regardless of its type, should be interpreted in isolation, a systematic review is generally the best form of evidence.[ 6 ] So, the preferred method is a good summary of research reports, i.e., systematic reviews and meta-analysis, which will give evidence-based answers to clinical situations.

There are two fundamental categories of research: Primary research and secondary research. Primary research is collecting data directly from patients or population, while secondary research is the analysis of data already collected through primary research. A review is an article that summarizes a number of primary studies and may draw conclusions on the topic of interest which can be traditional (unsystematic) or systematic.

Terminologies

Systematic review.

A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors.[ 7 ] To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful.[ 8 ] Systematic reviews are often called overviews.

The evidence-based practitioner, David Sackett, defines the following terminologies.[ 3 ]

• Review: The general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic.
• Overview: When a review strives to comprehensively identify and track down all the literature on a given topic (also called “systematic literature review”).
• Meta-analysis: A specific statistical strategy for assembling the results of several studies into a single estimate.

Systematic reviews adhere to a strict scientific design based on explicit, pre-specified, and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. Systematic reviews can also demonstrate where knowledge is lacking. This can then be used to guide future research. Systematic reviews are usually carried out in the areas of clinical tests (diagnostic, screening, and prognostic), public health interventions, adverse (harm) effects, economic (cost) evaluations, and how and why interventions work.[ 9 ]

Cochrane reviews

Cochrane reviews are systematic reviews undertaken by members of the Cochrane Collaboration which is an international not-for-profit organization that aims to help people to make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

Cochrane Primary Health Care Field is a systematic review of primary healthcare research on prevention, treatment, rehabilitation, and diagnostic test accuracy. The overall aim and mission of the Primary Health Care Field is to promote the quality, quantity, dissemination, accessibility, applicability, and impact of Cochrane systematic reviews relevant to people who work in primary care and to ensure proper representation in the interests of primary care clinicians and consumers in Cochrane reviews and review groups, and in other entities. This field would serve to coordinate and promote the mission of the Cochrane Collaboration within the primary healthcare disciplines, as well as ensuring that primary care perspectives are adequately represented within the Collaboration.[ 10 ]

Meta-analysis

A meta-analysis is the combination of data from several independent primary studies that address the same question to produce a single estimate like the effect of treatment or risk factor. It is the statistical analysis of a large collection of analysis and results from individual studies for the purpose of integrating the findings.[ 11 ] The term meta-analysis has been used to denote the full range of quantitative methods for research reviews.[ 12 ] Meta-analyses are studies of studies.[ 13 ] Meta-analysis provides a logical framework to a research review where similar measures from comparable studies are listed systematically and the available effect measures are combined wherever possible.[ 14 ]

The fundamental rationale of meta-analysis is that it reduces the quantity of data by summarizing data from multiple resources and helps to plan research as well as to frame guidelines. It also helps to make efficient use of existing data, ensuring generalizability, helping to check consistency of relationships, explaining data inconsistency, and quantifies the data. It helps to improve the precision in estimating the risk by using explicit methods.

Therefore, “systematic review” will refer to the entire process of collecting, reviewing, and presenting all available evidence, while the term “meta-analysis” will refer to the statistical technique involved in extracting and combining data to produce a summary result.[ 15 ]

Steps in doing systematic reviews/meta-analysis

Following are the six fundamental essential steps while doing systematic review and meta-analysis.[ 16 ]

Define the question

This is the most important part of systematic reviews/meta-analysis. The research question for the systematic reviews may be related to a major public health problem or a controversial clinical situation which requires acceptable intervention as a possible solution to the present healthcare need of the community. This step is most important since the remaining steps will be based on this.

Reviewing the literature

This can be done by going through scientific resources such as electronic database, controlled clinical trials registers, other biomedical databases, non-English literatures, “gray literatures” (thesis, internal reports, non–peer-reviewed journals, pharmaceutical industry files), references listed in primary sources, raw data from published trials and other unpublished sources known to experts in the field. Among the available electronic scientific database, the popular ones are PUBMED, MEDLINE, and EMBASE.

Sift the studies to select relevant ones

To select the relevant studies from the searches, we need to sift through the studies thus identified. The first sift is pre-screening, i.e., to decide which studies to retrieve in full, and the second sift is selection which is to look again at these studies and decide which are to be included in the review. The next step is selecting the eligible studies based on similar study designs, year of publication, language, choice among multiple articles, sample size or follow-up issues, similarity of exposure, and or treatment and completeness of information.

It is necessary to ensure that the sifting includes all relevant studies like the unpublished studies (desk drawer problem), studies which came with negative conclusions or were published in non-English journals, and studies with small sample size.

Assess the quality of studies

The steps undertaken in evaluating the study quality are early definition of study quality and criteria, setting up a good scoring system, developing a standard form for assessment, calculating quality for each study, and finally using this for sensitivity analysis.

For example, the quality of a randomized controlled trial can be assessed by finding out the answers to the following questions:

• Was the assignment to the treatment groups really random?
• Was the treatment allocation concealed?
• Were the groups similar at baseline in terms of prognostic factors?
• Were the eligibility criteria specified?
• Were the assessors, the care provider, and the patient blinded?
• Were the point estimates and measure of variability presented for the primary outcome measure?
• Did the analyses include intention-to-treat analysis?

Calculate the outcome measures of each study and combine them

We need a standard measure of outcome which can be applied to each study on the basis of its effect size. Based on their type of outcome, following are the measures of outcome: Studies with binary outcomes (cured/not cured) have odds ratio, risk ratio; studies with continuous outcomes (blood pressure) have means, difference in means, standardized difference in means (effect sizes); and survival or time-to-event data have hazard ratios.

Combining studies

Homogeneity of different studies can be estimated at a glance from a forest plot (explained below). For example, if the lower confidence interval of every trial is below the upper of all the others, i.e., the lines all overlap to some extent, then the trials are homogeneous. If some lines do not overlap at all, these trials may be said to be heterogeneous.

The definitive test for assessing the heterogeneity of studies is a variant of Chi-square test (Mantel–Haenszel test). The final step is calculating the common estimate and its confidence interval with the original data or with the summary statistics from all the studies. The best estimate of treatment effect can be derived from the weighted summary statistics of all studies which will be based on weighting to sample size, standard errors, and other summary statistics. Log scale is used to combine the data to estimate the weighting.

Interpret results: Graph

The results of a meta-analysis are usually presented as a graph called forest plot because the typical forest plots appear as forest of lines. It provides a simple visual presentation of individual studies that went into the meta-analysis at a glance. It shows the variation between the studies and an estimate of the overall result of all the studies together.

Forest plot

Meta-analysis graphs can principally be divided into six columns [ Figure 1 ]. Individual study results are displayed in rows. The first column (“study”) lists the individual study IDs included in the meta-analysis; usually the first author and year are displayed. The second column relates to the intervention groups and the third column to the control groups. The fourth column visually displays the study results. The line in the middle is called “the line of no effect.” The weight (in %) in the fifth column indicates the weighting or influence of the study on the overall results of the meta-analysis of all included studies. The higher the percentage weight, the bigger the box, the more influence the study has on the overall results. The sixth column gives the numerical results for each study (e.g., odds ratio or relative risk and 95% confidence interval), which are identical to the graphical display in the fourth column. The diamond in the last row of the graph illustrates the overall result of the meta-analysis.[ 4 ]

Interpretation of meta-analysis[ 4 ]

Thus, the horizontal lines represent individual studies. Length of line is the confidence interval (usually 95%), squares on the line represent effect size (risk ratio) for the study, with area of the square being the study size (proportional to weight given) and position as point estimate (relative risk) of the study.[ 7 ]

For example, the forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults is shown in Figure 2 .[ 17 ]

Forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults[ 17 ]

The overall effect is shown as diamond where the position toward the center represents pooled point estimate, the width represents estimated 95% confidence interval for all studies, and the black plain line vertically in the middle of plot is the “line of no effect” (e.g., relative risk = 1).

Therefore, when examining the results of a systematic reviews/meta-analysis, the following questions should be kept in mind:

• Heterogeneity among studies may make any pooled estimate meaningless.
• The quality of a meta-analysis cannot be any better than the quality of the studies it is summarizing.
• An incomplete search of the literature can bias the findings of a meta-analysis.
• Make sure that the meta-analysis quantifies the size of the effect in units that you can understand.

Subgroup analysis and sensitivity analysis

Subgroup analysis looks at the results of different subgroups of trials, e.g., by considering trials on adults and children separately. This should be planned at the protocol stage itself which is based on good scientific reasoning and is to be kept to a minimum.

Sensitivity analysis is used to determine how results of a systematic review/meta-analysis change by fiddling with data, for example, what is the implication if the exclusion criteria or excluded unpublished studies or weightings are assigned differently. Thus, after the analysis, if changing makes little or no difference to the overall results, the reviewer's conclusions are robust. If the key findings disappear, then the conclusions need to be expressed more cautiously.

Advantages of Systematic Reviews

Systematic reviews have specific advantages because of using explicit methods which limit bias, draw reliable and accurate conclusions, easily deliver required information to healthcare providers, researchers, and policymakers, help to reduce the time delay in the research discoveries to implementation, improve the generalizability and consistency of results, generation of new hypotheses about subgroups of the study population, and overall they increase precision of the results.[ 18 ]

Limitations in Systematic Reviews/Meta-analysis

As with all research, the value of a systematic review depends on what was done, what was found, and the clarity of reporting. As with other publications, the reporting quality of systematic reviews varies, limiting readers’ ability to assess the strengths and weaknesses of those reviews.[ 5 ]

Even though systematic review and meta-analysis are considered the best evidence for getting a definitive answer to a research question, there are certain inherent flaws associated with it, such as the location and selection of studies, heterogeneity, loss of information on important outcomes, inappropriate subgroup analyses, conflict with new experimental data, and duplication of publication.

Publication Bias

Publication bias results in it being easier to find studies with a “positive” result.[ 19 ] This occurs particularly due to inappropriate sifting of the studies where there is always a tendency towards the studies with positive (significant) outcomes. This effect occurs more commonly in systematic reviews/meta-analysis which need to be eliminated.

The quality of reporting of systematic reviews is still not optimal. In a recent review of 300 systematic reviews, few authors reported assessing possible publication bias even though there is overwhelming evidence both for its existence and its impact on the results of systematic reviews. Even when the possibility of publication bias is assessed, there is no guarantee that systematic reviewers have assessed or interpreted it appropriately.[ 20 ]

To overcome certain limitations mentioned above, the Cochrane reviews are currently reported in a format where at the end of every review, findings are summarized in the author's point of view and also give an overall picture of the outcome by means of plain language summary. This is found to be much helpful to understand the existing evidence about the topic more easily by the reader.

A systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods, and has been conducted according to explicit and reproducible methodology. A meta-analysis is a mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way. Although meta-analysis can increase the precision of a result, it is important to ensure that the methods used for the reviews were valid and reliable.

High-quality systematic reviews and meta-analyses take great care to find all relevant studies, critically assess each study, synthesize the findings from individual studies in an unbiased manner, and present balanced important summary of findings with due consideration of any flaws in the evidence. Systematic review and meta-analysis is a way of summarizing research evidence, which is generally the best form of evidence, and hence positioned at the top of the hierarchy of evidence.

Systematic reviews can be very useful decision-making tools for primary care/family physicians. They objectively summarize large amounts of information, identifying gaps in medical research, and identifying beneficial or harmful interventions which will be useful for clinicians, researchers, and even for public and policymakers.

Source of Support: Nil

Conflict of Interest: None declared.