what is systematic review in research pdf

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

• 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
• 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
• 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
• PMID: 30089228
• DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

• Guidelines as Topic
• Meta-Analysis as Topic*
• Publication Bias
• Review Literature as Topic
• Systematic Reviews as Topic*

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 17 October 2022.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2022, October 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 27 May 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Med Prim Care
• v.2(1); Jan-Mar 2013

Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare

S. gopalakrishnan.

Department of Community Medicine, SRM Medical College, Hospital and Research Centre, Kattankulathur, Tamil Nadu, India

P. Ganeshkumar

Healthcare decisions for individual patients and for public health policies should be informed by the best available research evidence. The practice of evidence-based medicine is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient's values and expectations. Primary care physicians need evidence for both clinical practice and for public health decision making. The evidence comes from good reviews which is a state-of-the-art synthesis of current evidence on a given research question. Given the explosion of medical literature, and the fact that time is always scarce, review articles play a vital role in decision making in evidence-based medical practice. Given that most clinicians and public health professionals do not have the time to track down all the original articles, critically read them, and obtain the evidence they need for their questions, systematic reviews and clinical practice guidelines may be their best source of evidence. Systematic reviews aim to identify, evaluate, and summarize the findings of all relevant individual studies over a health-related issue, thereby making the available evidence more accessible to decision makers. The objective of this article is to introduce the primary care physicians about the concept of systematic reviews and meta-analysis, outlining why they are important, describing their methods and terminologies used, and thereby helping them with the skills to recognize and understand a reliable review which will be helpful for their day-to-day clinical practice and research activities.

Introduction

Evidence-based healthcare is the integration of best research evidence with clinical expertise and patient values. Green denotes, “Using evidence from reliable research, to inform healthcare decisions, has the potential to ensure best practice and reduce variations in healthcare delivery.” However, incorporating research into practice is time consuming, and so we need methods of facilitating easy access to evidence for busy clinicians.[ 1 ] Ganeshkumar et al . mentioned that nearly half of the private practitioners in India were consulting more than 4 h per day in a locality,[ 2 ] which explains the difficulty of them in spending time in searching evidence during consultation. Ideally, clinical decision making ought to be based on the latest evidence available. However, to keep abreast with the continuously increasing number of publications in health research, a primary healthcare professional would need to read an insurmountable number of articles every day, covered in more than 13 million references and over 4800 biomedical and health journals in Medline alone. With the view to address this challenge, the systematic review method was developed. Systematic reviews aim to inform and facilitate this process through research synthesis of multiple studies, enabling increased and efficient access to evidence.[ 1 , 3 , 4 ]

Systematic reviews and meta-analyses have become increasingly important in healthcare settings. Clinicians read them to keep up-to-date with their field and they are often used as a starting point for developing clinical practice guidelines. Granting agencies may require a systematic review to ensure there is justification for further research and some healthcare journals are moving in this direction.[ 5 ]

This article is intended to provide an easy guide to understand the concept of systematic reviews and meta-analysis, which has been prepared with the aim of capacity building for general practitioners and other primary healthcare professionals in research methodology and day-to-day clinical practice.

The purpose of this article is to introduce readers to:

• The two approaches of evaluating all the available evidence on an issue i.e., systematic reviews and meta-analysis,
• Discuss the steps in doing a systematic review,
• Introduce the terms used in systematic reviews and meta-analysis,
• Interpret results of a meta-analysis, and
• The advantages and disadvantages of systematic review and meta-analysis.

Application

What is the effect of antiviral treatment in dengue fever? Most often a primary care physician needs to know convincing answers to questions like this in a primary care setting.

To find out the solutions or answers to a clinical question like this, one has to refer textbooks, ask a colleague, or search electronic database for reports of clinical trials. Doctors need reliable information on such problems and on the effectiveness of large number of therapeutic interventions, but the information sources are too many, i.e., nearly 20,000 journals publishing 2 million articles per year with unclear or confusing results. Because no study, regardless of its type, should be interpreted in isolation, a systematic review is generally the best form of evidence.[ 6 ] So, the preferred method is a good summary of research reports, i.e., systematic reviews and meta-analysis, which will give evidence-based answers to clinical situations.

There are two fundamental categories of research: Primary research and secondary research. Primary research is collecting data directly from patients or population, while secondary research is the analysis of data already collected through primary research. A review is an article that summarizes a number of primary studies and may draw conclusions on the topic of interest which can be traditional (unsystematic) or systematic.

Terminologies

Systematic review.

A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors.[ 7 ] To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful.[ 8 ] Systematic reviews are often called overviews.

The evidence-based practitioner, David Sackett, defines the following terminologies.[ 3 ]

• Review: The general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic.
• Overview: When a review strives to comprehensively identify and track down all the literature on a given topic (also called “systematic literature review”).
• Meta-analysis: A specific statistical strategy for assembling the results of several studies into a single estimate.

Systematic reviews adhere to a strict scientific design based on explicit, pre-specified, and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. Systematic reviews can also demonstrate where knowledge is lacking. This can then be used to guide future research. Systematic reviews are usually carried out in the areas of clinical tests (diagnostic, screening, and prognostic), public health interventions, adverse (harm) effects, economic (cost) evaluations, and how and why interventions work.[ 9 ]

Cochrane reviews

Cochrane reviews are systematic reviews undertaken by members of the Cochrane Collaboration which is an international not-for-profit organization that aims to help people to make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

Cochrane Primary Health Care Field is a systematic review of primary healthcare research on prevention, treatment, rehabilitation, and diagnostic test accuracy. The overall aim and mission of the Primary Health Care Field is to promote the quality, quantity, dissemination, accessibility, applicability, and impact of Cochrane systematic reviews relevant to people who work in primary care and to ensure proper representation in the interests of primary care clinicians and consumers in Cochrane reviews and review groups, and in other entities. This field would serve to coordinate and promote the mission of the Cochrane Collaboration within the primary healthcare disciplines, as well as ensuring that primary care perspectives are adequately represented within the Collaboration.[ 10 ]

Meta-analysis

A meta-analysis is the combination of data from several independent primary studies that address the same question to produce a single estimate like the effect of treatment or risk factor. It is the statistical analysis of a large collection of analysis and results from individual studies for the purpose of integrating the findings.[ 11 ] The term meta-analysis has been used to denote the full range of quantitative methods for research reviews.[ 12 ] Meta-analyses are studies of studies.[ 13 ] Meta-analysis provides a logical framework to a research review where similar measures from comparable studies are listed systematically and the available effect measures are combined wherever possible.[ 14 ]

The fundamental rationale of meta-analysis is that it reduces the quantity of data by summarizing data from multiple resources and helps to plan research as well as to frame guidelines. It also helps to make efficient use of existing data, ensuring generalizability, helping to check consistency of relationships, explaining data inconsistency, and quantifies the data. It helps to improve the precision in estimating the risk by using explicit methods.

Therefore, “systematic review” will refer to the entire process of collecting, reviewing, and presenting all available evidence, while the term “meta-analysis” will refer to the statistical technique involved in extracting and combining data to produce a summary result.[ 15 ]

Steps in doing systematic reviews/meta-analysis

Following are the six fundamental essential steps while doing systematic review and meta-analysis.[ 16 ]

Define the question

This is the most important part of systematic reviews/meta-analysis. The research question for the systematic reviews may be related to a major public health problem or a controversial clinical situation which requires acceptable intervention as a possible solution to the present healthcare need of the community. This step is most important since the remaining steps will be based on this.

Reviewing the literature

This can be done by going through scientific resources such as electronic database, controlled clinical trials registers, other biomedical databases, non-English literatures, “gray literatures” (thesis, internal reports, non–peer-reviewed journals, pharmaceutical industry files), references listed in primary sources, raw data from published trials and other unpublished sources known to experts in the field. Among the available electronic scientific database, the popular ones are PUBMED, MEDLINE, and EMBASE.

Sift the studies to select relevant ones

To select the relevant studies from the searches, we need to sift through the studies thus identified. The first sift is pre-screening, i.e., to decide which studies to retrieve in full, and the second sift is selection which is to look again at these studies and decide which are to be included in the review. The next step is selecting the eligible studies based on similar study designs, year of publication, language, choice among multiple articles, sample size or follow-up issues, similarity of exposure, and or treatment and completeness of information.

It is necessary to ensure that the sifting includes all relevant studies like the unpublished studies (desk drawer problem), studies which came with negative conclusions or were published in non-English journals, and studies with small sample size.

Assess the quality of studies

The steps undertaken in evaluating the study quality are early definition of study quality and criteria, setting up a good scoring system, developing a standard form for assessment, calculating quality for each study, and finally using this for sensitivity analysis.

For example, the quality of a randomized controlled trial can be assessed by finding out the answers to the following questions:

• Was the assignment to the treatment groups really random?
• Was the treatment allocation concealed?
• Were the groups similar at baseline in terms of prognostic factors?
• Were the eligibility criteria specified?
• Were the assessors, the care provider, and the patient blinded?
• Were the point estimates and measure of variability presented for the primary outcome measure?
• Did the analyses include intention-to-treat analysis?

Calculate the outcome measures of each study and combine them

We need a standard measure of outcome which can be applied to each study on the basis of its effect size. Based on their type of outcome, following are the measures of outcome: Studies with binary outcomes (cured/not cured) have odds ratio, risk ratio; studies with continuous outcomes (blood pressure) have means, difference in means, standardized difference in means (effect sizes); and survival or time-to-event data have hazard ratios.

Combining studies

Homogeneity of different studies can be estimated at a glance from a forest plot (explained below). For example, if the lower confidence interval of every trial is below the upper of all the others, i.e., the lines all overlap to some extent, then the trials are homogeneous. If some lines do not overlap at all, these trials may be said to be heterogeneous.

The definitive test for assessing the heterogeneity of studies is a variant of Chi-square test (Mantel–Haenszel test). The final step is calculating the common estimate and its confidence interval with the original data or with the summary statistics from all the studies. The best estimate of treatment effect can be derived from the weighted summary statistics of all studies which will be based on weighting to sample size, standard errors, and other summary statistics. Log scale is used to combine the data to estimate the weighting.

Interpret results: Graph

The results of a meta-analysis are usually presented as a graph called forest plot because the typical forest plots appear as forest of lines. It provides a simple visual presentation of individual studies that went into the meta-analysis at a glance. It shows the variation between the studies and an estimate of the overall result of all the studies together.

Forest plot

Meta-analysis graphs can principally be divided into six columns [ Figure 1 ]. Individual study results are displayed in rows. The first column (“study”) lists the individual study IDs included in the meta-analysis; usually the first author and year are displayed. The second column relates to the intervention groups and the third column to the control groups. The fourth column visually displays the study results. The line in the middle is called “the line of no effect.” The weight (in %) in the fifth column indicates the weighting or influence of the study on the overall results of the meta-analysis of all included studies. The higher the percentage weight, the bigger the box, the more influence the study has on the overall results. The sixth column gives the numerical results for each study (e.g., odds ratio or relative risk and 95% confidence interval), which are identical to the graphical display in the fourth column. The diamond in the last row of the graph illustrates the overall result of the meta-analysis.[ 4 ]

Interpretation of meta-analysis[ 4 ]

Thus, the horizontal lines represent individual studies. Length of line is the confidence interval (usually 95%), squares on the line represent effect size (risk ratio) for the study, with area of the square being the study size (proportional to weight given) and position as point estimate (relative risk) of the study.[ 7 ]

For example, the forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults is shown in Figure 2 .[ 17 ]

Forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults[ 17 ]

The overall effect is shown as diamond where the position toward the center represents pooled point estimate, the width represents estimated 95% confidence interval for all studies, and the black plain line vertically in the middle of plot is the “line of no effect” (e.g., relative risk = 1).

Therefore, when examining the results of a systematic reviews/meta-analysis, the following questions should be kept in mind:

• Heterogeneity among studies may make any pooled estimate meaningless.
• The quality of a meta-analysis cannot be any better than the quality of the studies it is summarizing.
• An incomplete search of the literature can bias the findings of a meta-analysis.
• Make sure that the meta-analysis quantifies the size of the effect in units that you can understand.

Subgroup analysis and sensitivity analysis

Subgroup analysis looks at the results of different subgroups of trials, e.g., by considering trials on adults and children separately. This should be planned at the protocol stage itself which is based on good scientific reasoning and is to be kept to a minimum.

Sensitivity analysis is used to determine how results of a systematic review/meta-analysis change by fiddling with data, for example, what is the implication if the exclusion criteria or excluded unpublished studies or weightings are assigned differently. Thus, after the analysis, if changing makes little or no difference to the overall results, the reviewer's conclusions are robust. If the key findings disappear, then the conclusions need to be expressed more cautiously.

Advantages of Systematic Reviews

Systematic reviews have specific advantages because of using explicit methods which limit bias, draw reliable and accurate conclusions, easily deliver required information to healthcare providers, researchers, and policymakers, help to reduce the time delay in the research discoveries to implementation, improve the generalizability and consistency of results, generation of new hypotheses about subgroups of the study population, and overall they increase precision of the results.[ 18 ]

Limitations in Systematic Reviews/Meta-analysis

As with all research, the value of a systematic review depends on what was done, what was found, and the clarity of reporting. As with other publications, the reporting quality of systematic reviews varies, limiting readers’ ability to assess the strengths and weaknesses of those reviews.[ 5 ]

Even though systematic review and meta-analysis are considered the best evidence for getting a definitive answer to a research question, there are certain inherent flaws associated with it, such as the location and selection of studies, heterogeneity, loss of information on important outcomes, inappropriate subgroup analyses, conflict with new experimental data, and duplication of publication.

Publication Bias

Publication bias results in it being easier to find studies with a “positive” result.[ 19 ] This occurs particularly due to inappropriate sifting of the studies where there is always a tendency towards the studies with positive (significant) outcomes. This effect occurs more commonly in systematic reviews/meta-analysis which need to be eliminated.

The quality of reporting of systematic reviews is still not optimal. In a recent review of 300 systematic reviews, few authors reported assessing possible publication bias even though there is overwhelming evidence both for its existence and its impact on the results of systematic reviews. Even when the possibility of publication bias is assessed, there is no guarantee that systematic reviewers have assessed or interpreted it appropriately.[ 20 ]

To overcome certain limitations mentioned above, the Cochrane reviews are currently reported in a format where at the end of every review, findings are summarized in the author's point of view and also give an overall picture of the outcome by means of plain language summary. This is found to be much helpful to understand the existing evidence about the topic more easily by the reader.

A systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods, and has been conducted according to explicit and reproducible methodology. A meta-analysis is a mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way. Although meta-analysis can increase the precision of a result, it is important to ensure that the methods used for the reviews were valid and reliable.

High-quality systematic reviews and meta-analyses take great care to find all relevant studies, critically assess each study, synthesize the findings from individual studies in an unbiased manner, and present balanced important summary of findings with due consideration of any flaws in the evidence. Systematic review and meta-analysis is a way of summarizing research evidence, which is generally the best form of evidence, and hence positioned at the top of the hierarchy of evidence.

Systematic reviews can be very useful decision-making tools for primary care/family physicians. They objectively summarize large amounts of information, identifying gaps in medical research, and identifying beneficial or harmful interventions which will be useful for clinicians, researchers, and even for public and policymakers.

Source of Support: Nil

Conflict of Interest: None declared.

A systematic exploration of scoping and mapping literature reviews

• Brief Report
• Open access
• Published: 23 May 2024

Cite this article

You have full access to this open access article

• Eirini Christou   ORCID: orcid.org/0000-0001-6928-1013 1 ,
• Antigoni Parmaxi   ORCID: orcid.org/0000-0002-0687-0176 1 &
• Panayiotis Zaphiris   ORCID: orcid.org/0000-0001-8112-5099 1  

Systematic literature mapping can help researchers identify gaps in the research and provide a comprehensive overview of the available evidence. Despite the importance and benefits of conducting systematic scoping and mapping reviews, many researchers may not be familiar with the methods and best practices for conducting these types of reviews. This paper aims to address this gap by providing a step-by-step guide to conducting a systematic scoping or mapping review, drawing on examples from different fields. This study adopts a systematic literature review approach aiming to identify and present the steps of conducting scoping and mapping literature reviews and serves as a guide on conducting scoping or mapping systematic literature reviews. A number of 90 studies were included in this study. The findings describe the steps to follow when conducting scoping and mapping reviews and suggest the integration of the card sorting method as part of the process. The proposed steps for undertaking scoping and mapping reviews presented in this manuscript, highlight the importance of following a rigorous approach for conducting scoping or mapping reviews.

Similar content being viewed by others

A scoping review on the conduct and reporting of scoping reviews.

How methodological frameworks are being developed: evidence from a scoping review

Literature Reviews: An Overview of Systematic, Integrated, and Scoping Reviews

Avoid common mistakes on your manuscript.

1 Introduction

An essential component of academic research is literature review. A systematic literature review, also known as a systematic review, is a method for locating, assessing, and interpreting all research related to a specific research question, topic, or phenomenon of interest [ 1 ].

Scoping and mapping reviews are variations of systematic literature mapping [ 2 ]. Both mapping and scoping reviews can help researchers to understand the scope and breadth of the literature in a given field, identify gaps in the research, and provide a comprehensive overview of the available evidence. Systematic literature mapping purposely focuses on a narrower but more general academic or policy issue and does not try to synthesize the results of research to address a particular subject. The scoping review is exploratory in nature, whereas the mapping review can be conclusive in describing the available evidence and identifying gaps. Mapping review includes a thorough, systematic search of a wide field. It identifies the body of literature that is currently available on a subject and points out any glaring gaps in the evidence [ 3 ].

1.1 Rationale

Despite the importance and benefits of conducting systematic scoping and mapping reviews, many researchers may not be familiar with the methods and best practices for conducting these types of reviews. This paper aims to address this gap by providing a step-by-step guide to conducting a systematic scoping or mapping review, drawing on examples from different fields.

This study adopts a systematic literature review approach aiming to identify and present the differences and the steps of conducting scoping and mapping literature review. The paper provides practical guidance on how to address common challenges in conducting systematic scoping or mapping reviews, such as dealing with the volume of studies identified, managing the data extraction and synthesis process, and ensuring rigor and reproducibility in the review methodology. The main research questions that guide this study are:

RQ1: What is a systematic scoping review and how is it conducted?

RQ2: What is a systematic mapping review and how is it conducted?

RQ3: What are the main differences between systematic scoping and systematic mapping reviews?

Overall, this paper will be a valuable resource for researchers who are interested in conducting a systematic scoping or mapping review. By providing clear guidance and practical examples, the paper aims to promote best practices in systematic scoping and mapping review methodology. The study is organized as follows: The following section presents the methodology of the study, followed by the results showing the process of the scoping and mapping literature review and presenting some examples. Finally, suggestions on how to plan and perform a quality scoping and mapping review are presented.

2 Methodology

The methodology of this paper was adopted by Xiao and Watson [ 4 ].

2.1 Literature search

The search was conducted in two well-known online databases, Web of Science and EBSCOHost, across various disciplines. The searched terms combined keywords related to the performance of scoping and mapping literature review, such as “systematic literature review”, “methodology”, “map”, “mapping” and “scoping”. The title of each manuscript was used to determine its initial relevance. If the content of the title suggested that it would explain the method of the literature review process, we obtained the full reference, which included the author, year, title, and abstract, for additional analysis.

2.2 Initial search results

The query string used for the database search is the following: systematic literature review AND methodology AND (“map” OR “mapping” OR “scoping”). Abstract search was conducted in both databases for the last 10 years (2013–2022). A search on EBSCOHost revealed 643 results of which 291 were duplicated and automatically removed. After applying the database filters to limit the articles to peer-reviewed academic journal articles written in English, a number of 102 papers were excluded. Additional 109 papers were duplicated and removed manually. After an initial screening of the titles, a total of 13 studies were identified as relevant to the methodology of the scoping and mapping literature review. A search on Web of Science, revealed 888 results of which 9 were duplicate and removed, and 157 were found to be related to the methodology of scoping or mapping literature reviews after the first title screening. Last search was conducted on the 2nd of November 2022. Both sources revealed 161 related studies, excluding 9 duplicates that were removed.

2.3 Inclusion and exclusion criteria

Only studies that provide instructions on how to perform a scoping or mapping review were included in this paper. Reviews of the literature on certain subjects and in languages other than English were excluded. The study is limited to papers published within the last 10 years, aiming to collect recent information for performing scoping or mapping reviews. Inclusion and exclusion criteria can be found in Table  1 .

2.4 Screening

To further assess the 161 studies’ applicability to the study topic, their abstracts were reviewed. The manuscripts were evaluated independently and in parallel by two researchers. The researchers’ differing opinions were discussed and settled. Then the full-text of a total of 20 studies was acquired for quality evaluation.

2.5 Eligibility and quality evaluation

To further assess the quality and relevance of the studies, the full-text papers were reviewed. Journal articles and books published by prominent publishers were included in the review as they contained high-quality research. Because there is no peer review procedure, the majority of technical reports and online presentations were excluded.

Two researchers worked independently and simultaneously on evaluating eligibility and quality. Any disagreements between them were discussed and resolved. A total of ten (10) studies were excluded after careful review: one study was excluded because it lacked instructions on how the scoping or mapping review methodology was conducted, three studies were excluded because the methodology was not related to scoping or mapping review, while five studies were disregarded because they focused on a particular subject. One of the studies’ full text couldn’t be accessed. This resulted in ten (10) eligible for full-text analysis.

2.6 Iterations

Through backward and forward searching, additional 18 studies were discovered, from which only 10 met the inclusion criteria. The forward and backward search was also used to find manuscripts that applied scoping or mapping literature review methodology. After finding the article that established the scoping or mapping review methodology, articles that had cited the methodology paper to find instances of best practices in different fields were searched. Following consideration of examples’ adherence to the methodology, preference was given to planning-related articles. In total, 90 studies were analyzed in this study, i.e. 10 methodological papers that describe the application of scoping or mapping review, as well as 80 papers that demonstrate the application of the scoping and mapping methodology in different fields, that are used as examples. The PRISMA flow diagram (see Fig.  1 ) depicts the process of the search strategy [ 5 ].

PRISMA flow diagram

2.7 Extraction and analysis of data

Data were extracted in the process of scoping literature reviews, including information with regards to formulating the problem, establishing and validating the review procedure, searching the literature, screening for inclusion, evaluating quality, extracting data, analyzing and synthesizing data, and reporting the findings (Xiao & Watson, 2019). NVivo software was used for all data extraction and coding procedures. Initially, two researchers each took information from articles for cross-checking. The two researchers reached an agreement on what to extract from the publications after reviewing a few articles together. Then the first author classified the data based on the research questions.

In this section we present the findings of our review.

3.1 Defining “Scoping” and “mapping” review

According to [ 2 ], scoping and mapping reviews are variations of systematic literature mapping that focus on narrower but more general academic or policy issues. A scoping review is exploratory in nature, seeking to identify the nature and extent of research on a particular topic, and can be used to identify gaps in the literature. An example of a research question suitable for a scoping review is “What engagement strategies do educators use in classroom settings to facilitate teaching and learning of diverse students in undergraduate nursing programs?“ [ 6 ]. A mapping review, on the other hand, is a thorough and systematic search of a wide field of literature that aims to identify the body of literature currently available on a subject and point out any glaring gaps in the evidence. An example of a research question suitable for a mapping review is “What are the currently available animal models for cystic fibrosis” [ 3 ]. Overall, each type of review has its own specific aims and can be useful for different types of research questions.

3.1.1 Defining scoping review

There is no single definition for scoping reviews in the literature. According to [ 7 ], scoping review is a type of knowledge synthesis that uses a systematic process to map the evidence on a subject and identify key ideas, theories, sources, and knowledge gaps. The goal of a scoping review is to include all relevant information that is available, including ‘grey’ literature, which includes unpublished research findings, therefore including all available literature and evidence, but the reviewers can decide what type of publications they would like to include [ 8 , 9 , 10 , 11 ].

Scoping review process is sometimes used as a preliminary step before a systematic literature review, in cases where the topic or research area in focus has not yet been extensively reviewed or is complicated or heterogeneous in nature and the types of evidence available remain unclear [ 3 ]. For example, while a scoping review might serve as the foundation for a full systematic review, it does not aim to evaluate the quality of the evidence like systematic reviews do [ 8 ]. Moreover, scoping review is also referred to as a “pilot study” [ 12 ], that can be used as a “trial run” of the entire systematic map; it helps to mold the intended approach for the review and inform the protocol development.

Rapid and scoping meta-reviews were also referred as types of scoping reviews. A “rapid review” is a particular kind of scoping review, which aims to provide an answer to a particular query and can shorten the process compared to a full systematic review [ 3 ]. The “scoping meta-review” (SMR) is a scoping evaluation of systematic reviews that offers researchers a flexible framework for field mapping and a way to condense pertinent research activities and findings, similar to a scoping review [ 13 ].

Almost all of the scoping studies identified in the corpus, draw from previews scoping review frameworks, such as the one proposed by [ 14 , 15 ] and the authors’ manual provided by the Joanna Briggs Institute [ 11 , 16 , 17 , 18 ].

3.1.2 Defining mapping review

A mapping review, also referred to as a “systematic map”, is “a high-level review with a broad research question” [ 3 ](p.133). The mapping review includes a thorough, systematic search of a wide field. It identifies the body of literature that is currently available on a subject and points out any gaps in the evidence. The mapping review can be conclusive in describing the available evidence and identifying gaps, whereas the scoping review is exploratory in nature [ 3 ].

The term “mapping” is used to describe the process of synthesizing and representing the literature numerically and thematically in tables, figures, or graphical representations, which can be thought of as the review output. Mapping enables researchers to pinpoint potential areas for further study as well as gaps in the literature [ 19 ].

Systematic mapping uses the same strict procedures as systematic reviews do. However, systematic mapping can be used to address open or closed-framed questions on broad or specific topics, because it is not constrained by the requirement to include fully specified and defined key elements [ 12 ]. Systematic mapping is especially useful for broad, multifaceted questions about an interesting topic that might not be appropriate for systematic review because they involve a variety of interventions, populations, or outcomes, or because they draw on evidence that is not just from primary research [ 12 ].

3.2 Process of conducting mapping and scoping reviews

As noted in the previous sections, mapping reviews and scoping reviews both aim to provide a broad overview of the literature, but the former focuses on the scope of the literature while the latter focuses on the nature and extent of available evidence on a specific research question or topic. In understanding the process for conducting mapping and scoping reviews, we adopted the eight steps proposed by Xiao and Watson [ 4 ] that are common for all types of reviews: (1) Formulate the problem; (2) Establishing and validating the review procedure; (3) Searching the literature; (4) Screening for inclusion; (5) Evaluating quality; (6) Extracting data; (7) Analyzing and synthesizing data; (8) Reporting the findings. The steps are explained in detail below and describe the methodology for both scoping and mapping reviews, distinguishing their differences where applicable. A summary of the review types along with their characteristics and steps as identified from the literature are presented in Table  2 .

3.2.1 Step 1 formulate the problem

The first step for undertaking a mapping or a scoping review is to formulate the problem by setting the research question that should be investigated, taking into account the topic’s scope [ 12 ]. It is important to clearly state the review objectives and specific review questions for the scoping review. The objectives should indicate what the scoping review is trying to achieve [ 10 , 20 ].

In mapping reviews, it can be helpful to create a conceptual framework or model (visual or textual) to describe what will be explored by the map when determining the mapping review’s scope. It should also be determined whether the topic’s scope is broad, specific, or likely to be supported by a substantial body of evidence [ 12 ].

3.2.1.1 Defining the research question(s)

Prior to beginning their search and paper selection process, the authors should typically define their research question(s) [ 3 ]. There are specific formats that are recommended for structuring the research question(s), as well as the exclusion and inclusion criteria of mapping and scoping reviews [ 21 ] (see Table  3 ).

PCC (Population, Concept, and Context) and PICO format (Population, Intervention, Comparator and Outcome) are often used in scoping and mapping reviews. It is recommended that research questions for scoping reviews follow the PCC format and include all of its components [ 17 , 18 , 21 ]. Information about the participants (e.g. age), the principal idea or “concept,” and the setting of the review, should all be included in the research question. The context should be made explicit and may take into account geographical or locational considerations, cultural considerations, and particular racial or gender-based concerns [ 10 ].

Researchers use the PCC format in order to determine the eligibility of their research questions, as well as to define their inclusion criteria (e.g [ 22 , 23 , 24 , 25 , 26 ]). Most scoping reviews have a single main question, but some of them are better served by one or more sub-questions that focus on specific PCC characteristics [ 8 , 18 ].

3.2.2 Step 2. Establishing and validating the review procedure

A protocol is crucial for scoping and mapping reviews because it pre-defines the scoping review’s goals and procedures [ 11 , 17 , 18 , 20 ], it clearly states all methodological decisions since the very beginning [ 2 ], and it also specifies the strategy to be used at each stage of the review process [ 12 ]. Similar to all systematic reviews, scoping reviews start with the creation of an a-priori protocol that includes inclusion and exclusion criteria that are directly related to the review’s objectives and questions [ 7 , 11 , 17 , 18 , 20 ]. In order to decrease study duplication and improve data reporting transparency, the use of formalized, registered protocols is suggested [ 18 , 19 ]. The international prospective register of systematic reviews, known as PROSPERO, states that scoping reviews (and literature reviews) are currently ineligible for registration in the database. While this could change in the future, scoping reviews can currently be registered with the Open Science Framework ( https://osf.io/ ) or Figshare ( https://figshare.com/ ), and their protocols can be published in select publications, including the JBI Evidence Synthesis [ 18 ].

Scoping and mapping reviews should require at least two reviewers in order to minimize reporting bias, as well as to ensure consistency and clarity [ 3 , 16 , 17 , 18 ]. The reviewers should include a plan for the results presentation during the protocol development, such as a draft chart or table that could be improved at the end when the reviewers become more familiar with the information they have included in the review [ 17 , 18 ].

3.2.3 Step 3. Searching the literature

Searching the literature requires to prepare a search strategy, decide on search terms, search databases or journals, and perform a manual search [ 27 ]. For example, deciding on search terms, can follow an iterative process that is further explained in the sub-section below. Thinking about searching in terms of broader to narrower strategies is helpful. Fewer databases and/or journals will be checked out in narrower searches (search only in the title, keywords, and abstract fields), which are frequently used in scoping reviews, while multiple databases can be checked for mapping reviews [ 2 ].

Search strategy

Mapping and scoping review search should aim to be as thorough as possible [ 12 ] to find both published and unpublished evidence. An inclusive approach is frequently preferred for scoping reviews to prevent potential omission of crucial information [ 10 , 17 , 18 ].

According to JBI, there should be a three-step search process for scoping reviews [ 10 , 17 , 18 ]. The first step is a quick search of at least two databases followed by a text word check of the article’s title, abstract, and body of text that are then analyzed. All determined index terms and keywords are used in the second stage across all included databases. In the third stage, additional studies should be looked up in the identified reports and articles’ reference lists [ 10 , 11 , 18 ]. The reviewers may look solely at the reference lists of the studies that were chosen from the full-text and/or included in the review, or they may look at the reference lists of all identified studies. In any case, it needs to be made very clear which group of studies will be looked at [ 8 , 11 , 18 ]. As reviewers gain more familiarity with the body of available evidence, new keywords, sources, and possibly helpful search terms may be found and incorporated into the search strategy, hence the search for a scoping review may be quite iterative. If so, it is crucial that the entire search process and the outcomes are open to scrutiny and audit [ 11 , 18 ].

In the same line, it is recommended for mapping reviews to search multiple databases [ 2 ] in all pertinent searchable fields (e.g., title, abstract, keywords, etc.) [ 3 ]. Thematic keywords, along with all of their synonyms and regional/temporal variations, are joined together to form Boolean strings using Boolean signs. Building looser, multiple Boolean strings instead of well-targeted ones (for example, using OR instead of AND, NOT, and exact phrases, respectively) is preferable. The latter runs the risk of omitting crucial references, whereas the former may return a sizable sample of sparsely relevant references [ 2 ]. Focusing the search on a specific component and then filtering all the results can be more effective for mapping reviews [ 3 ].

3.2.4 Step 4. Screening for inclusion

Screening and choosing the studies to be included in a review are the main objectives of this phase. According to [ 27 ], there are two levels of screening. Titles and abstracts are scanned in the first level to limit the range of the studies to be included, while full texts are scanned in the second level to re-examine the relevance of the studies and to settle disagreements between reviewers regarding the study selection. Discussions, meetings, consulting a third reviewer, and determining inter-rater reliability/agreement (using Cohen’s kappa coefficient or intraclass correlation coefficient) are the most typical ways to resolve disagreements. Soaita et al. (2020) [ 2 ] also support that the sample of retrieved references should be ‘cleaned-up’ once it has been finalized and duplicates have been automatically removed.

Different methodological approaches, including primary research articles, summary articles, opinion pieces, and grey literature, can all be included in the literature that scoping reviews identify and analyze [ 7 , 18 , 19 ], but they may also serve as an exclusion criterion [ 2 ]. Peters, Godfrey, et al. (2020) [ 18 ] advice against limiting source inclusion based on language unless there are compelling justifications for doing so (such as practical considerations).

According to the PRISMA extension for scoping reviews (PRISMA-ScR), a description of the study selection process must be provided in both a narrative and flow diagram format. Including the date of the most recent literature search, enables the reader to assess how current the scoping review is [ 7 ].

3.2.4.1 Inclusion and exclusion criteria

Inclusion criteria offer a framework on which the reviewers can decide which sources to include in the scoping review. To ensure transparency and replicability, the exclusion and inclusion criteria need to be documented [ 7 , 8 , 10 , 11 , 17 ]. Authors should specify any limitations by year, language, publication status, or other factors, and explain why each one was put in place [ 7 ].

When it comes to mapping reviews, criteria should be created whenever possible with participation from stakeholders. Depending on the type of research questions, stakeholders may include practitioners, designers, policy makers, scientists and research funding bodies, but attention should be paid to avoid bias [ 12 ].

3.2.5 Step 5. Evaluating quality

Scoping and mapping reviews are not concerned with quality assessment as a criterion for inclusion [ 2 ]. Assessments of reporting quality and bias risk are typically outside the scope. Although it is possible to extract study characteristics that reflect study and reporting quality, bias cannot be assessed against a specific hypothesis if a mapping review is exploratory [ 3 ].

3.2.6 Step 6. Extracting data

The process of data extraction for a scoping review is also known as “charting the results”. A draft charting table or form needs to be created to capture the key details about the relevance of the included studies to the review question, as well as the characteristics of the included studies. The data extraction process can be iterative, with the charting table being constantly updated.

The reviewers should become familiar with the source results and test the extraction form on two or three studies to ensure that all relevant results are extracted [ 7 , 8 , 10 , 11 , 17 , 18 , 28 ]. In order to increase reporting transparency, authors should explain the main revisions with a justification if the charting process was iterative (i.e., the form was continuously updated). If appropriate, details about the procedures used to collect and verify information from the researchers of the included sources of evidence should be provided [ 7 ]. Author(s), year of publication, source origin, country of origin, objectives, purpose, study population, sample size, methodology, intervention type and comparator, concept, duration of the intervention, how outcomes are measured, and key findings that are related to the review question are all types of information that may be extracted [ 7 , 8 , 10 , 11 , 17 ].

When it comes to data extraction for mapping reviews, it is restricted to important study characteristics and outcomes due to the size of a mapping review [ 3 ]. The process of mapping is intended to produce a practical and organized resource that provides enough detail about studies to be helpful in further work [ 12 ].

To move beyond a straightforward list of citations, it is crucial to maintain a high level of clarity throughout any databases that are created. Studies that are discussed in several papers or that seem to be connected should be marked in the database. In the future, this helps prevent the double counting of research findings in syntheses that might overlook connections between study lines in the databases [ 12 ].

Aiming to capture the key characteristics of the included studies in the scoping and mapping reviews, we suggest the use of a guiding table for extracting data (see Supplementary_Material_1_Guiding_Table).

3.2.7 Step 7. Analyzing and synthesizing data

Authors may extract results and map descriptively. Simple frequency counts of concepts, populations, characteristics, or other fields of data will suffice for many scoping reviews [ 17 , 18 ]. In-depth analysis of quantitative data is not typically required in scoping reviews, although in some cases the authors may take into consideration a more advanced analysis depending on the nature and purpose of their review. A meta-analysis or interpretive qualitative analysis is probably not necessary in scoping reviews [ 17 , 18 ].

When it comes to mapping reviews, no results synthesis is carried out [ 12 ]. Different analytical approaches can be used to map chronological, geographical, conceptual, and thematic trends, which include some form of coding, once the sample has been limited to the pertinent references [ 2 ]. It is possible to identify correlations, trends, gaps, and clusters using simple numerical accounts of frequencies in each category (for example, the number of studies looking at a specific species) and more complex cross-tabulations (for instance, the number of studies looking at the effectiveness of a specific intervention, in a particular farming system, for a named species). Users have access to the map and can query it to find information pertaining to any chosen combinations of the subsets of the meta-data [ 12 ].

3.2.8 Step 8. Reporting the findings

Authors should specify exactly how the evidence will be presented, whether it be in a narrative format, table, or visual representation, such as a map or diagram [ 7 ].

In scoping reviews, a summary of all the relevant information gathered can be presented [ 8 ] using a logical and descriptive summary of the findings based on the research questions [ 10 , 11 , 17 ]. The distribution of studies by year or period of publication, countries of origin, field of intervention, and research methodologies, may be displayed in the tables and charts accompanied with a narrative summary that explains how the results relate to the review’s objectives [ 7 , 11 , 17 , 18 ].

The conclusions should be consistent with the review objective or question based on the findings of the scoping review [ 10 ]. Following the conclusions, specific recommendations for future research based on gaps in knowledge identified by the review results can be presented. Because of the lack of a methodological quality appraisal, recommendations for practice may be unable to be developed; however, suggestions based on the conclusions may be made [ 10 ].

A scoping review’s results section should include a PRISMA flow diagram and details the outcomes of the search strategy and selection procedure [ 7 , 17 ] outlining the grounds for exclusion at the full-text level of screening [ 7 ]. For example, a study [ 29 ] used the PRISMA-ScR extension for scoping reviews to ensure all important sections have been covered in their review.

Mapping reviews may place more emphasis on describing the evidence. The use of pivot tables and pivot charts is helpful for quickly visualizing the amount (and quality, if it is measured) of evidence across a variety of meta-data variables [ 12 ]. Such visualizations can display the quantity of research, the conclusions of a critical appraisal, the sample size across nations, outcomes, populations, or variables. These visualizations can contain categorical variables as additional dimensions. The geographic distribution of study effort and type may be particularly important in mapping reviews with a global or large-scale reach [ 12 ].

4 Discussion

This systematic literature review aimed to describe the process of conducting mapping and scoping literature reviews. In summary, the main difference between the two types of reviews is in their focus and scope. Mapping reviews provide a comprehensive overview of the literature while scoping reviews identify gaps and inconsistencies in the literature and outline potential areas for future research.

A lot of the methodological papers included in this systematic literature review (e.g [ 10 , 19 , 28 ]), referred to the “consultation process” as an additional, optional step that has been suggested by [ 14 ]. In this stage, subject experts or potential review users like practitioners, consumers, and policymakers may be consulted [ 8 ]. Researchers argue that this step should be mandatory [ 15 , 28 ]. In agreement with Levac et al.’s [ 15 ] choice, Daudt et al. [ 28 ] encourage the use of the consultation stage whenever it is practical because it adds richness to the entire research process and, consequently, the findings. Despite the fact that stakeholder consultations can make scoping review planning and execution more difficult, they guarantee that the findings are pertinent to educational practice and/or policies [ 19 ].

Scoping and mapping reviews should require more than one author to eliminate bias and ensure their quality. The card-sorting technique is suggested to be employed within the review process as a means for resolving discrepancies between the stakeholders and come to an agreement on the categorization and evaluation of the data to be included. Other studies (e.g [ 30 , 31 , 32 , 33 ]), propose the card-sorting technique as a method for resolving disagreements between people’s disparities, as well as to evaluate and verify extracted themes from datasets. Card sorting is a quick and reliable sorting method that finds patterns in how users would expect to find content or functionality. Due to the patterns and insights it exposes about how people organize and categorize content, card sorting is a successful approach for resolving categorization disagreements [ 34 ]. According to Wood and Wood [ 35 ], the majority of card sorting projects involve an open sort, where participants receive a list of items and are asked to organize them in the most appropriate way. However, in some cases, a pre-existing set of categories is given to the participants, the so-called closed card sorting project. This assumes that the existing categories are already well-organized, and the goal is to make minor adjustments. Wood and Wood [ 35 ], suggest that it’s best to start with an open sort and analyze the data before conducting a closed sort for validation. If a closed sort is necessary, it should be kept simple, and the results may not be optimal. For example, in a study [ 30 ] that aimed to review the use of makerspaces for educational purposes, the card sorting technique was used for the development of the coding scheme. A three-member academic committee, consisting of three professors took part in the card sorting exercise where they went through the abstracts of the relevant papers and were asked to categorize each manuscript after discussion. They then categorized the manuscripts in the three major themes and 11 subcategories that emerged during the card sorting exercise [ 30 ]. Similarly, the authors of [ 31 ] employed the card sorting technique in their research in order to agree on the main categorization and sub-categorization of the articles identified for inclusion in their review. Card sorting can be integrated as an additional step when conducting scoping and mapping reviews, as it provides useful insights from the experts’ perspective and makes the mapping process more inclusive (see Fig.  2 ).

Proposed steps for conducting scoping and mapping reviews

5 Conclusion

Scoping and mapping reviews need a methodological framework that is rigorous, consistent, and transparent, so that the results can be trusted and the review replicated. This provides enough information for the readers to evaluate the review’s accuracy, relevance, and thoroughness [ 8 ]. Scoping reviews should be carried out in accordance with established methodological guidance and reported using reporting standards (like PRISMA-ScR) guidelines [ 36 ]. The proposed steps for undertaking scoping and mapping reviews presented in this manuscript, highlight the importance of following a rigorous approach for conducting scoping or mapping reviews. Overall, this paper is a valuable resource for researchers who are interested in conducting a systematic scoping or mapping review in different fields and are looking to apply these review methods to their own research questions.

5.1 Limitations and future work

This study does not lack limitations. As specific keywords and specific databases were searched, not all relevant work is included. The study was also limited to the past 10 years, letting out methodologies and frameworks for scoping and mapping literature reviews that were not published within the specific timeframe. The fact that the number of methodological papers identified for inclusion are limited to ten, makes it difficult to clarify the differences between mapping and scoping reviews. Therefore, further research is encouraged in order to clarify and verify the differences and similarities between the two. The application of the proposed process for conducting systematic scoping and mapping reviews on specific topics will verify the process.

Data availability

The data that support the findings of this study are available from the corresponding author, [EC], upon reasonable request.

Kitchenham, B., Charters, S.: Guidelines for performing Systematic Literature Reviews in Software Engineering, (2007). https://doi.org/10.1109/ACCESS.2016.2603219

Soaita, A.M., Serin, B., Preece, J.: A methodological quest for systematic literature mapping. Int. J. Hous. Policy. 20 , 320–343 (2020). https://doi.org/10.1080/19491247.2019.1649040

Article   Google Scholar  

Leenaars, C., Tsaioun, K., Stafleu, F., Rooney, K., Meijboom, F., Ritskes-Hoitinga, M., Bleich, A.: Reviewing the animal literature: How to describe and choose between different types of literature reviews. Lab. Anim. 55 , 129–141 (2021). https://doi.org/10.1177/0023677220968599

Xiao, Y., Watson, M.: Guidance on conducting a systematic literature review. J. Plan. Educ. Res. 39 , 93–112 (2019). https://doi.org/10.1177/0739456X17723971

Haddaway, N.R., Page, M.J., Pritchard, C.C., McGuinness, L.A.: PRISMA: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis, Campbell Syst. Rev. 18 (2022). (2020). https://doi.org/10.1002/CL2.1230

Iduye, D., Vukic, A., Waldron, I., Price, S., Sheffer, C., McKibbon, S., Dorey, R., Yu, Z.: Educators’ strategies for engaging diverse students in undergraduate nursing education programs: A scoping review protocol. JBI Evid. Synth. 19 , 1178–1185 (2021). https://doi.org/10.11124/JBIES-20-00039

Tricco, A.C., Lillie, E., Zarin, W., O’Brien, K.K., Colquhoun, H., Levac, D., Moher, D., Peters, M.D.J., Horsley, T., Weeks, L., Hempel, S., Akl, E.A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M.G., Garritty, C., Lewin, S., Godfrey, C.M., MacDonald, M.T., Langlois, E.V., Soares-Weiser, K., Moriarty, J., Clifford, T., Tunçalp, Ö., Straus, S.E.: PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann. Intern. Med. 169 , 467–473 (2018). https://doi.org/10.7326/M18-0850/SUPPL_FILE/M18-0850_SUPPLEMENT.PDF

McKinstry, C., Brown, T., Gustafsson, L.: Scoping reviews in occupational therapy: The what, why, and how to. Aust Occup. Ther. J. 61 , 58–66 (2014). https://doi.org/10.1111/1440-1630.12080 WE - Science Citation Index Expanded (SCI-EXPANDED)

Peterson, J., Pearce, P.F., Ferguson, L.A., Langford, C.A.: Understanding scoping reviews: Definition, purpose, and process. J. Am. Assoc. Nurse Pract. 29 , 12–16 (2017). https://doi.org/10.1002/2327-6924.12380

Khalil, H., Peters, M., Godfrey, C.M., McInerney, P., Soares, C.B., Parker, D.: An evidence-based Approach to Scoping Reviews. Worldviews Evidence-Based Nurs. 13 , 118–123 (2016)

Peters, M.D.J., Godfrey, C., McInerney, P., Munn, Z., Trico, A., Khalil, H.: Chap. 11: Scoping reviews, in: JBI Man. Evid. Synth. JBI. (2017). https://doi.org/10.46658/JBIMES-20-12

James, K.L., Randall, N.P., Haddaway, N.R.: A methodology for systematic mapping in environmental sciences. Environ. Evid. 5 , 1–13 (2016). https://doi.org/10.1186/s13750-016-0059-6

Sarrami-Foroushani, P., Travaglia, J., Debono, D., Clay-Williams, R., Braithwaite, J.: Scoping Meta-review: Introducing a New Methodology. Clin. Transl Sci. 8 , 77–81 (2015). https://doi.org/10.1111/CTS.12188

Arksey, H., O’malley, L.: Scoping studies: Towards a methodological framework. Int. J. Soc. Res. Methodol. 8 , 19–32 (2005). https://doi.org/10.1080/1364557032000119616

Levac, D., Colquhoun, H., O’Brien, K.K.: Scoping studies: Advancing the methodology. Implement. Sci. 5 , 1–9 (2010). https://doi.org/10.1017/cbo9780511814563.003

Peters, M., Godfrey, C., McInerney, P., Soares, C., Parker, K.H.D.: Chap. 11: Scoping reviews. In: Aromataris, E., Munn, Z. (eds.) JBI Rev. Man. JBI, Adelaide (2015)

Google Scholar  

Peters, M.D.J., Marnie, C., Tricco, A.C., Pollock, D., Munn, Z., Alexander, L., McInerney, P., Godfrey, C.M., Khalil, H.: Updated methodological guidance for the conduct of scoping reviews. JBI Evid. Synth. 18 , 2119–2126 (2020). https://doi.org/10.11124/JBIES-20-00167 WE - Emerging Sources Citation Index (ESCI)

Peters, M.D.J., Godfrey, C., McInerney, P., Munn, Z., Trico, A., Khalil, H.: Chap. 11: Scoping Reviews, JBI Man. Evid. Synth. (2020). https://doi.org/10.46658/JBIMES-20-12

Thomas, A., Lubarsky, S., Durning, S.J., Young, M.E.: Knowledge syntheses in medical education: Demystifying scoping reviews. Acad. Med. 92 , 161–166 (2017). https://doi.org/10.1097/ACM.0000000000001452

Peters, M.D.J., Godfrey, C.M., Khalil, H., McInerney, P., Parker, D., Soares, C.B.: Guidance for conducting systematic scoping reviews. Int. J. Evid. Based Healthc. 13 , 141–146 (2015). https://doi.org/10.1097/XEB.0000000000000050

Sager, M., Pistone, I.: Mismatches in the production of a scoping review: Highlighting the interplay of (in)formalities. J. Eval Clin. Pract. 25 , 930–937 (2019). https://doi.org/10.1111/jep.13251

Balsiger, F., Wagner, B., Jende, J.M.E., Marty, B., Bendszus, M., Scheidegger, O., Kurz, F.T.: Methodologies and MR parameters in quantitative magnetic resonance neurography: A scoping review protocol. METHODS Protoc. 5 (2022). https://doi.org/10.3390/mps5030039 WE - Emerging Sources Citation Index (ESCI)

Kesztyus, D., Brucher, S., Kesztyus, T.: Use of infrared thermography in medical diagnostics: A scoping review protocol. BMJ Open. 12 (2022). https://doi.org/10.1136/bmjopen-2021-059833 WE - Science Citation Index Expanded (SCI-EXPANDED)

Olaniyi, A.A., Ncama, B.P., Amod, H.: Mapping evidence of neonatal resuscitation training on the practices of Unskilled Birth attendants in Low-Resource Countries: Protocol for a scoping review. JMIR Res. Protoc. 10 (2021). https://doi.org/10.2196/18935 WE - Emerging Sources Citation Index (ESCI)

Rosca, E.C., Tudor, R., Cornea, A., Simu, M.: Parkinson’s Disease in Romania: A scoping review protocol. BRAIN Sci. 11 (2021). https://doi.org/10.3390/brainsci11020251 WE - Science Citation Index Expanded (SCI-EXPANDED)

Walker, K., Asoodar, M., Rudolph, J., Meguerdichian, M., Yusaf, T., Campbell-Taylor, K., van Merrienboer, J.: Optimising expert dyad performance in acute care settings: A scoping review protocol. BMJ Open. 11 (2021). https://doi.org/10.1136/bmjopen-2020-047260 WE - Science Citation Index Expanded (SCI-EXPANDED)

Chong, S.W., Lin, T.J., Chen, Y.: A methodological review of systematic literature reviews in higher education: Heterogeneity and homogeneity. Educ. Res. Rev. 35 (2022). https://doi.org/10.1016/j.edurev.2021.100426

Daudt, H.M.L., Van Mossel, C., Scott, S.J.: Enhancing the scoping study methodology: A large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med. Res. Methodol. 13 , 1–9 (2013). https://doi.org/10.1186/1471-2288-13-48

Qiu, Y.J., Osadnik, C.R., Team, V., Weller, C.D.: Physical activity as an Adjunct to Compression Therapy on Healing outcomes and recurrence in patients with venous Leg ulcers: A scoping review protocol. Front. Med. 8 (2021). https://doi.org/10.3389/fmed.2021.614059 WE - Science Citation Index Expanded (SCI-EXPANDED)

Konstantinou, D., Parmaxi, A., Zaphiris, P.: Mapping research directions on makerspaces in education, EMI. Educ. Media Int. 58 , 223–247 (2021). https://doi.org/10.1080/09523987.2021.1976826

Pallaris, G., Zaphiris, P., Parmaxi, A.: Mapping the landscape of Makerspaces in higher education: An inventory of research findings. Interact. Technol. Smart Educ. (2022). https://doi.org/10.1108/ITSE-01-2022-0013

Parmaxi, A., Zaphiris, P., Papadima-Sophocleous, S., Ioannou, A.: Mapping the landscape of computer-assisted language learning: An inventory of research. Interact. Technol. Smart Educ. 10 (2013). https://doi.org/10.1108/ITSE-02-2013-0004

Parmaxi, A., Zaphiris, P.: Computer-mediated communication in computer-assisted language learning: Implications for culture-centered design. Univers. Access. Inf. Soc. 15 , 169–177 (2016). https://doi.org/10.1007/S10209-015-0405-4/TABLES/3

Morville, P., Rosenfeld, L.: Information architecture for the World Wide Web: Designing large-scale web sites, (2006). https://books.google.com/books?hl=en&lr=&id=2d2Ry2hZc2MC&oi=fnd&pg=PR5&dq=Morville+%26+Rosenfeld,+2007&ots=opbfyu0ODb&sig=cAyUSw0mmdVRYf5ARSNp6DaYYLA (accessed January 17, 2024)

Wood, J.R., Wood, L.E.: Card sorting: Current practices and beyond. J. Usability Stud. 4 , 1–6 (2008). http://usabilityprofessionals.org/upa_publications/jus/2008november/JUS_Wood_Nov2008.pdf

Munn, Z., Pollock, D., Khalil, H., Alexander, L., Mclnerney, P., Godfrey, C.M., Peters, M., Tricco, A.C.: What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid. Synth. 20 , 950–952 (2022). https://doi.org/10.11124/JBIES-21-00483 WE - Emerging Sources Citation Index (ESCI)

Download references

Acknowledgements

This work has been supported through funding from the Cyprus University of Technology.

Open access funding provided by the Cyprus Libraries Consortium (CLC).

Author information

Authors and affiliations.

Cyprus University of Technology, Limassol, Cyprus

Eirini Christou, Antigoni Parmaxi & Panayiotis Zaphiris

You can also search for this author in PubMed   Google Scholar

Contributions

E.C. and A.P. wrote the main manuscript text. All authors contributed and agreed on the methodology to be followed. A.P. and E.C. screened and decided on the papers to be included in the study. All authors reviewed the manuscript.

Corresponding author

Correspondence to Eirini Christou .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Christou, E., Parmaxi, A. & Zaphiris, P. A systematic exploration of scoping and mapping literature reviews. Univ Access Inf Soc (2024). https://doi.org/10.1007/s10209-024-01120-3

Download citation

Accepted : 15 May 2024

Published : 23 May 2024

DOI : https://doi.org/10.1007/s10209-024-01120-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Scoping review
• Mapping review
• Review methodology
• Systematic literature mapping
• Card sorting
• Find a journal
• Publish with us
• Track your research

Help | Advanced Search

Computer Science > Computation and Language

Title: text generation: a systematic literature review of tasks, evaluation, and challenges.

Abstract: Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review comprising 244 selected papers between 2017 and 2024. This review categorizes works in text generation into five main tasks: open-ended text generation, summarization, translation, paraphrasing, and question answering. For each task, we review their relevant characteristics, sub-tasks, and specific challenges (e.g., missing datasets for multi-document summarization, coherence in story generation, and complex reasoning for question answering). Additionally, we assess current approaches for evaluating text generation systems and ascertain problems with current metrics. Our investigation shows nine prominent challenges common to all tasks and sub-tasks in recent text generation publications: bias, reasoning, hallucinations, misuse, privacy, interpretability, transparency, datasets, and computing. We provide a detailed analysis of these challenges, their potential solutions, and which gaps still require further engagement from the community. This systematic literature review targets two main audiences: early career researchers in natural language processing looking for an overview of the field and promising research directions, as well as experienced researchers seeking a detailed view of tasks, evaluation methodologies, open challenges, and recent mitigation strategies.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .