quantitative literature review definition

Archer Library

Quantitative research: literature review .

• Archer Library This link opens in a new window
• Research Resources handout This link opens in a new window
• Locating Books
• Library eBook Collections This link opens in a new window
• A to Z Database List This link opens in a new window
• Research & Statistics
• Literature Review Resources
• Citations & Reference

Exploring the literature review 

Literature review model: 6 steps.

Adapted from The Literature Review , Machi & McEvoy (2009, p. 13).

Your Literature Review

Step 2: search, boolean search strategies, search limiters, ★ ebsco & google drive.

1. Select a Topic

"All research begins with curiosity" (Machi & McEvoy, 2009, p. 14)

Selection of a topic, and fully defined research interest and question, is supervised (and approved) by your professor. Tips for crafting your topic include:

• Be specific. Take time to define your interest.
• Topic Focus. Fully describe and sufficiently narrow the focus for research.
• Academic Discipline. Learn more about your area of research & refine the scope.
• Avoid Bias. Be aware of bias that you (as a researcher) may have.
• Document your research. Use Google Docs to track your research process.
• Research apps. Consider using Evernote or Zotero to track your research.

Consider Purpose

What will your topic and research address?

In The Literature Review: A Step-by-Step Guide for Students , Ridley presents that literature reviews serve several purposes (2008, p. 16-17).  Included are the following points:

• Historical background for the research;
• Overview of current field provided by "contemporary debates, issues, and questions;"
• Theories and concepts related to your research;
• Introduce "relevant terminology" - or academic language - being used it the field;
• Connect to existing research - does your work "extend or challenge [this] or address a gap;" 
• Provide "supporting evidence for a practical problem or issue" that your research addresses.

★ Schedule a research appointment

At this point in your literature review, take time to meet with a librarian. Why? Understanding the subject terminology used in databases can be challenging. Archer Librarians can help you structure a search, preparing you for step two. How? Contact a librarian directly or use the online form to schedule an appointment. Details are provided in the adjacent Schedule an Appointment box.

2. Search the Literature

Collect & Select Data: Preview, select, and organize

Archer Library is your go-to resource for this step in your literature review process. The literature search will include books and ebooks, scholarly and practitioner journals, theses and dissertations, and indexes. You may also choose to include web sites, blogs, open access resources, and newspapers. This library guide provides access to resources needed to complete a literature review.

Books & eBooks: Archer Library & OhioLINK

Databases: Scholarly & Practitioner Journals

Review the Library Databases tab on this library guide, it provides links to recommended databases for Education & Psychology, Business, and General & Social Sciences.

Expand your journal search; a complete listing of available AU Library and OhioLINK databases is available on the Databases  A to Z list . Search the database by subject, type, name, or do use the search box for a general title search. The A to Z list also includes open access resources and select internet sites.

Databases: Theses & Dissertations

Review the Library Databases tab on this guide, it includes Theses & Dissertation resources. AU library also has AU student authored theses and dissertations available in print, search the library catalog for these titles.

Did you know? If you are looking for particular chapters within a dissertation that is not fully available online, it is possible to submit an ILL article request . Do this instead of requesting the entire dissertation.

Newspapers:  Databases & Internet

Consider current literature in your academic field. AU Library's database collection includes The Chronicle of Higher Education and The Wall Street Journal .  The Internet Resources tab in this guide provides links to newspapers and online journals such as Inside Higher Ed , COABE Journal , and Education Week .

The Chronicle of Higher Education has the nation’s largest newsroom dedicated to covering colleges and universities.  Source of news, information, and jobs for college and university faculty members and administrators

The Chronicle features complete contents of the latest print issue; daily news and advice columns; current job listings; archive of previously published content; discussion forums; and career-building tools such as online CV management and salary databases. Dates covered: 1970-present.

Search Strategies & Boolean Operators

There are three basic boolean operators:  AND, OR, and NOT.

Used with your search terms, boolean operators will either expand or limit results. What purpose do they serve? They help to define the relationship between your search terms. For example, using the operator AND will combine the terms expanding the search. When searching some databases, and Google, the operator AND may be implied.

Overview of boolean terms

About the example: Boolean searches were conducted on November 4, 2019; result numbers may vary at a later date. No additional database limiters were set to further narrow search returns.

Database Search Limiters

Database strategies for targeted search results.

Most databases include limiters, or additional parameters, you may use to strategically focus search results.  EBSCO databases, such as Education Research Complete & Academic Search Complete provide options to:

• Limit results to full text;
• Limit results to scholarly journals, and reference available;
• Select results source type to journals, magazines, conference papers, reviews, and newspapers
• Publication date

Keep in mind that these tools are defined as limiters for a reason; adding them to a search will limit the number of results returned.  This can be a double-edged sword.  How? 

• If limiting results to full-text only, you may miss an important piece of research that could change the direction of your research. Interlibrary loan is available to students, free of charge. Request articles that are not available in full-text; they will be sent to you via email.
• If narrowing publication date, you may eliminate significant historical - or recent - research conducted on your topic.
• Limiting resource type to a specific type of material may cause bias in the research results.

Use limiters with care. When starting a search, consider opting out of limiters until the initial literature screening is complete. The second or third time through your research may be the ideal time to focus on specific time periods or material (scholarly vs newspaper).

★ Truncating Search Terms

Expanding your search term at the root.

Truncating is often referred to as 'wildcard' searching. Databases may have their own specific wildcard elements however, the most commonly used are the asterisk (*) or question mark (?).  When used within your search. they will expand returned results.

Asterisk (*) Wildcard

Using the asterisk wildcard will return varied spellings of the truncated word. In the following example, the search term education was truncated after the letter "t."

Explore these database help pages for additional information on crafting search terms.

• EBSCO Connect: Basic Searching with EBSCO
• EBSCO Connect: Searching with Boolean Operators
• EBSCO Connect: Searching with Wildcards and Truncation Symbols
• ProQuest Help: Search Tips
• ERIC: How does ERIC search work?

★ EBSCO Databases & Google Drive

Tips for saving research directly to Google drive.

Researching in an EBSCO database?

It is possible to save articles (PDF and HTML) and abstracts in EBSCOhost databases directly to Google drive. Select the Google Drive icon, authenticate using a Google account, and an EBSCO folder will be created in your account. This is a great option for managing your research. If documenting your research in a Google Doc, consider linking the information to actual articles saved in drive.

EBSCO Databases & Google Drive

EBSCOHost Databases & Google Drive: Managing your Research

This video features an overview of how to use Google Drive with EBSCO databases to help manage your research. It presents information for connecting an active Google account to EBSCO and steps needed to provide permission for EBSCO to manage a folder in Drive.

About the Video:  Closed captioning is available, select CC from the video menu.  If you need to review a specific area on the video, view on YouTube and expand the video description for access to topic time stamps.  A video transcript is provided below.

• EBSCOhost Databases & Google Scholar

Defining Literature Review

What is a literature review.

A definition from the Online Dictionary for Library and Information Sciences .

A literature review is "a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works" (Reitz, 2014). 

A systemic review is "a literature review focused on a specific research question, which uses explicit methods to minimize bias in the identification, appraisal, selection, and synthesis of all the high-quality evidence pertinent to the question" (Reitz, 2014).

Recommended Reading

About this page

EBSCO Connect [Discovery and Search]. (2022). Searching with boolean operators. Retrieved May, 3, 2022 from https://connect.ebsco.com/s/?language=en_US

EBSCO Connect [Discover and Search]. (2022). Searching with wildcards and truncation symbols. Retrieved May 3, 2022; https://connect.ebsco.com/s/?language=en_US

Machi, L.A. & McEvoy, B.T. (2009). The literature review . Thousand Oaks, CA: Corwin Press: 

Reitz, J.M. (2014). Online dictionary for library and information science. ABC-CLIO, Libraries Unlimited . Retrieved from https://www.abc-clio.com/ODLIS/odlis_A.aspx

Ridley, D. (2008). The literature review: A step-by-step guide for students . Thousand Oaks, CA: Sage Publications, Inc.

Archer Librarians

Schedule an appointment.

Contact a librarian directly (email), or submit a request form. If you have worked with someone before, you can request them on the form.

• ★ Archer Library Help • Online Reqest Form
• Carrie Halquist • Reference & Instruction
• Jessica Byers • Reference & Curation
• Don Reams • Corrections Education & Reference
• Diane Schrecker • Education & Head of the IRC
• Tanaya Silcox • Technical Services & Business
• Sarah Thomas • Acquisitions & ATS Librarian
• << Previous: Research & Statistics
• Next: Literature Review Resources >>
• Last Updated: Jun 27, 2024 11:14 AM
• URL: https://libguides.ashland.edu/quantitative

Archer Library • Ashland University © Copyright 2023. An Equal Opportunity/Equal Access Institution.

University Libraries

• Research Guides
• Blackboard Learn
• Interlibrary Loan
• Study Rooms
• University of Arkansas

Literature Reviews

• Qualitative or Quantitative?
• Getting Started
• Finding articles
• Primary sources? Peer-reviewed?
• Review Articles/ Annual Reviews...?
• Books, ebooks, dissertations, book reviews

Qualitative researchers TEND to:

Researchers using qualitative methods tend to:

• t hink that social sciences cannot be well-studied with the same methods as natural or physical sciences
• feel that human behavior is context-specific; therefore, behavior must be studied holistically, in situ, rather than being manipulated
• employ an 'insider's' perspective; research tends to be personal and thereby more subjective.
• do interviews, focus groups, field research, case studies, and conversational or content analysis.

Image from https://www.editage.com/insights/qualitative-quantitative-or-mixed-methods-a-quick-guide-to-choose-the-right-design-for-your-research?refer-type=infographics

Qualitative Research (an operational definition)

Qualitative Research: an operational description

Purpose : explain; gain insight and understanding of phenomena through intensive collection and study of narrative data

Approach: inductive; value-laden/subjective; holistic, process-oriented

Hypotheses: tentative, evolving; based on the particular study

Lit. Review: limited; may not be exhaustive

Setting: naturalistic, when and as much as possible

Sampling : for the purpose; not necessarily representative; for in-depth understanding

Measurement: narrative; ongoing

Design and Method: flexible, specified only generally; based on non-intervention, minimal disturbance, such as historical, ethnographic, or case studies

Data Collection: document collection, participant observation, informal interviews, field notes

Data Analysis: raw data is words/ ongoing; involves synthesis

Data Interpretation: tentative, reviewed on ongoing basis, speculative

• Qualitative research with more structure and less subjectivity
• Increased application of both strategies to the same study ("mixed methods")
• Evidence-based practice emphasized in more fields (nursing, social work, education, and others).

Some Other Guidelines

• Guide for formatting Graphs and Tables
• Critical Appraisal Checklist for an Article On Qualitative Research

Quantitative researchers TEND to:

Researchers using quantitative methods tend to:

• think that both natural and social sciences strive to explain phenomena with confirmable theories derived from testable assumptions
• attempt to reduce social reality to variables, in the same way as with physical reality
• try to tightly control the variable(s) in question to see how the others are influenced.
• Do experiments, have control groups, use blind or double-blind studies; use measures or instruments.

Quantitative Research (an operational definition)

Quantitative research: an operational description

Purpose: explain, predict or control phenomena through focused collection and analysis of numberical data

Approach: deductive; tries to be value-free/has objectives/ is outcome-oriented

Hypotheses : Specific, testable, and stated prior to study

Lit. Review: extensive; may significantly influence a particular study

Setting: controlled to the degree possible

Sampling: uses largest manageable random/randomized sample, to allow generalization of results to larger populations

Measurement: standardized, numberical; "at the end"

Design and Method: Strongly structured, specified in detail in advance; involves intervention, manipulation and control groups; descriptive, correlational, experimental

Data Collection: via instruments, surveys, experiments, semi-structured formal interviews, tests or questionnaires

Data Analysis: raw data is numbers; at end of study, usually statistical

Data Interpretation: formulated at end of study; stated as a degree of certainty

This page on qualitative and quantitative research has been adapted and expanded from a handout by Suzy Westenkirchner. Used with permission.

Images from https://www.editage.com/insights/qualitative-quantitative-or-mixed-methods-a-quick-guide-to-choose-the-right-design-for-your-research?refer-type=infographics.

• << Previous: Books, ebooks, dissertations, book reviews
• Last Updated: Jan 8, 2024 2:51 PM
• URL: https://uark.libguides.com/litreview
• See us on Instagram
• Follow us on Twitter
• Phone: 479-575-4104

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core Collection This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: May 2, 2024 10:39 AM
• URL: https://libguides.northwestern.edu/literaturereviews

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support
• Artificial Intelligence (AI) Tools

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Meryl Brodsky : Communication and Information Studies

Hannah Chapman Tripp : Biology, Neuroscience

Carolyn Cunningham : Human Development & Family Sciences, Psychology, Sociology

Larayne Dallas : Engineering

Janelle Hedstrom : Special Education, Curriculum & Instruction, Ed Leadership & Policy ​

Susan Macicak : Linguistics

Imelda Vetter : Dell Medical School

For help in other subject areas, please see the guide to library specialists by subject .

Periodically, UT Libraries runs a workshop covering the basics and library support for literature reviews. While we try to offer these once per academic year, we find providing the recording to be helpful to community members who have missed the session. Following is the most recent recording of the workshop, Conducting a Literature Review. To view the recording, a UT login is required.

• October 26, 2022 recording
• Last Updated: Jun 18, 2024 1:00 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Griffith University

Popular sites

Home > Griffith Sciences > School of Environment and Science > Research > Systematic Quantitative Literature Review

Systematic Quantitative Literature Review

School of Environment and Science

• Learning and teaching

A smart and effective method for undertaking literature reviews particularly for research students and others new to a discipline.

Narrative methods that are commonly used in many research theses, rely on the expertise and experience of the author, making them challenging for novices. In contrast, the method we use and recommend involves systematically searching the literature using online database and other sources to find all relevant papers that fit specific criteria (systematically identifying the literature), entering information about each study into a personal database, then compiling tables that summarise the current status of the literature (quantifying the literature). The results are reliable, quantifiable and reproducible.

Using this method, it’s also possible to determine if there are suitable datasets for meta-analysis. By mapping the literature we can not only identify what is known, but also, but where there are gaps: a critical issue in advancing research and designing PhD research programs.

Reliable, quantifiable and reproducible

The method works well for specific topics, but also for summarising diverse inter-disciplinary research. Using this method many of our students and others have gone on to publish their reviews. Importantly for PhD students, the database can be updated during the PhD thesis allowing them to easily identify relevant papers and produce their final thesis without having to re-read all the literature.

• Slides from workshop on the method (PDF 4MB)
• Slides from the advanced workshop on the method (PDF 5MB)
• More resources on doing your PhD as a series of papers

The method and its benefits

• Pickering, C.M. and Byrne, J. (2014). The benefits of publishing systematic quantitative literature reviews for PhD candidates and other early career researchers. Higher Education Research and Development. 33: 534-548
• Pickering, C., Grignon, J., Steven, R., Guitart, D. and Byrne. J. (2015). Publishing not perishing: How research students transition from novice to knowledgeable using systematic quantitative literature reviews. Studies in Higher Education. 40:10, 1756-1769, DOI: 10.1080/03075079.2014.914907. A pre-print copy of the paper is available here , or the final published version from the publisher's website.

Research study

Videos about the method

Overview of method

Being systematic

Creating your own review database

Writing the review

Why publish during your PhD?

Rochele Steven discusses using the method

Julien Grignon discusses using the method

Advanced SQLR 1 - Challenges in being systematic

Advanced SQLR 2 - Coding challenges

Advanced SQLR 3 - Advanced data analysis

Advanced SQLR 4 - Reviewers comments

Three circles for structuring a literature review

Eloise Stephenson - Ross River virus ecology

There are now hundreds of papers published using this method. A full list of them is available from google scholar.

Some select examples showing how they have been done, including searching strategies, ways to analysis the data and address some concerns regarding use/non-use of grey literature, factors affecting demand for, and supply of research by country etc, addressed in the advanced videos include:

• Guitart, D., Pickering, C.M. and Byrne, J. (2012). Past results and future directions in urban community gardens research. Urban Forestry and Urban Greening. 11: 364-373 — This was one of the original reviews using our methods. It highlights the importance of gap analysis and dealing with reviewing a very diverse literature including disciplines and methods used, and the capacity to review both quantitative and qualitative literature.
• Steven, R. Pickering, C.M. and Castley, G. (2011). A review of the impacts of nature based recreation on birds. Journal of Environmental Management. 92: 2287-2294 — This early SQLR provides a detailed example of why gap analysis is important and ways of quantifying differences between the supply of literature and need for research by calculating bird diversity vs number of studies per region.
• Pickering, C.M., Rossi, S.D., Hernando, A. and Barros, A. (2018). Current knowledge and future research directions for monitoring and management of visitors in recreational and protected areas. Journal of Outdoor Recreation and Tourism. 21: 10-18 — This SQLR of abstracts from a conference, includes a detailed examination of factors affecting the supply of research including why research is often dominated by literature from North America and Europe, and why that can be an issue. It also includes more sophisticated ways to access the results by using multi-dimensional analysis.
• Verrall, B. and Pickering, C.M. (2020). Alpine vegetation in the context of climate change: A global review of past research and future directions. Science of the Total Environment. 748:141344 . This is an example of a bibliometric/scientometric literature review – an alternative method using vosViewer to visuals patterns in keywords over time in the literature and changes in who is publishing on a topic and where.
• Thomas, S. (2014). Blue carbon: Knowledge gaps, critical issues and novel approaches — This review also uses Leximancer analysis (text analysis) of themes to map concepts in the literature based on keywords and phrases in the papers.
• Riebe, L., Girardi, A. and Whitsed, C. (2016). A systematic literature review of teamwork pedagogy in higher education. Small Group Research. 47: 619-664 — This paper from the education area includes an excellent example of how to use keywords and search terms in a stepped down way to identify a complex literature.
• Liao, Y., Deschamps, F., de Freitas Rocha Loures, E. and Ramos, L.F.P. (2017). Past, present and future of Industry 4.0- a systematic literature review and research agenda proposal. International Journal of Production Research. 55: 3609-3629 — This paper includes a good example about how to set out the justification for which papers to include and exclude including less clear-cut topics. It also includes an excellent way to quantify the disciplines represented by papers using well recognised (SCImago) data for each journal.
• Turner, J.A., Babcock, R.C., Hovey, R., and Kendrick, G.A. (2017). Deep thinking: a systematic review of mesophotic coral ecosystems. ICES Journal of Marine Science — This review uses column diagrams to clearly illustrate how over time the methods used in the discipline and the types of analysis conducted become more complex as the literature develops.
• Pahlevan-Sharif, S. Mura, P., and Wijesinghe, S.N.R. (2019). A systematic review of systematic reviews in tourism. Journal of Hospitality and Tourism Management. 39: 158.165 — This recent paper provides an overview review of other literature reviews. It highlights the types of methods used in reviews in tourism including recommendations for future reviews.
• Bezerra, M.F., Lacerda, L.D. and Lai, C-T. (2019). Trace metals and persistent organic pollutants contamination in batoids (Chondrichthyes: Batoidea): A systematic review. Environmental Pollution. 248: 684-695 — This recent review includes a broad SQLR and meta-analysis of a science/quantitative area.
• Nikulina, V., Simon, D., Ny, H. and Baumann, H. (2019). Context-adapted urban planning for rapid transitioning of personal mobility towards sustainability: A systematic literature review. Sustainability: 11: — This paper combines a SQLR, with narrative commentary on themes, and a bibliometric analysis.

Further resources and contacts

• Griffith University Research Centres
• Professor Catherine Pickering
• Pickering, C.M. (2012). Writing Ecology Research Papers. Environment Futures Research Centre. Griffith University, Gold Coast

Article in The Conversation:

• Pickering, C.M. and Byrne, J. (2014). How to find the knowns and unknowns in any research. The Conversation.

Important reference for how to report systematic literature reviews required by many journals:

• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71 - with more details available here.
• Boote, B.N. and Beile P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher. 34: 3-15.
• Crisp, B.R. (2015) Systematic reviews: A social work perspective. Australian Social Work, 68:3, 284-295.
• Murray, R. (2011). How to Write a Thesis. McGraw Hill Open University Press. Maidenhead, England (Chapter on writing a literature reviews).
• Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing, Oxford, England.
• Randolph J.J. (2009). A guide to writing the dissertation literature review. Practical Assessment, Research and Evaluation. 14: 1-13.

Here are examples of the types of excel databases used in some Systematic Quantitative Literature Reviews:

• Steven et al. 2011 database of papers on impacts nature based tourism on birds (XLSX 16KB)
• Ballantyne and Pickering In review databases of papers on environmental impacts of recreation trails (XLSX 32KB)
• Byrne and Portanger 2014 database of papers climate change, energy policy and justice (XLSX 48KB)

Some of the journals publishing SQLR include:

• Analyse und Kritik
• ASEE Annual Conference and Exposition, Conference Proceedings
• Asia and the Pacific Policy Studies
• Asian Journal of Criminology
• Austral Entomology
• Australian Social Work
• Behaviour Change
• Biological Conservation
• BMC Health Services Research
• CIRP Journal of Manufacturing Science and Technology
• Climatic Change
• Conservation Biology
• Corruption in Sport: Causes, Consequences, and Reform
• Crop Protection
• Crystal Research and Technology
• Cuadernos de Desarrollo Rural
• Cuadernos de Turismo
• Current Issues in Tourism
• Documents d'Analisi Geografica
• Ecologia Austral
• Ecological Economics
• Ecology and Society
• Education Sciences
• Educational Media International
• Energy and Buildings
• Environment International
• Environment Systems and Decisions
• Environmental Modelling and Software
• Environmental Pollution
• Environmental Science and Policy
• European Journal of Higher Education
• European Journal of Information Systems
• Evaluation and Program Planning
• Event Management
• Fish and Fisheries
• Frontiers in Ecology and the Environment
• Global Environmental Change
• Habitat International
• Higher Education Research and Development
• ICES Journal of Marine Science
• IEEE Internet of Things Journal
• IFAC-PapersOnLine
• International Journal of Disaster Risk Reduction
• International Journal of Language and Communication Disorders
• International Journal of Managing Projects in Business
• International Journal of Mental Health Nursing
• International Journal of Production Research
• International Journal of the Commons
• International Journal of Tourism Research
• International Social Work
• Issues in Educational Research
• Journal of Business Research
• Journal of Cleaner Production
• Journal of Environmental Management
• Journal of Healthcare Leadership
• Journal of Hospitality and Tourism Research
• Journal of Hospitality Marketing and Management
• Journal of Information Technology in Construction
• Journal of Outdoor Recreation and Tourism
• Journal of Place Management and Development
• Journal of Reproductive and Infant Psychology
• Journal of Sustainable Tourism
• Journal of Technical Education and Training
• Journal of the Medical Library Association
• Journal of Travel and Tourism Marketing
• Journal of Urbanism
• Journal of Vocational Rehabilitation
• Landscape and Urban Planning
• Local Environment
• Managing Sport and Leisure
• Natural Hazards and Earth System Sciences
• Nurse Education in Practice
• Ocean and Coastal Management
• Procedia CIRP
• Procedia Engineering
• Procedia Manufacturing
• Proceedings of AISB Annual Convention 2018
• Proceedings of the International Conference on e-Learning, ICEL
• Progress in Transplantation
• Reliability Engineering and System Safety
• Renewable and Sustainable Energy Reviews
• Restoration Ecology
• Scientometrics
• Sex Education
• Small Group Research
• Solar Energy
• South African Computer Journal
• Speech, Language and Hearing
• Studies in Higher Education
• Sustainability (Switzerland)
• Teaching and Learning in Medicine
• Teaching and Teacher Education
• The Routledge Companion to Management Information Systems
• Tourism Management
• Tourism Management Perspectives
• Training and Education in Professional Psychology
• Transportation Research Record
• Urban Ecosystems
• Urban Forestry and Urban Greening
• Water Research
• Wildlife Research
• Wiley Interdisciplinary Reviews: Climate Change
• World Leisure Journal

Connect with us

If you would like to work, study or collaborate with us, get in touch

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

About Systematic Reviews

Are Systematic Reviews Qualitative or Quantitative?

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

A systematic review is designed to be transparent and replicable. Therefore, systematic reviews are considered reliable tools in scientific research and clinical practice. They synthesize the results using multiple primary studies by using strategies that minimize bias and random errors. Depending on the research question and the objectives of the research, the reviews can either be qualitative or quantitative. Qualitative reviews deal with understanding concepts, thoughts, or experiences. Quantitative reviews are employed when researchers want to test or confirm a hypothesis or theory. Let’s look at some of the differences between these two types of reviews.

To learn more about how long it takes to do a systematic review , you can check out the link to our full article on the topic.

Differences between Qualitative and Quantitative Reviews

The differences lie in the scope of the research, the methodology followed, and the type of questions they attempt to answer. Some of these differences include:

Research Questions

As mentioned earlier qualitative reviews attempt to answer open-ended research questions to understand or formulate hypotheses. This type of research is used to gather in-depth insights into new topics. Quantitative reviews, on the other hand, test or confirm existing hypotheses. This type of research is used to establish generalizable facts about a topic.

Type of Sample Data

The data collected for both types of research differ significantly. For qualitative research, data is collected as words using observations, interviews, and interactions with study subjects or from literature reviews. Quantitative studies collect data as numbers, usually from a larger sample size.

Data Collection Methods

To collect data as words for a qualitative study, researchers can employ tools such as interviews, recorded observations, focused groups, videos, or by collecting literature reviews on the same subject. For quantitative studies, data from primary sources is collected as numbers using rating scales and counting frequencies. The data for these studies can also be collected as measurements of variables from a well-designed experiment carried out under pre-defined, monitored conditions.

Data Analysis Methods

Data by itself cannot prove or demonstrate anything unless it is analyzed. Qualitative data is more challenging to analyze than quantitative data. A few different approaches to analyzing qualitative data include content analysis, thematic analysis, and discourse analysis. The goal of all of these approaches is to carefully analyze textual data to identify patterns, themes, and the meaning of words or phrases.

Quantitative data, since it is in the form of numbers, is analyzed using simple math or statistical methods. There are several software programs that can be used for mathematical and statistical analysis of numerical data.

Presentation of Results

Learn more about distillersr.

(Article continues below)

Final Takeaway – Qualitative or Quantitative?

3 reasons to connect.

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Galdas P, Darwin Z, Fell J, et al. A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN). Southampton (UK): NIHR Journals Library; 2015 Aug. (Health Services and Delivery Research, No. 3.34.)

A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN).

Chapter 2 quantitative review methods.

A systematic review and meta-analysis was conducted based upon a protocol published on the PROSPERO database (registration number CRD42013005394, URL: www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42013005394 ).

Deviations from the original protocol are presented in Box 1 .

Deviations from original PROSPERO protocol The target population are male adults (aged 18 years or over) living with one or more long-term conditions.

• Search strategy

We searched the following databases using a search strategy developed in conjunction with an information specialist from the Centre for Reviews and Dissemination, University of York (see Appendix 1 ): Cochrane Database of Systematic Reviews (CDSR); Database of Abstracts of Reviews of Effects (DARE) (up to July 2013); PROSPERO (International Prospective Register of Systematic Reviews) (up to July 2013); and Medical Literature Analysis and Retrieval System Online (MEDLINE) (January 2012 to July 2013). The breadth of the literature identified meant we took a pragmatic approach and limited our search to CDSR; see Box 1 .

Inclusion/exclusion criteria

Randomised controlled trials (RCTs) investigating self-management support interventions in men with LTCs (identified via Cochrane systematic reviews of self-management support interventions) were included. Studies which analysed the effects of self-management support interventions in sex groups within a RCT were also identified and synthesised separately.

The following population, intervention, comparison and outcome criteria were used:

• Population and setting : adults, 18 years of age or older, diagnosed with a LTC. We limited the review to studies of patients with 14 ‘exemplar’ LTCs (informed by disease areas prioritised in the PRISMS study and team discussions): asthma, diabetes, depression, hypertension, heart failure, chronic obstructive pulmonary disease (COPD), arthritis, chronic kidney disease, chronic pain, human immunodeficiency virus (HIV), testicular cancer, prostate cancer, prostate hyperplasia and chronic skin conditions in any setting. Studies including inpatients with depression were excluded. Studies including patients with multimorbidity involving at least one ‘exemplar’ condition were considered.

An intervention primarily designed to develop the abilities of patients to undertake management of health conditions through education, training and support to develop patient knowledge, skills or psychological and social resources.

• Comparison : any comparison group. We considered studies using ‘care as usual’ or any other intervention.
• Outcomes : effectiveness, cost-effectiveness. We extracted data on the effect of interventions on health status, clinical measures, health behaviour, health-care use, self-efficacy, knowledge and understanding, communication with health-care professionals (HCPs) and effects on members/carers.
• Study design : RCTs identified via eligible Cochrane systematic reviews. Only papers published in the English language were included, as translation was not feasible in the time frame of the project. In instances where records were unobtainable, attempts were made to contact authors to request the information.

Criteria for defining a self-management support intervention The intervention should, through some means of education, training or support, help people with a LTC by:

• Identification of studies

We piloted the screening criteria on a sample of papers before undertaking the main screening, in order to identify and resolve any inconsistencies. Screening was conducted in two phases:

• identification of relevant Cochrane systematic reviews
• identification of relevant RCTs within included Cochrane systematic reviews.

For phase 1, an initial screen by title and abstract was conducted by one researcher. Two researchers then screened each article independently according to the screening criteria to identify relevant systematic reviews. Disagreements were resolved by a third researcher (principal investigator) as required.

For phase 2, each Cochrane review was screened independently for eligible RCTs by two researchers. The eligibility of each RCT was checked using the study information presented within Cochrane reviews before full papers were sourced. Full texts of each RCT were independently screened by two researchers and disagreements were resolved by a third researcher (principal investigator) as required.

For this review we focused on identifying male-only RCTs and trials which analysed the effects of interventions by sex groups. Agreement on Cochrane review eligibility was 89% and agreement on male-only RCT inclusion/exclusion and identification of RCTs containing sex group analyses was > 90%.

• Data extraction

We designed a data extraction sheet and piloted this on a sample of papers prior to the main data extraction. Relevant data from each included article were extracted by a member of the review team and checked for completeness and accuracy by a second member of the team. Disagreements were discussed and resolved by a third person (principal investigator) as required. In instances where key information for meta-analysis was missing, efforts were made to contact authors. We extracted data on study and population characteristics, intervention details (setting, duration, frequency, individual/group, delivered by), outcome measures of health status, clinical measures, health behaviour, health-care use, self-efficacy, knowledge and understanding, communication with HCPs and items for quality assessment (Cochrane risk of bias tool 35 ). Items for economic evaluations [hospital admission, service use, health-related quality of life (HRQoL), incremental cost-effectiveness ratios] were also extracted.

Where studies were reported in multiple publications, each publication was included and relevant data were extracted.

• Quality assessment strategy

We extracted data on the methodological quality of all included male-only RCTs and appraised this using the Cochrane risk of bias tool. Quality appraisal was undertaken by two researchers independently and disagreements were resolved through discussion. Sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting and other sources of bias were assessed, assigning low, high or unclear risk of bias, as appropriate. The purpose of the quality appraisal was to describe the quality of the evidence base, not to give an inclusion/exclusion criterion.

Randomised controlled trials containing sex group analyses were assessed for quality using assessment criteria adapted from Pincus et al. 36 and Sun et al. 37 ‘Yes’, ‘No’ and ‘Unclear’ were recorded as responses to the following quality appraisal questions:

• Was the group hypothesis considered a priori?
• Was gender included as a stratification factor at randomisation?
• Was gender one of a small number of planned group hypotheses tested (≤ 5)?
• Was the study free of other bias (randomisation, allocation concealment, outcome reporting)?
• Data analysis

Meta-analysis was conducted using Review Manager version 5.2 (The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen, Denmark).

Data were extracted, analysed and presented as standardised mean difference (SMD) to account for the different instruments used, unless otherwise stated. As a guide to the magnitude of effect, we categorised an effect size of 0.2 as representing a ‘small’ effect, 0.5 a ‘moderate’ effect and 0.8 a ‘large’ effect. 38

A random-effects model was used to combine study data. Statistical heterogeneity was assessed with the I 2 value, with ‘low’ heterogeneity set at ≤ 25%, ‘moderate’ 50% and ‘high’ 75%.

In instances where studies contained multiple intervention groups, each group was extracted and analysed independently, dividing the control group sample size to avoid double counting in the analysis.

The following outcome measures were used in the analysis where possible: HRQoL, depression, anxiety, fatigue, stress, distress, pain and self-efficacy. Where a study contained more than one measure of a particular outcome (e.g. depression measured by the Centre for Epidemiologic Studies Depression Scale 39 and Beck Depression Inventory 40 ), the tool most established in the wider literature was chosen for meta-analysis. If the tool had multiple subscales, a judgement was made about the most relevant subscale. Where studies reported at multiple time periods, outcome measures reported at or closest to 6 months were used, as measures around this time were by far the most frequently reported.

Unless otherwise specified in the results section, positive effect sizes indicate beneficial outcomes for HRQoL and self-efficacy outcomes, while negative effect sizes indicate beneficial outcomes for depression, anxiety, fatigue, stress, distress and pain outcomes.

We conducted four types of analysis, described below.

Analysis 1: ‘within-Cochrane review analysis’

Analysis 1 sought to determine whether studies in males show larger, similar or smaller effects than studies in females and mixed-sex groups within interventions included within the ‘parent’ Cochrane review. We screened all included Cochrane reviews of self-management support interventions to identify those that contained analysis on outcomes of interest and at least two relevant male-only RCTs. Where an eligible review was identified that met these criteria, the studies were categorised as male only, mixed sex and female only ( Figure 1 ).

Analysis 1: ‘within-Cochrane review analysis’.

Such comparisons across trials do not have the protection of randomisation, and there may be differences between the studies included in each sex group which account for differences in effects between groups. We presented data on the comparability of these trials within these three categories, including the age of the included patient populations, and on the quality of the studies (using allocation concealment as an indicator of quality).

We report the effect size [together with significance and 95% confidence interval (CI)] of self-management support in each sex group (male only, mixed sex, female only). We conducted analyses to test whether or not interventions showed significantly different effects in sex groups. It should be noted that the power to detect significant differences in such analyses can be limited.

Analysis 2: ‘across-Cochrane review analysis’

Analysis 2 sought to determine whether studies in males show larger, similar or smaller effects than studies in females and mixed-sex groups within types of self-management support pooled across reviews.

In analysis 2, data were pooled according to broad intervention type across reviews, rather than within individual reviews as in analysis 1 ( Figure 2 ). This allowed us to determine whether broad types/components of self-management support interventions show larger, similar or smaller effects in males than in females and mixed populations. Limitations in the data meant that we were able to conduct analyses on only physical activity, education, peer support, and HCP monitoring and feedback interventions.

Analysis 2: ‘across-Cochrane review analysis’.

We report the effect size (together with significance and 95% CI) of self-management support in each sex group (male only, mixed sex, female only). We conducted analyses to test whether or not interventions showed significantly different effects in sex groups. It should be noted that the power to detect significant differences in such analyses can be limited.

Analysis 3: ‘male-only intervention type analyses’

We conducted a meta-analysis on trials including males only, according to broad intervention type – physical activity, education, peer support, and HCP monitoring and feedback – and compared effects between intervention types ( Figure 3 ). This allowed us to determine whether or not certain broad categories of self-management support intervention were effective in men.

Analysis 3: ‘male-only intervention type analyses’.

Analysis 4: ‘within-trial sex group analysis’

We identified RCTs which analysed the effects of self-management support interventions in sex groups. We sought to extract relevant data on the direction and size of moderating effects in secondary analysis (i.e. whether males show larger, similar or smaller effects than females), and assess these effects in the context of relevant design data, such as sample size, and the quality of the secondary analysis ( Figure 4 ).

Analysis 4: ‘within-trial sex group analysis’.

Sex group analyses within trials do in theory provide greater comparability in terms of patient and intervention characteristics than analyses 1–3.

A mixture of LTCs was included within each analysis, constituting the main analysis. Although this was not in the original protocol, we attempted to conduct an analysis by each disease area. We found there were sufficient data to conduct a sex-comparative analysis in only cancer studies; the results are presented in Appendix 2 .

• Coding interventions for analysis

The plan to use the behavioural change techniques (BCT) taxonomy was dropped (see Box 1 on protocol deviations). Post hoc, we took a pragmatic approach to coding interventions. Development of the intervention categories was informed by the published literature identified in this project and previous work conducted by the PRISMS and RECURSIVE project teams. 7 , 33 Table 1 provides a list of the categories and their associated description. Categories were designed to be broadly representative of the interventions identified and facilitate comparison of intervention types in the analysis. Two members of the review team independently assessed the ‘type’ of self-management support intervention in each study in order to categorise it, and disagreements were identified and resolved by discussion with a team member.

Self-management support intervention categories and description

• Economic evaluation

The review of cost-effectiveness studies was initially planned as a two-stage review. First, we would review economic evaluations of self-management interventions on males only. Subsequently, we would review all economic evaluations with group analyses in which the costs and effects for males and females could be separated.

Study quality was assessed using a modified version of the Drummond checklist where appropriate. 45

• Study characteristics

Setting and sample

We identified a total of 40 RCTs on self-management support interventions conducted in male-only samples (some trials have more than one reference) ( Figure 5 ). The majority of the studies were conducted in the USA ( n  = 23), 46 – 70 with the remainder conducted in the UK ( n  = 6), 71 – 78 Canada ( n  = 5), 79 – 83 Spain ( n  = 3), 84 – 88 Sweden ( n  = 1), 89 Poland ( n  = 1) 90 and Greece ( n  = 1). 91 Males with prostate cancer were the most frequently studied male-only population ( n  = 15) included in this review. 48 , 49 , 52 , 58 , 59 , 61 , 64 – 66 , 68 , 69 , 72 , 78 , 80 , 89 Other disease areas included hypertension ( n  = 6), 47 , 71 , 79 , 82 , 83 , 85 , 86 COPD ( n  = 6), 54 , 55 , 73 – 76 , 81 , 84 , 87 , 88 heart failure ( n  = 4), 62 , 67 , 90 , 91 type 2 diabetes ( n  = 3), 46 , 50 , 51 , 70 diabetes of unspecified type ( n  = 1), 56 arthritis ( n  = 1) 63 and testicular cancer ( n  = 1). 77 One multimorbidity study recruited obese men with type 2 diabetes and chronic kidney disease. 57 The age of participants ranged from 25 to 89 years and, where reported, ethnicity was predominantly white. Only one study reported socioeconomic status using a validated tool; 63 the majority of other publications included a description of education or annual income.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram for the quantitative review.

Self-management support interventions

A total of 51 distinct self-management support interventions were reported across the 40 included male-only studies. Physical activity ( n  = 16), 49 , 57 , 62 , 72 – 76 , 78 , 80 , 81 , 84 , 87 – 91 education ( n  = 36), 46 – 55 , 58 – 61 , 63 – 67 , 70 – 72 , 77 , 79 – 81 , 83 – 88 peer support ( n  = 17) 47 , 49 , 53 , 56 , 68 – 72 , 80 and HCP monitoring and feedback ( n  = 25) 46 , 47 , 50 – 52 , 56 , 57 , 60 , 61 , 66 – 68 , 70 , 71 , 75 , 76 , 78 – 80 , 82 – 89 were the most frequently reported components of these interventions. Three interventions with a psychological component, 64 , 77 two interventions containing a financial incentive component 82 , 83 and one study containing an action plan component 19 were also identified.

Twenty-three of the interventions were aimed at individuals, 46 , 48 , 50 – 52 , 54 , 55 , 60 , 61 , 64 , 65 , 67 – 69 , 75 – 78 , 82 – 86 20 were aimed at groups 47 , 53 , 58 , 59 , 62 , 66 , 70 , 71 , 79 , 89 – 91 and the remainder used a mixed individual and group approach ( n  = 6). 49 , 56 , 72 – 74 , 80 , 81 , 87 , 88 It was unclear what approach was used in two studies. 57 , 63 Over half of the interventions lasted 0–5 months ( n  = 28), 47 , 53 , 58 – 64 , 67 – 69 , 71 – 80 , 85 , 86 12 interventions ranged between 6 and 11 months, 46 , 52 , 54 – 57 , 66 , 70 , 84 , 90 , 91 six interventions were 12 months or longer 49 , 65 , 81 , 82 , 84 , 87 , 88 and in five cases the total programme duration was unclear. 48 , 83 , 89

The mode of administration of the interventions varied. They included telephone-based support ( n  = 6), 60 , 61 , 65 , 67 face-to-face delivery ( n  = 21), 47 , 53 – 55 , 58 , 59 , 62 – 64 , 66 , 68 – 70 , 77 , 83 , 89 – 91 remote unsupervised activities ( n  = 2), 75 , 76 , 78 a combination of face-to-face delivery and remote unsupervised activities ( n  = 20), 46 – 51 , 57 , 71 – 74 , 79 – 82 , 84 – 89 and a combination of face-to-face delivery and telephone support ( n  = 2). 52 , 56

In terms of setting, interventions were reported to be home-based ( n  = 11), 46 , 52 , 60 , 61 , 65 , 67 , 75 , 76 , 78 at a non-home location such as a dedicated gym, pharmacy, hospital clinic, work, university laboratory, coffee shop or other community-based venue ( n  = 12), 53 – 55 , 62 – 64 , 68 – 70 , 77 , 85 , 86 , 90 a combination of home and non-home-based venue ( n  = 14) 48 – 51 , 56 , 57 , 72 – 74 , 79 – 84 , 87 , 88 or not clearly reported in the publication ( n  = 14). 47 , 58 , 59 , 66 , 71 , 89 , 91

Half of the studies 79 – 82 , 46 , 48 – 51 , 53 , 56 , 58 , 59 , 66 , 70 , 72 , 78 , 84 , 87 , 88 reported on some aspect of compliance with the self-management intervention and most participants were followed up for 6 months or less ( n  = 24) following participation in the intervention.

Table 2 provides an overview of study details and Table 3 includes detailed descriptions of the self-management support intervention.

Male-only study characteristics

Male-only studies: self-management support intervention characteristics

• Quality assessment: risk of bias

Study quality was assessed using the Cochrane risk of bias tool, 92 which covers six key domains: sequence generation, allocation concealment, blinding performance, incomplete outcome data, selective outcome reporting and other sources of bias.

Studies were often poorly reported, making judgements of quality difficult. With the exception of selective outcome reporting, the most frequent rating for all domains was an unclear risk of bias. For the selective outcome-reporting domain, a low risk of bias was most frequently reported assignment. Table 4 describes the risk of bias allocation for each study by each domain. Figure 6 presents a summary of the male-only study quality assessment findings.

Male-only study Cochrane risk of bias findings

Summary of male-only study Cochrane risk of bias findings.

Included under terms of UK Non-commercial Government License .

• Cite this Page Galdas P, Darwin Z, Fell J, et al. A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN). Southampton (UK): NIHR Journals Library; 2015 Aug. (Health Services and Delivery Research, No. 3.34.) Chapter 2, Quantitative review methods.
• PDF version of this title (3.9M)

In this Page

Other titles in this collection.

• Health Services and Delivery Research

Recent Activity

• Quantitative review methods - A systematic review and metaethnography to identif... Quantitative review methods - A systematic review and metaethnography to identify how effective, cost-effective, accessible and acceptable self-management support interventions are for men with long-term conditions (SELF-MAN)

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Open access
• Published: 24 June 2024

Development of a quantitative index system for evaluating the quality of electronic medical records in disease risk intelligent prediction

• Jiayin Zhou 1   na1 ,
• Jie Hao 1   na1 ,
• Mingkun Tang 1 ,
• Haixia Sun 1 ,
• Jiayang Wang 2 ,
• Jiao Li 1 &
• Qing Qian 1  

BMC Medical Informatics and Decision Making volume  24 , Article number:  178 ( 2024 ) Cite this article

105 Accesses

Metrics details

This study aimed to develop and validate a quantitative index system for evaluating the data quality of Electronic Medical Records (EMR) in disease risk prediction using Machine Learning (ML).

Materials and methods

The index system was developed in four steps: (1) a preliminary index system was outlined based on literature review; (2) we utilized the Delphi method to structure the indicators at all levels; (3) the weights of these indicators were determined using the Analytic Hierarchy Process (AHP) method; and (4) the developed index system was empirically validated using real-world EMR data in a ML-based disease risk prediction task.

The synthesis of review findings and the expert consultations led to the formulation of a three-level index system with four first-level, 11 second-level, and 33 third-level indicators. The weights of these indicators were obtained through the AHP method. Results from the empirical analysis illustrated a positive relationship between the scores assigned by the proposed index system and the predictive performances of the datasets.

The proposed index system for evaluating EMR data quality is grounded in extensive literature analysis and expert consultation. Moreover, the system’s high reliability and suitability has been affirmed through empirical validation.

The novel index system offers a robust framework for assessing the quality and suitability of EMR data in ML-based disease risk predictions. It can serve as a guide in building EMR databases, improving EMR data quality control, and generating reliable real-world evidence.

Peer Review reports

Introduction

The onset of the digital health era has led to a paradigm shift in health management, transitioning from a focus on reactive treatment to proactive prevention [ 1 ]. Disease risk intelligent prediction has become a vital strategy in proactive health management, aiming to identify potential risk factors and prevent the progression of diseases. By harnessing the capabilities of Artificial Intelligence (AI) technologies and Machine Learning (ML) approaches, healthcare professionals can gain valuable insights into diseases, enabling the development of more effective preventive treatment plans [ 2 , 3 ].

Johnson [ 4 ] applied four different ML-based models to predict subsequent deaths or cardiovascular events in a cohort of 6,892 patients. The study found that the ML-based model had superior discrimination ability compared to traditional coronary Computed Tomography (CT) scores in identifying patients at risk of adverse cardiovascular events. Electronic medical records (EMR) data, as a valuable real-world data source, plays a critical role in disease risk prediction using ML techniques [ 4 ]. An EMR refers to a digital version of a patient’s medical record, encompassing medical history, medications, test results, and other relevant information [ 5 , 6 ]. Healthcare providers commonly utilize EMRs to document and track patient information, enabling comprehensive decision-making regarding patient care. Furthermore, clinical researchers can leverage de-identified EMR data to study disease patterns, develop novel treatments, and advance medical knowledge. The integration of ML with EMRs has recently shown significant improvements in predicting patient outcomes, such as identifying individuals with suspected coronary artery disease [ 7 ] or forecasting the likelihood of open-heart surgery [ 8 ]. These advancements highlight the potential of ML in enhancing the efficiency of clinical decision-making [ 9 ].

Nevertheless, several studies have raised concerns about the quality of EMRs in clinical research, emphasizing issues such as lack of data standardization, incomplete or missing clinical data, and discrepancies in data types and element representations [ 10 , 11 ]. Ensuring the quality of EMR data is crucial, as it forms the bedrock for effective utilization of EMRs. High-quality EMR data not only supplies robust evidence, but also accelerates the clinical research process, shortens its timeline, and reduces associated risks. Therefore, controlling and evaluating EMR data quality are pivotal in upholding the overall quality and integrity of clinical research.

Despite numerous studies investigating the assessment of EMR data quality in clinical research, it is noteworthy that the body of literature evaluating EMR data quality is growing [ 12 , 13 ]. However, publicly published clinical studies employing ML techniques and utilizing EMR data frequently overlook data quality or implement methods lacking expert knowledge or evidential support. While methods for data quality evaluation have been described in the informatics literature, researchers without specialized knowledge in this field may find difficulty choosing the appropriate evaluation method in line with the available data and research problems [ 14 ]. Furthermore, the existing quality assessment framework primarily relies on qualitative approaches, making objective measurement of quality and suability challenging.

In this paper, we aim to develop and validate a quantitative index system for evaluating the quality of EMR in disease risk prediction using ML. The proposed index system is intended to provide guidance for utilizing EMR data in research, enhance the quality of EMR data within a Hospital Information System (HIS), and facilitate the implementation of clinical decision-making research based on EMR data. By applying the proposed index system, researchers and healthcare professionals can make knowledgeable decisions regarding the use of EMR data for ML-based disease prediction research, ultimately improving patient care and advancing medical knowledge.

In this paper, we present the development of a quantitative index system, depicted in Fig.  1 , designed to ensure the quality control of EMR data in disease prediction models. The development process incorporated the use of the Delphi method and the analytic hierarchy process (AHP). In addition, an empirical study was undertaken to validate the effectiveness of the developed index system using real-world EMR data in disease risk intelligent prediction.

Workflow of the study

Sketching the preliminary index system

Preliminary indicator identification, definition, and organization.

The initial set of indicators was determined through a comprehensive literature review of studies published before September 27, 2021, obtained from the PubMed database. The search query used was “(machine learning) AND (electronic medical records) AND (disease prediction)”, which resulted in 549 papers. The inclusion criteria required that the research data be related to EMR or HIS and that disease risk was predicted using ML techniques. Review articles and papers deemed to have low relevance were excluded, leading to the removal of 225 papers based on the fulfillment of the exclusion criteria after reading abstracts.

Further screening was conducted by reading the full papers to eliminate studies that did not involve EMR or HIS data or utilized disease prediction methods other than machine learning. Additionally, 18 relevant papers were included by examining the reference lists of the selected studies. Ultimately, a total of 229 papers were retained for the development of the preliminary index system. The detailed process of paper screening is illustrated in Fig.  2 .

Flowchart of paper screening

Upon analyzing the review results, we formulated an initial multi-level index system consisting of four first-level, 11 second-level, and 33 third-level indicators. The first-level indicators represent broad dimensions of data quality, while the second-level indicators correspond to the general dimensions specifically for EMR data quality. The third-level indicators capture specific dimensions relevant to EMR-based disease prediction models.

Calculation methods

We utilized the AHP method to determine the weights of the first- and second-level indicators in the three-level index system. The weights of the third-level indicators were calculated using percentages or binary values according to their definitions. The calculation formulas of these third-level indicators will be assessed in the forthcoming Delphi consultation.

Developing a three-level index system using the Delphi method

Questionnaire compilation and expert consultation.

We conducted a Delphi consultation to gather feedback from experts based on the preliminary index system. The consultation questionnaire, provided in Additional file 1 , consists of four parts: experts' basic information (see Table S1), familiarity and judgment basis with AI-based disease prediction (see Table S2-S3), evaluation tables for the preliminary index system (see Table S4-S6), and an evaluation table for the calculation formulas of the third-level indicators (see Table S7). The importance of the preliminary indicators was measured using a 5-point Likert scale, ranging from “very unimportant” to “very important”. To ensure the extensibility of the preliminary index system, three additional options were included: delete, modify, and new indicator(s). For the calculation formulas part, experts were asked to provide a yes or no response, and if the answer was no, a suggestion for modification was requested.

A total of twenty experts specializing in healthcare/EMR data governance and medical AI were selected for the Delphi consultation. The inclusion criteria for the selection were as follows: (1) holding a Ph.D. degree or being a senior technical associate; (2) possessing more than two years of research experience in related fields; (3) being familiar with the construction and evaluation of EMR data; and (4) being able to give feedback in a timely manner. We conducted a single-round consultation since the nature of our consulting panel was relatively small and homogeneous [ 15 ].

Key coefficients and statistical analyses

To achieve relatively consistent and reliable feedback from the questionnaire, we calculated four metrics: the experts' positive coefficients, expert authority coefficients (Cr), coefficient of variation (CV), and Kendall's coefficient of concordance. The experts' positive coefficients were determined based on the response rate to the questionnaire. A response rate of 70% or higher is considered satisfactory [ 16 ]. The Cr was calculated as the average of the familiarity coefficient (Cs) and the judgment coefficient (Ca), reflecting the reliability of the expert consultation. A Cr value of 0.7 or above is considered acceptable. The CV measures the consistency of indicators on the same level. A CV value less than 0.25 is expected, indicating a high level of consistency [ 17 ]. Kendall's coefficient of concordance evaluates the overall consistency of all indicators in the system. It ranges from zero to one, with a value greater than 0.2 considered acceptable [ 18 ]. All statistical analyses were performed using Microsoft Excel/IBM SPSS 25.0.

Using the AHP method for weight assignment

We applied the AHP method to determine the weights of indicators at each level, which is a well-known technique in multiple criteria decision-making [ 19 ]. AHP enables the quantification of criteria and opinions that are difficult to measure numerically, and its outcomes are free from subjective influence due to its use of pairwise comparisons and eigenvalues [ 20 ].

In this study, our AHP method was conducted in three steps. First, we obtained the importance ratings of experts for each indicator. Then, we averaged these ratings for each indicator and performed pairwise comparisons among indicators at the same level that belong to the same upper level. This step allowed us to construct multiple judgment matrices based on their ratios.

Second, we calculated the eigenvectors of each indicator by normalizing the judgement matrix. A larger eigenvector for an indicator represents a higher relative importance. The relative weights of indicators at the same level were determined by standardizing the eigenvectors. For the first-level indicators, their relative weights were equal to their absolute weights. For the second- and third-level indicators, their absolute weights were calculated by multiplying their relative weights with the absolute weight of the upper level.

Third, we performed a consistency test using the consistency ratio (CR) to evaluate the consistency of the judgment matrices. A CR below 0.1 indicated that the judgment matrices were consistent and that the obtained weights were considered valid [ 21 ]. The steps of the AHP method are illustrated in Fig.  3 .

Flowchart of the AHP method

Evaluating the index system through prediction tasks

To further validate the suitability of the proposed index system, an empirical study was conducted using real-world EMR data for disease risk prediction.

Dataset construction

To ensure a fair assessment, we opted to generate multiple datasets from a single EMR data resource. The chosen data resource needed to be large-scale, open-access, and regularly updated. Once the data resource was identified, we constructed several datasets with varying sample types but maintaining the same set of attributes.

For each dataset, we computed the scores of 33 third-level indicators using the established calculation formulas. The weights of the proposed index system were applied to obtain weighted scores for all indicators within each dataset. The overall score of a dataset was subsequently computed by summing the scores of the first-level indicators.

Predictive modeling

In the context of disease risk prediction, we considered three widely used ML models: logistic regression (LR), support vector machine (SVM), and random forest (RF). LR is a traditional classification algorithm used to estimate the probability of an event occurring [ 22 ]. SVM, a nonlinear classifier, employs a kernel function to transform input data into a higher-dimensional space, making it effective in handling complex relationships and nonlinear patterns [ 23 ]. RF is an ensemble method that combines the predictions from multiple decision trees. It has shown great success in disease risk prediction tasks by reducing overfitting and improving predictive accuracy [ 24 ]. For our analysis, we used the scikit-learn python library [ 25 ] to implement LR, SVM, and RF.

Reliability analysis

In our study, we conducted reliability analysis to examine the relationship between the scores obtained from our constructed datasets and the performance of predictive models. we applied Pearson correlation for assessing linear relationships [ 26 ] and Spearman correlation for nonlinearity [ 27 ]. The Pearson correlation coefficient was calculated using the formula:

Here, \({x}_{i}\) and \({y}_{i}\) represent individual data points from the two respective datasets, while \(\overline{x }\) and \(\overline{{\text{y}} }\) denote the mean values of these datasets. A Pearson correlation coefficient near 1 or -1 indicates a strong linear relationship between dataset scores and model performance, whereas a value close to 0 suggests a very weak linear relationship.

Similarly, the Spearman correlation coefficient was calculated using the formula:

Here, \({d}_{i}\) represents the difference in rank between the two datasets for the i -th observation, and n denotes the total number of observations. A Spearman correlation coefficient near 1 or -1 indicates a strong nonlinear relationship, while a value close to 0 suggests a very weak relationship.

In both analyses, statistical significance was established with a p-value less than 0.05. This finding indicates a significant correlation between the scores of our constructed datasets and the performance of predictive models. Thus, this statistically significant outcome supports the reliability of our proposed index system in evaluating the data quality of EMR for intelligent disease risk prediction.

The characteristics of the experts

In the Delphi consultation, a total of twenty experts were invited, of which 17 actively participated, yielding a response rate of 85.0%. Out of the 17 experts, 16 provided feedback that met the credibility criteria for a Delphi study, resulting in an effective response rate of 94.1%. These response rates reflect a high degree of expert engagement.

Most of the participating experts were male, held Ph.D. degrees, and specialized in medical informatics or medical AI. Over half of the experts were aged between 40 and 50 years, and 62.5% had between 10 and 20 years of work experience. Moreover, 68.7% of the experts occupied senior associate positions or higher. For detailed information, see Table S8 in Additional file 3 .

The key coefficients of the Delphi method

The degree of expert authority (Cr) is defined by two factors: the expert's familiarity with the consultation content (Cs) and the basis of expert judgment (Ca). Of the 16 participating experts, 7 were found to be very familiar with the content, while 9 were relatively familiar. This indicates an overall sound understanding of the field among the experts. Only two experts exhibited a low judgment basis, suggesting that the majority of the experts were well-equipped to offer informed judgment. Details of expert familiarity and judgment basis can be found in Table S9 in Additional file 3 .

Cr was calculated to be 0.89, with Cs and Ca values of 0.88 and 0.90, respectively. These values indicate a high level of expert authority and reliability in the consultation results. The CV values for the first-level indicators were less than 0.16, for the second-level indicators were less than 0.20, and for the third-level indicators were no more than 0.25. These low CV values indicate a high level of consistency among experts' scores for the preliminary indicators at each level. Kendall's coefficients of concordance for all three levels were greater than 0.30, indicating a substantial level of agreement among the experts. Additionally, the p-values for the preliminary second- and third-level indicators were very small, further confirming the consistency of experts' scores for each preliminary indicator. Overall, the results demonstrate a high level of consistency and reliability in the experts' assessments for each preliminary indicator.

The final weighted three-level index system

Experts' comments focused on changes to the definition of indicators. After further discussions with experts, all preliminary indicators were included in the final weighted three-level index system, as shown in Table  1 . No new indicators were added to the system. The index system comprises four first-level indicators, 11 second-level indicators, and 33 third-level indicators, with the weights determined using the AHP method and percentages.

The first-level indicators represent a series of data quality characteristics that determine the suitability of EMR data for disease risk intelligent prediction research. The second-level indicators provide a concrete representation or evaluation of the first-level indicators, making it easier for users to understand their extension or evaluation. The third-level indicators further specify the second-level indicators, providing clear quality requirements for different levels of granularity in the EMR dataset, such as data records, data elements, and data element values. This facilitates users in understanding the evaluation needs and contents more clearly. For detailed information on the indicators, please see Additional file 2 .

Data preprocessing

In this empirical study, the MIMIC-III clinical database was chosen as the representative real-world EMR data resource. MIMIC-III Footnote 1 is an extensive and freely accessible database that contains comprehensive health-related data from more than 46,000 patients admitted to intensive care unit (ICU) at the Beth Israel Deaconess Medical Center between 2001 and 2012 [ 28 ]. For this study, we utilized MIMIC-III v1.4, which is the latest version released in 2016 [ 29 ] and ensures effective control over EMR data.

Sepsis is a leading cause of mortality among ICU patients, highlighting the importance of accurate sepsis risk prediction for precise treatments in the ICU [ 30 ]. Hence, we selected sepsis as the disease prediction task using the MIMIC-III database. Potential predictors were extracted from the records of vital signs, routine blood examinations [ 31 ], liver function tests [ 32 ] and demographic information. The outcome variable for the prediction task is the occurrence of sepsis. Furthermore, we obtained five different populations of ICU patients with a high risk of sepsis from the MIMIC-III database. The number of patients in each population, categorized as elderly (> 80 years old), long-stay (> 30 days of length of stay, LLOS), ischemic stroke, acute renal failure (ARF), and cirrhosis (CIR), is presented in Table S10 in Additional file 3 .

Scoring datasets

According to the proposed index system, we evaluated the five datasets and assigned scores to each indicator based on their respective weights in the system. The detailed list of scores of divergent indicators for each dataset can be found in Table S11 in Additional file 3 . In Table  2 , we present the scores of first-level indicators. It is important to note that the scores for the operability indicator were consistent across all five datasets, with a value of 0.251. This is because these datasets were obtained from a single resource.

When considering the overall scores, the LLOS dataset achieved the highest score of 0.966, indicating a higher level of quality, while the ARF dataset obtained the lowest score of 0.907. These scores provide an assessment of the datasets' suitability and quality for disease risk prediction using the proposed index system.

Experimental results

Additional data processing was conducted to prepare the datasets for training ML models. To address the missing values, median imputation was applied to predictors with a small proportion of missing values in each dataset. To mitigate potential bias arising from imbalanced datasets, we applied undersampling on the majority class to achieve a balanced ratio of 1:1. Each dataset was then randomly split into 80% for training and 20% for testing. To ensure fairness in model comparison, the predictors were normalized, and a tenfold cross-validation was performed during the training process.

Regarding model hyperparameters, the LR model applied the 'liblinear' solver method. The SVM model utilized a Radial Basis Function kernel, with a regularization parameter (C) set to 1.0, and the gamma value was set to 'scale'. For the RF model, it was constructed with 10 trees (n_estimators = 10), a maximum tree depth of 7 (max_depth = 7), and optimal feature selection (max_features = ‘auto’).

The evaluation of model performance was based on accuracy (ACC), precision, and area under the curve (AUC). Accuracy represents the proportion of correct predictions made by a model among all predictions. Precision measures the proportion of true positive predictions among all positive predictions made by a model. AUC, also known as the area under the receiver operating characteristic curve, is a metric used to evaluate the performance of binary classification models [ 33 ].

Table 3 displays the model performance on the five datasets. Among the three models, LLOS achieved the highest performance across all three evaluation metrics. On the other hand, ARF had the lowest performance in most cases, except for precision in the LR model.

Association analysis

The relationships between the scores of datasets and the performance metrics of the models were analyzed as follows. First, a normality test was conducted on each pair of scores. If the scores passed the normality test, a Pearson correlation analysis was performed. Otherwise, a Spearman correlation analysis was conducted. Table 4 shows that all correlations, except for LR-Precision, were strongly positive and statistically significant. The SVM-Precision correlation showed the strongest effect among them.

We have developed a quantitative evaluation index system to assess the suitability of EMRs in disease risk intelligent prediction research. The proposed index system was validated through an empirical study using MIMIC-III datasets for predicting sepsis. Three popular ML models were performed, and the predictive results demonstrated that datasets with higher scores achieved better performance across three ML models. Our result is consistent with a previous study that showed the impact of data quality on prediction performance [ 34 ]. Additionally, the association analyses revealed a strong positive relationship between the scores of datasets and the combination of the ML model and evaluation metric. These findings confirm that the proposed index system was effective in evaluating the quality of EMR data in disease risk prediction using ML techniques.

Compared to the general framework for evaluating EMR data quality, our proposed index system was constructed by incorporating both the quality characteristics of EMR data and the specific research activities in ML-based disease risk prediction. It differs from the frameworks developed by Johnson [ 35 ] and Lv [ 36 ], which focused on summarizing literature on general medical data rather than specifically on EMR data. Although Weiskopf [ 37 ] specified EMR data as a required condition for a literature search, they did not explicitly address the situation of using EMR data in their development. The proposed index system considers not only the practical foundation of EMR data but also the data processing operations and operational objectives of EMRs at different stages of prediction model construction. This approach makes the evaluation index system more focused on its research purpose and enhances its explanatory power.

Another significant contribution of the proposed index system is the quantitative evaluation of EMR data quality in disease risk prediction. This provides researchers with guidance or standards for quantifying the EMR datasets for specific research purposes. Most current EMR data quality evaluation systems for clinical research rely on qualitative indicators [ 38 ]. Qualitative indicators are often based on typical cases, statements, and supporting materials, which may lack objectivity. While the study of Weiskopf [ 37 ] incorporated quantitative evaluation, it still relied on subjective scoring of each dimension by experts to calculate the mean value. The evaluation model proposed by Zan [ 39 ] utilized objective measurement indicators, but it primarily focused on binary classification and only included first-level indicators.

The proposed three-level index system was developed using a combination of qualitative and quantitative approaches. The naming and definition of all three levels of indicators were constructed through an extensive literature review and expert consultation. The first-level indicators correspond to the core qualitative aspects of evaluating EMR data quality in ML-based disease risk prediction. The second-level indicators serve as a refinement of the first-level qualitative indicators. The third-level indicators are quantitative in nature and can be obtained through objective quantitative calculations, such as assessing the coverage bias of the outcome variables in the integrity of the third-level indicators. Through the AHP method, the weights of the first- and second-level indicators can be obtained by the weights of third-level indicators in a hierarchical way.

Our study has several limitations. First, the calculation of the weights for the third-level indicators was based on simple percentages. This calculation method may neglect variations in the importance of different indicators. Second, the empirical study was conducted using the MIMIC-3 v1.4 database. Although the MIMIC-3 dataset is a widely used resource in research, the use of a single data resource may restrict the generalizability of our findings. Certain indicators for comparing data resources may be hard to validate without diverse EMR data resources. Hence, future validation studies using another data resource should be conducted to ensure the robustness of the proposed index system.

In this paper, we developed a quantitative three-level index system, which included four first-level, 11 second-level, and 33 third-level indicators, to evaluate the EMR data quality in ML-based disease risk prediction. The reliability of the proposed index system has been verified through an empirical study with real-world data.

The proposed index system can benefit both EMR users for research and data managers. For EMR users for research, the proposed index system could provide them with a measurement for the suitability of EMR data in ML-based disease risk predictions. For EMR data managers, it could guide the direction of EMR database construction and improve the EMR data quality control. Eventually, we hope that the proposed index system can promote the generation of real-world evidence from reliable real-world EMR data.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its additional files.

https://physionet.org/content/mimiciii/1.4/

Abbreviations

Artificial intelligence

Analytic hierarchy process

Acute renal failure

Area under curve

Judgment coefficient

Consistency ratio

Expert authority coefficients

Familiarity coefficient

Computed tomography

Coefficient of variation

• Electronic medical record

Hospital information system

Intensive care units

 > 30 Days of length of stay

Logistic regression

• Machine learning

Support vector machine

Random forest

Waldman SA, Terzic A. Healthcare evolves from reactive to proactive. Clin Pharmacol Ther. 2019;105(1):10.

Article   PubMed   Google Scholar  

Razzak MI, Imran M, Xu G. Big data analytics for preventive medicine. Neural Comput Appl. 2020;32:4417–51.

Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94–8.

Article   PubMed   PubMed Central   Google Scholar  

Dzau VJ, Balatbat CA. Health and societal implications of medical and technological advances. Sci Transl Med. 2018;10(463):eaau4778.

Institute of Medicine. The Computer-Based Patient Record: An Essential Technology for Health Care. Washington DC: National Academy Press; 1997.

Google Scholar  

Ambinder EP. Electronic health records. Journal of oncology practice. 2005;1(2):57.

Motwani M, Dey D, Berman DS, et al. Machine learning for prediction of all-cause mortality in patients with suspected coronary artery disease: a 5-year multi centre prospective registry analysis. Eur Heart J. 2017;38(7):500–7.

PubMed   Google Scholar  

Allyn J, Allou N, Augustin P, et al. A comparison of a machine learning model with EuroSCORE II in predicting mortality after elective cardiac surgery: a decision curve analysis. PLoS ONE. 2017;12(1): e0169772.

Geissbuhler A, Miller RA. Clinical application of the UMLS in a computerized order entry and decision-support system. Proc AMIA Symp. 1998;1998:320–4.

Field D, Sansone SA. A special issue on data standards. OMICS. 2006;10(2):84–93.

Article   CAS   Google Scholar  

Mead CN. Data interchange standards in healthcare IT–computable semantic interoperability: now possible but still difficult, do we really need a better mousetrap? J Healthc Inf Manag. 2006;20(1):71–8.

Reimer AP, Milinovich A, Madigan EA. Data quality assessment framework to assess electronic medical record data for use in research. Int J Med Inform. 2016;90:40–7.

Johnson SG, Speedie S, Simon G, et al. Application of an ontology for characterizing data quality for a secondary use of EHR data. Appl Clin Inform. 2016;7(1):69–88.

Ozonze O, Scott PJ, Hopgood AA. Automating electronic health record data quality assessment. J Med Syst. 2023;47(1):23.

Strasser A. Delphi method variants in IS research: a taxonomy proposal. In: PAC15 2016 Proceedings. 2016. https://aisel.aisnet.org/pacis2016/224 . Accessed 5 July 2023.

Babbie ER. The Practice of Social Research. Mason, OH: CENGAGE Learning Custom Publishing; 2014.

Bryman A. Social Research Methods. London, England: Oxford University Press; 2015.

Ruan Y, Song S, Yin Z, et al. Comprehensive evaluation of military training-induced fatigue among soldiers in China: A Delphi consensus study. Front Public Health. 2022;10:1004910.

Shim JP. Bibliographical research on the analytic hierarchy process (AHP). Socio-Econ Plann Sci. 1989;23(3):161–7.

Article   Google Scholar  

Ho W. Integrated analytic hierarchy process and its applications–A literature review. Eur J Oper Res. 2008;186(1):211–28.

Lane EF, Verdini WA. A consistency test for AHP decision makers. Decis Sci. 1989;20(3):575–90.

Johnson AE, Pollard TJ, Mark RG. “MIMIC-III clinical database (version 1.4),” PhysioNet. 2016; https://doi.org/10.13026/C2XW26 .

Nusinovici S, Tham YC, Yan MYC, et al. Logistic regression was as good as machine learning for predicting major chronic diseases. J Clin Epidemiol. 2020;122:56–69.

Watanabe T, Kessler D, Scott C, et al. Disease prediction based on functional connectomes using a scalable and spatially-informed support vector machine. Neuroimage. 2014;96:183–202.

Yang L, Wu H, Jin X, et al. Study of cardiovascular disease prediction model based on random forest in eastern China. Sci Rep. 2020;10(1):5245.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine learning in Python. J Machine Learn Res. 2011;12:2825–30.

Cohen I, Huang Y, Chen J, et al. Pearson correlation coefficient. In: Noise reduction in speech processing. Heidelberg: Springer; 2009:1–4.

Hauke J, Kossowski T. Comparison of values of Pearson’s and Spearman’s correlation coefficients on the same sets of data. Quaestion Geograph. 2011;30(2):87–93.

Johnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Scientific data. 2016;3(1):1–9.

Mayr FB, Yende S, Angus DC. Epidemiology of severe sepsis. Virulence. 2014;5(1):4–11.

Lan P, Wang SJ, Shi QC, et al. Comparison of the predictive value of scoring systems on the prognosis of cirrhotic patients with suspected infection. Medicine. 2018;97(28): e11230.

Lan P, Pan K, Wang S, et al. High serum iron level is associated with increased mortality in patients with sepsis. Sci Rep. 2018;8(1):11072.

Saito T, Rehmsmeier M. Precrec: fast and accurate precision-recall and ROC curve calculations in R. Bioinformatics. 2017;33(1):145–7.

Article   CAS   PubMed   Google Scholar  

Ferencek A, Kljajić BM. Data quality assessment in product failure prediction models. J Decis Syst. 2020;29(Suppl 1):79–86.

Johnson SG, Speedie S, Simon G, et al. A data quality ontology for the secondary use of EHR data. AMIA Annu Symp Proc. 2015;2015:1937–46.

PubMed   PubMed Central   Google Scholar  

Tian Q, Chen Y, Han Z, et al. Research on evaluation indexes of clinical data quality. J Med Inform. 2020;41(10):9–17.

Weiskopf NG, Bakken S, Hripcsak G, et al. A data quality assessment guideline for electronic health record data reuse. EGEMS (Wash DC). 2017;5(1):14.

Kahn MG, Callahan TJ, Barnard J, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 2016;4(1):1244.

Cai L, Zhu Y. The challenges of data quality and data quality assessment in the big data era. Data Sci J. 2015;14:2–2.

Download references

Acknowledgements

The authors would like to thank all those who participated in the expert consultation.

This work was supported by the Chinese Academy of Medical Sciences Initiative for Innovative Medicine (Grant No. 2021-I2M-1–057 and Grant No. 2021-I2M-1–056), National Key Research and Development Program of China (Grant No. 2022YFC3601001), and National Social Science Fund of China (Grant No. 21BTQ069).

Author information

Jiayin Zhou and Jie Hao contributed equally to this work.

Authors and Affiliations

Institute of Medical Information, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, People’s Republic of China

Jiayin Zhou, Jie Hao, Mingkun Tang, Haixia Sun, Jiao Li & Qing Qian

Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, People’s Republic of China

Jiayang Wang

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to this study. QQ led and designed the study. HS designed the study and structured the manuscript. JZ drafted and revised the manuscript. JH drafted, revised the manuscript, and provided assistance with experiment interpretation. JW and MT conducted the empirical study. JL provided critical revision. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Qing Qian .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval was not deemed necessary for this study by the Ethics Committee at the Institute of Medical Information & Library, Chinese Academy of Medical Sciences, in accordance with national guidelines and local legislation. Written informed consent was obtained from all participants. All methods were performed in accordance with the relevant guidelines and regulations. This study was conducted in compliance with the Declaration of the Helsinki.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3. , rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Zhou, J., Hao, J., Tang, M. et al. Development of a quantitative index system for evaluating the quality of electronic medical records in disease risk intelligent prediction. BMC Med Inform Decis Mak 24 , 178 (2024). https://doi.org/10.1186/s12911-024-02533-z

Download citation

Received : 05 July 2023

Accepted : 13 May 2024

Published : 24 June 2024

DOI : https://doi.org/10.1186/s12911-024-02533-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Quality control
• Data management
• Disease risk prediction

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

• Open access
• Published: 27 June 2024

Chemotherapy-related cardiotoxicity and its symptoms in patients with breast cancer: a scoping review

• Hyunjoo Kim 1 , 2 ,
• Bomi Hong 3 ,
• Sanghee Kim 4 ,
• Seok-Min Kang 5 &
• Jeongok Park   ORCID: orcid.org/0000-0003-4978-817X 4  

Systematic Reviews volume  13 , Article number:  167 ( 2024 ) Cite this article

22 Accesses

Metrics details

Chemotherapy-related cardiotoxicity is a significant concern because it is a major cause of morbidity. This study aimed to provide in-depth information on the symptoms of chemotherapy-related cardiotoxicity (CRCT) by exploring literature that concurrently reports the types and symptoms of CRCT in patients with breast cancer.

A scoping review was performed according to an a priori protocol using the Joanna Briggs Institute’s guidelines. The participants were patients with breast cancer. The concept was the literature of specifically reported symptoms directly matched with CRCT and the literature, in English, from 2010, and the context was open. The search strategy included four keywords: “breast cancer,” “chemotherapy,” “cardiotoxicity,” and “symptoms.” All types of research designs were included; however, studies involving patients with other cancer types, animal subjects, and symptoms not directly related to CRCT were excluded. Data were extracted and presented including tables and figures.

A total of 29 articles were included in the study, consisting of 23 case reports, 4 retrospective studies, and 2 prospective studies. There were no restrictions on the participants’ sex; however, all of them were women, except for one case report. The most used chemotherapy regimens were trastuzumab, capecitabine, and doxorubicin or epirubicin. The primary CRCT identified were myocardial dysfunction and heart failure, followed by coronary artery disease, pulmonary hypertension, and other conditions. Major tests used to diagnose CRCT include echocardiography, electrocardiography, serum cardiac enzymes, coronary angiography, computed tomography, and magnetic resonance imaging. In all case reports, CRCT was diagnosed through an incidental checkup according to the patient’s symptom presentation; however, only 10 of these studies showed a baseline checkup before chemotherapy. The five most common CRCT symptoms were dyspnea, chest pain, peripheral edema, fatigue, and palpitations, which were assessed by patient-reported symptom presentation rather than using a symptom assessment tool. Dyspnea with trastuzumab treatment and chest pain with capecitabine treatment were particularly characteristic. The time for first symptom onset after chemotherapy ranged from 1 hour to 300 days, with anthracycline-based regimens requiring 3–55 days, trastuzumab requiring 60–300 days, and capecitabine requiring 1–7 days.

Conclusions

This scoping review allowed data mapping according to the study design and chemotherapy regimens. Cardiac assessments for CRCT diagnosis were performed according to the patient’s symptoms. There were approximately five types of typical CRCT symptoms, and the timing of symptom occurrence varied. Therefore, developing and applying a CRCT-specific and user-friendly symptom assessment tool are expected to help healthcare providers and patients manage CRCT symptoms effectively.

Peer Review reports

Breast cancer is currently the most common cancer worldwide. Its incidence and mortality rates in East Asia in 2020 accounted for 24% and 20% of the global rates, respectively, and these rates are expected to continue increasing until 2040 [ 1 ]. In the USA, since the mid-2000s, the incidence rate of breast cancer has been increasing by 0.5% annually, while the mortality rate has been decreasing by 1% per year from 2011 to 2020 [ 2 ]. Despite the improved long-term survival rate in patients with breast cancer due to the development of chemotherapy, the literature has highlighted that cardiotoxicity, a cardiac problem caused by chemotherapy, could be a significant cause of death among these patients [ 3 ]. Chemotherapy-related cardiotoxicity (CRCT) can interfere with cancer treatment and progress to congestive heart failure during or after chemotherapy [ 4 ], potentially lowering the survival rate and quality of life of patients with cancer [ 5 ].

The term cardiotoxicity was first used in the 1970s to describe cardiac complications resulting from chemotherapy regimens, such as anthracyclines and 5-fluorouracil. The early definition of cardiotoxicity centered around heart failure, but the current definition is broad and still imprecise [ 6 ]. The 2022 guidelines on cardio-oncology from the European Society of Cardiology (ESC) define cardiotoxicity as including cardiac dysfunction, myocarditis, vascular toxicity, arterial hypertension, and cardiac arrhythmias. Some of these definitions reflect the symptoms. For example, cardiac dysfunction, which accounts for 48% of cardiotoxicity in patients with cancer, is divided into asymptomatic and symptomatic cardiac dysfunction. Asymptomatic cardiac dysfunction is defined based on left ventricular ejection fraction (LVEF), myocardial global longitudinal strain, and cardiac biomarkers. Symptomatic cardiac dysfunction indicates heart failure and presents with ankle swelling, breathlessness, and fatigue [ 7 ]. The ESC guidelines for heart failure present more than 20 types of symptoms [ 8 ]; however, to the best of our knowledge, few studies have been conducted to determine which heart failure symptoms and their characteristics are associated with CRCT in patients with breast cancer. Similarly, there is a lack of information related to vascular toxicity such as myocardial infarction [ 7 ].

Professional societies in cardiology and oncology have proposed guidelines for the prevention and management of cardiotoxicity in patients with cancer. According to the American Society of Clinical Oncology and the ESC, it is recommended to identify high-risk patients, comprehensively evaluate clinical signs and symptoms associated with CRCT, and conduct cardiac evaluations before, during, and after chemotherapy [ 7 , 9 , 10 ]. In addition, guidelines for patients with cancer, including those for breast cancer survivorship care, emphasize that patients should be aware of the potential risk of CRCT and report symptoms, such as fatigue or shortness of breath to their healthcare providers [ 7 , 11 , 12 ]. Although these guidelines encompass cardiac monitoring as well as symptom observation, many studies have focused solely on objective diagnostic tests, such as echocardiography, cardiac magnetic resonance, and cardiac biomarkers [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ], which means that there is little interest in CRCT symptoms in patients under breast cancer care.

This lack of interest in CRCT symptoms may be related to the absence of a specific symptom assessment tool for CRCT. Symptom monitoring of CRCT in patients with breast cancer was conducted through patient interviews and reported using the appropriate terminology [ 23 ]. In terms of interviews, patients with cancer experienced the burden of expressing symptoms between cardiovascular problems and cancer treatment. Qualitative research on patients with cancer indicates that these patients experience a daily battle to distinguish the symptoms they experience during chemotherapy [ 24 ]. To reduce the burden of identifying CRCT symptoms, it is crucial to educate patients with breast cancer undergoing chemotherapy about these symptoms. To report cardiotoxicity, healthcare providers in oncology can use a dictionary of terms called the Common Terminology Criteria for Adverse Events (CTCAE) for reporting adverse events in patients with cancer [ 25 ]. Patients can also use Patient-Reported Outcome (PRO), which allows unfiltered reporting of symptoms directly to the clinical database [ 26 ]. PRO consists of 78 symptomatic adverse events out of approximately 1,000 types of CTCAE [ 27 ]. Basch et al. suggested that PRO could enable healthcare providers to identify patient symptoms before they worsen, thereby improving the overall survival rate of patients with metastatic cancer [ 28 ]. This finding implies that symptoms can provide valuable clues for enhancing the timeliness and accuracy of clinical assessments of CRCT [ 29 ]. Therefore, it is necessary to explore the scope of research focusing on CRCT symptoms for prevention and early detection of CRCT in patients with breast cancer. The detailed research questions are as follows:

What are the general characteristics of the studies related to CRCT in patients with breast cancer?

What diagnostic tools and monitoring practices are used to detect CRCT?

What are the characteristics and progression of symptoms associated with CRCT?

A scoping review is a research method for synthesizing evidence that involves mapping the scope of evidence on a particular topic [ 30 ]. It aims to clarify key concepts and definitions, identify key characteristics of factors related to a concept, and highlight gaps or areas for further research [ 30 ]. This study used a scoping review methodology based on the Joanna Briggs Institute (JBI) framework. The JBI methodology, refined from the framework initially developed by Arksey and O’Malley [ 31 ], involves developing a research question, establishing detailed inclusion and exclusion criteria, and selecting and analyzing literature accordingly [ 32 ]. In contrast to systematic reviews, scoping reviews can encompass a variety of study designs and are particularly suitable when the topic has not been extensively studied [ 33 ]; hence, the decision was made to conduct a scoping review.

Development of a scoping review protocol

To conduct this review, an a priori scoping review protocol was developed to enhance transparency and increase the usefulness and reliability of the results. The protocol included the title, objective, review questions, introduction, eligibility criteria, participants, concept, context, types of evidence source, methods, search strategy, source of evidence selection, data extraction, data analysis and presentation, and deviation from the protocol [ 34 ] (Supplementary File 1).

Eligibility criteria

A participant-concept-context (PCC) framework was constructed based on the following research criteria. The participants were patients with breast cancer. The concept was that studies that specifically reported symptoms directly matched to CRCT in patients with breast cancer and the literature, published in English since 2010, in line with the year the CRCT guidelines were announced by the Cardio-Oncology Society. The context was open. We included all types of research designs. The exclusion criteria were studies that included patients with other types of cancer, involved animal subjects, and reported symptoms not directly related to CRCT.

Search strategy

The keywords consisted of “breast cancer,” “chemotherapy,” “cardiotoxicity,” and “symptoms.” The keywords for “cardiotoxicity” were constructed according to the clinical cardiotoxicity report and ESC guidelines [ 7 , 35 ]. The keywords for “symptoms” included 40 specific symptoms of arrhythmia, heart failure, and cardiac problems [ 36 , 37 ] (Supplementary Table 1). We used PubMed, Embase, and CINAHL.

Source of evidence selection

Duplicate studies were removed using EndNote 21. The titles and abstracts were then reviewed according to the inclusion criteria, the primary literature was selected, and the final literature was selected through a full-text review. Any disagreements were resolved through discussions between the investigators.

Data extraction

The data from the literature included the general characteristics of the study, as well as information on the patients, chemotherapy, cardiotoxicity, and symptoms. The general characteristics of the study included author, publication year, country of origin, study design; patient information including sample size, sex, age, cancer type, and cancer stage; chemotherapy information including chemotherapy regimen; cardiotoxicity information including type of cardiotoxicity, diagnostic tests, and times of assessment; and symptom information including type of symptom, characteristics of symptom worsening or improvement, onset time, progression time, and time to symptom improvement. Information on whether to receive chemotherapy after the diagnosis of cardiotoxicity was explored.

Data analysis and presentation

The contents of the included studies were divided into three categories: (1) general characteristics, which encompassed study designs, patients, and medications; (2) type of CRCT and cardiac assessment for CRCT; and (3) characteristics and progression of the symptoms associated with CRCT. CRCT symptom-related data are presented in tables and figures.

In total, 487 studies were identified through database searches, and 116 duplicates were subsequently removed. After reviewing the titles and abstracts, we excluded 197 studies in which participants had cancers other than breast cancer, no symptoms, or symptom-related expressions. Of the remaining 174 studies, 146 were excluded after full-text review. Among the excluded studies, 79 were mainly clinical trials that the symptoms were not directly related to CRCT, 62 did not report specific symptoms, four were in the wrong population, and one was unavailable for full-text review. An additional study was included after a review of references, bringing the final count to 29 studies included in the analysis (Fig. 1 ).

Preferred reporting items for systematic reviews flowchart

General characteristics of studies including designs, sex and age, chemotherapy regimen, and CRCT criteria

Table 1 presents the general characteristics of the studies included in this review. The majority of these studies were published in the USA ( n =14), with Japan ( n =3), and Romania ( n =2) following. The study designs primarily consisted of case reports ( n =23), retrospective studies ( n =4), and prospective studies ( n =2).

All case reports involved female patients, except for one involving a male patient. Five quantitative studies did not specify or limit the sex of the participants, and one retrospective study included only female patients. In terms of cancer stage, the majority of studies involved patients with advanced breast cancer ( n =13), while a smaller number involved patients with early-stage breast cancer ( n =4). Twelve studies did not specify the cancer stage. Approximately 20 types of chemotherapy regimens are currently in use. Trastuzumab, which is a human epidermal growth factor receptor 2 (HER2) blocker, was mentioned in the majority of studies ( n =8), followed by capecitabine (an antimetabolite) ( n =7), and doxorubicin or epirubicin (anthracycline-based chemotherapy) ( n =6). Current chemotherapy and previous treatment methods were described together, with the exception of eight studies. Six quantitative studies defined the CRCT criteria, five of which were based on decreased LVEF and one of which was based on significant cardiac symptoms and/or electrocardiogram changes. Twenty-three case reports described the cardiovascular diagnosis as CRCT.

Diagnostic tools and monitoring practice for CRCT

Table 2 displays the types of CRCT, diagnostic tools, and times of cardiac assessment according to chemotherapy regimens. The most prevalent CRCT were myocardial dysfunction and heart failure, identified in 12 case studies, respectively. This was followed by coronary artery disease, represented in 8 case studies, pulmonary hypertension in 2 case studies, and a single case study of periaortitis. The most used test for diagnosing CRCT was echocardiography ( n =22), followed by EKG ( n =20), various types of cardiac enzymes ( n =16), coronary angiography (CAG, n =12), computed tomography ( n =6), and magnetic resonance imaging (MRI, n =4). Regarding the CRCT symptom assessment tools, the CTCAE was used in two studies, the New York Heart Association classification for heart failure in two studies, the dyspnea assessment scale in one study, and symptoms of cardiac origin, which consisted of chest pain, dyspnea, and palpitations in one study.

Regarding the times of cardiac evaluation, two studies performed regular cardiac checkups including before, during, and after chemotherapy. There were 10 case studies and six quantitative studies describing cardiac function testing before chemotherapy, of which seven studies performed regular cardiac screening tests and two studies mentioned cardiac screening even after the completion of chemotherapy. The frequency of regular checkups varied from every 3 months to every two to four cycles. In all case reports ( n =23), CRCT were diagnosed through incidental checkups based on patients’ symptom presentation, and in most cases, several tests were performed subsequentially for CRCT diagnosis. In one case study, cardiac evaluation was conducted 3 days after the patient’s initial symptom presentation, when the symptoms became more severe.

Characteristics and progression of symptoms associated with CRCT

Table 3 shows the descriptive scope of the CRCT-related symptoms according to the chemotherapy regimens used in the included studies. The mapping factors included initial symptoms, symptom onset or severity, symptom progression, medical management, and CRCT results. One of the most frequent symptoms associated with CRCT was dyspnea, which was discussed in 19 studies and described as difficulty in breathing, shortness of breath, or New York Heart Association (NYHA) class II or III. When dyspnea appeared as the initial symptom of CRCT, the symptom progression was worsening in eight case studies and persistent in two cases. Chest pain was described in 12 studies as a symptom characterized by a squeezing, tingling, burning, tightened, or atypical feeling that was relieved by rest and exacerbated by exertion. Other symptoms included peripheral edema ( n =6), fatigue ( n =5), and palpitation ( n =2). The symptoms were assessed by patient-reported symptom presentation rather than using a symptom assessment tool.

The symptoms could be categorized based on the type of chemotherapy regimens used. In the case studies involving anthracycline-based regimen and HER2 blockers, dyspnea was the most frequently observed symptom ( n =7), followed by peripheral edema ( n =2), and chest pain or discomfort ( n =2). In case studies where antimetabolites were used, specifically capecitabine, chest pain was a common and prominent symptom. This chest pain typically manifested between 1 and 7 days after drug administration and persisted until treatment. Notably, four out of seven patients reported this symptom on the first day of chemotherapy, according to the case reports. The time for first symptom onset after chemotherapy ranged from 1 hour to 300 days, with anthracycline-based regimens requiring 3–55 days, trastuzumab requiring 60–300 days, and capecitabine requiring 1–7 days. Figure 2 shows the progression of symptoms in case studies, detailing the time of symptom onset, the date of symptom reporting, and the date of treatment completion following the use of chemotherapy. The studies that did not specify any of the dates of symptom onset, reporting, and completion of treatment were excluded from the figure.

Figure 3 shows symptoms according to the main types of chemotherapy regimens reported in case studies. Dyspnea with trastuzumab and chest pain with capecitabine are particularly characteristic. A retrospective study included in this scoping review reported that chest pain was the most common symptom associated with capecitabine, followed by dyspnea and palpitation [ 40 ]. Furthermore, peripheral edema was primarily observed with anthracycline, alkylating, and HER2 blockers, while fatigue was noted with various anticancer drugs, irrespective of the type of chemotherapy regimen.

Ongoing chemotherapy was discontinued after CRCT was detected in 20 case studies. When patients presented symptoms indicative of CRCT, the majority were promptly hospitalized for further evaluation, medication, or interventional treatment. The majority of studies indicated the initiation of cardiac medication ( n =21), with three case studies involving coronary intervention and two involving treatment with wearable devices. Most management procedures were conducted in a general ward or an intensive care unit.

In most case studies, symptoms improved following cardiac treatment, with either complete or partial recovery of LVEF observed in 19 instances. However, a few studies reported a poor prognosis, including two instances of death. LVEF recovered in most patients within 6 months when treated with an anthracycline-based regimen and HER2 blockers (Fig. 2 ). A retrospective study reported that the rates of complete or partial recovery of CRCT following treatment with doxorubicin-based chemotherapy and trastuzumab were 42.9% and 86.1%, respectively [ 39 ]. Another retrospective study noted that the recovery time of CRCT when treated with HER2 blockers increased in correlation with the severity of the NYHA class, ranging from 8 to 80 weeks [ 38 ]. In the case of the antimetabolite capecitabine, all patients recovered within a day to a week, except one patient who did not recover.

This scoping review was conducted to explore the scope of studies focusing on CRCT symptoms, including the general characteristics of the studies, diagnostic tools, monitoring practices related to detecting CRCT, and the characteristics and progression of symptoms associated with CRCT. The primary findings of this review were as follows: (1) common symptoms related to CRCT and differences in symptoms according to the chemotherapy regimens used were identified; (2) the symptoms reported by the patient served as clues to suspect a specific type of CRCT; and (3) regular monitoring practices for CRCT prevention and detection were insufficient.

First, the current study identified common symptoms such as dyspnea, chest pain, peripheral edema, fatigue, and palpitation associated with CRCT, as well as variations in symptoms depending on the chemotherapy regimen used in patients with breast cancer. Among these symptoms, dyspnea, edema, and chest pain were frequently observed in patients receiving anthracycline-based and/or HER2 blocker drugs. These symptoms, which are associated with heart failure, appeared later compared to those observed with capecitabine, as depicted in Fig. 2 . This may be due to the known impact of anthracycline-based and/or HER2 blocker regimens on cardiomyocytes and other cells, leading to myocardial damage [ 42 ]. Therefore, the symptoms are related to heart failure, potentially resulting from the impairment of ventricular filling or ejection in patients undergoing treatment with these regimens [ 43 ].

In a similar vein, Attin et al. (2022) documented the occurrence of symptoms such as lower extremity edema, chest pain, difficulty breathing, and fatigue before the diagnosis of CRCT in women undergoing breast cancer treatment. They conducted a retrospective and longitudinal investigation of the symptoms, signs, and cardiac tests of 15 patients who experienced CRCT, using their electronic medical records. In their study, cardiotoxicity was defined by an echocardiogram or MRI showing a decrease in LVEF of 5 to 10%, with a specialist’s confirmation note. They compared the number of symptom occurrences during the first half of the year with those during the second half of the year prior to the diagnosis of cardiotoxicity. Specifically, the frequency of lower-extremity edema significantly increased from three occurrences in the first half of the year to 17 occurrences in the second half of the year. The frequency of symptoms for dyspnea and chest pain also increased from 10 and 8 times, respectively, to 16 and 14 times in the second half of the year. While there was limited information on the doses or timing of chemotherapy, 87% of the patients received the same chemotherapy regimens, namely anthracyclines and/or HER2 blockers [ 44 ]. This suggests that the increase in symptom occurrence over time may be related to the accumulation of anthracycline and the duration of anti-HER2 therapy [ 45 ].

Salyer et al. (2019) conducted a study on the prevalent symptoms of heart failure and their clustering. They identified three symptom clusters: sickness behavior, gastrointestinal disturbance, and discomfort of illness. Notably, dyspnea, edema, and pain were grouped into the discomfort of illness cluster, which aligns with the symptoms we observed in patients treated with anthracyclines and/or HER2 blockers [ 46 ]. Therefore, it is crucial for patients undergoing treatment with anthracyclines and/or HER2 blockers to be vigilant for symptoms such as dyspnea, edema, or chest pain, as these are indicative of heart failure.

Chest pain caused by vasospasm was a predominant symptom in patients taking antimetabolite regimens such as oral capecitabine, and it manifested as the following types of cardiotoxicities: vasospasm-related arrhythmia, myocardial disease, and ischemia [ 47 ]. Vasospasm can be triggered by endothelial dysfunction, hypersensitive vascular smooth muscle, reactive oxidative stress, or chemotherapy regimens [ 48 , 49 ]. According to previous studies, in patients using antimetabolite drugs such as 5-fluorouracil or capecitabine, chest pain was usually reported to occur from several hours to 72 hours after the first administration [ 47 , 50 , 51 , 52 , 53 ]. To detect chemotherapy-related coronary vasospasm in the early stage, it is recommended to carefully monitor typical or atypical symptoms of chest pain and EKG monitoring during drug infusion [ 54 ]. Muco et al. (2022) reported severe outcomes resulting from delayed management of vasospastic angina symptoms. The patient’s cardiac evaluation was performed 3 days after the onset of symptoms, and unfortunately, she did not recover from brain damage caused by coronary vasospastic sequelae. The authors stressed the importance of medical teams recognizing the symptoms of CRCT through vigilant monitoring and patient education [ 55 ].

As seen in the symptoms of CRCT caused by heart failure and vasospasm, careful observation of symptoms and conducting appropriate tests are crucial to prevent cardiotoxicity and minimize damage. These characteristics of CRCT and the associated symptoms related to chemotherapy regimens can provide crucial educational content for healthcare providers and patients preparing for chemotherapy. In addition, CRCT and symptom progression according to chemotherapy regimens could be used to formulate research questions for future systematic reviews.

Second, the preventive management of CRCT necessitates adherence to recommended guidelines. The 2022 ESC guidelines on cardio-oncology have updated the classification of CRCT and the monitoring protocols based on the chemotherapy regimens used [ 7 ]. The CRCT identified in the current study aligns with the drug toxicity outlined in the 2022 ESC guidelines. These guidelines advocate for regular cardiac monitoring before, during, and after chemotherapy to prevent and manage CRCT induced by anthracycline and HER2 blockers [ 7 , 12 ]. In this scoping review, two of 23 records described cardiac monitoring before, during, and after chemotherapy. An Australian multicenter study revealed that 59% of patients were referred to a cardiologist before CRCT occurred, but only 15% of patients diagnosed with CRCT had consulted a cardiologist before chemotherapy [ 41 ]. Given the declining mortality rates among cancer patients, managing CRCT requires a collaborative approach between oncology and cardiology to minimize mortality and morbidity in patients with breast cancer undergoing chemotherapy [ 7 ]. Therefore, it remains crucial to emphasize adherence to cardiac monitoring guidelines and foster cooperation between oncology and cardiology.

Additionally, symptom assessment is important for the early detection of patients with CRCT. The studies included in the current scoping review assessed whether patients’ symptoms could detect CRCT using interviews with patients, the New York Heart Association classification, a dyspnea assessment scale, and CTCAE tools. The United States National Cancer Institute recommends that healthcare providers use CTCAE and patients with cancer use PRO to report adverse events, including symptoms. CTCAE is a broad and comprehensive terminology that encompasses adverse events related to cancer treatment, has been used since the 1980s [ 25 ], and is not specialized in cardiotoxicity. Additionally, a discrepancy between CTCAE and PRO discovered that healthcare providers often underestimate both the incidence and duration of symptoms compared to the patients [ 56 , 57 , 58 ]. Specifically, healthcare providers tend to report symptom severity as lower than that reported by patients. For instance, there are notable discrepancies between healthcare providers and patients when reporting severe or very severe symptoms of fatigue, dyspnea, and limb edema in patients with early-stage breast cancer undergoing chemotherapy. The reported rates were 8% and 22% for fatigue, 0% and 4% for dyspnea, and 0% and 5% for limb edema, from healthcare providers and patients, respectively. Therefore, it is necessary to develop a user-friendly questionnaire to assess the various symptoms of CRCT.

Finally, we found that once CRCT was confirmed, cardiac treatment was promptly initiated and chemotherapy was frequently halted until CRCT resolution. A Delphi study on the use of anthracycline and trastuzumab proposed altering the treatment schedule or discontinuing treatment until there was an improvement in LVEF [ 59 ]. However, the professional societies did not provide definitive recommendations regarding continuing or ceasing ongoing chemotherapy. Instead, they suggested that the decision to continue or discontinue ongoing chemotherapy should be made based on the patient’s potential risks and benefits [ 60 ]. For example, Polk et al. (2016) reported that out of 22 patients with CRCT resulting from capecitabine, six continued medications with or without cardiac treatment; some of these patients experienced the same symptoms, while others did not exhibit significant symptoms [ 40 ]. Further research is required to explore the continuation or discontinuation of chemotherapy when CRCT is confirmed.

This study has some limitations. First, although we did not restrict the patients’ sex when reviewing the literature, most patients, except for one, were female. This may be related to the lower incidence of breast cancer in men. Second, although this scoping review mapped CRCT symptoms according to chemotherapy regimens, including anthracycline-based drugs, HER2 blockers, and antimetabolites, it did not cover cardiotoxicity related to other types of chemotherapy regimens. Thus, exploring the symptoms by focusing on expanded chemotherapy regimens and cardiovascular toxic diseases will assist in overcoming this limitation. Third, of the 29 studies, 23 were case reports with some grey literature, which may be justified by the nature of scoping reviews that allow for inclusion irrespective of the data source [ 61 ] and the study type. Experimental or observational clinical studies use objective criteria, such as diagnostic tests to generate primary evidence. However, case reports have led to new medical discoveries regarding the prevention and treatment of diseases [ 62 ]. Given the nature of case reports, specific symptoms that could provide clues for evaluating CRCT in patients with breast cancer are most often found in these reports. We incorporated grey literature to gather more comprehensive information on CRCT-related symptoms. However, to mitigate the potential issue of unverified quality in grey literature, we initially organized 16 studies from peer-reviewed literature and subsequently incorporated the grey literature into our findings. This approach helped to clarify the results of the peer-reviewed literature, particularly the types of chemotherapy regimens [ 63 ]. Finally, regarding the literature selection criteria, we examined articles written in English and published since 2010, the year the cardio-oncology guidelines were announced, thereby excluding articles published before 2010.

This scoping review allowed data mapping according to the study design and chemotherapy regimens. The key messages included a type of CRCT, cardiac assessment, and in-depth information regarding the CRCT symptoms. There were approximately five typical CRCT symptoms, including dyspnea, chest pain, peripheral edema, fatigue, and palpitations, and the timing of symptom occurrence varied. The symptoms were assessed by patient-reported symptom presentation rather than using a symptom assessment tool. Therefore, developing and applying a CRCT-specific and user-friendly symptom assessment tool are expected to help healthcare providers and patients manage CRCT symptoms effectively.

Availability of data and materials

The datasets generated during and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne M, Vignat J, Gralow JR, Cardoso F, Siesling S, Soerjomataram I. Current and future burden of breast cancer: global statistics for 2020 and 2040. The Breast. 2022;66:15–23.

Article   PubMed   PubMed Central   Google Scholar  

Siegel RL, Miller KD, Wagle NS, Jemal A. Cancer statistics, 2023. CA Cancer J Clin. 2023;73(1):17–48.

Article   PubMed   Google Scholar  

Agha A, Wang X, Wang M, Lehrer EJ, Horn SR, Rosenberg JC, Trifiletti DM, Diaz R, Louie AV, Zaorsky NG. Long-term risk of death from heart disease among breast cancer patients. Front Cardiovasc Med. 2022;9: 784409.

Oikawa M, Ishida T, Takeishi Y. Cancer therapeutics-related cardiovascular dysfunction: Basic mechanisms and clinical manifestation. J Cardiol. 2023;81(3):253–9.

Piepoli MF, Adamo M, Barison A, Bestetti RB, Biegus J, Böhm M, Butler J, Carapetis J, Ceconi C, Chioncel O, et al. Preventing heart failure: a position paper of the Heart Failure Association in collaboration with the European Association of Preventive Cardiology. Eur J Heart Fail. 2022;24(1):143–68.

Chung R, Ghosh AK, Banerjee A: Cardiotoxicity: precision medicine with imprecise definitions. In., vol. 5: Archives of Disease in childhood; 2018: e000774.

Lyon AR, López-Fernández T, Couch LS, Asteggiano R, Aznar MC, Bergler-Klein J, Boriani G, Cardinale D, Cordoba R, Cosyns B, et al. 2022 ESC Guidelines on cardio-oncology developed in collaboration with the European Hematology Association (EHA), the European Society for Therapeutic Radiology and Oncology (ESTRO) and the International Cardio-Oncology Society (IC-OS): developed by the task force on cardio-oncology of the European Society of Cardiology (ESC). European Heart Journal - Cardiovascular Imaging. 2022;23(10):e333–465.

McDonagh TA, Metra M, Adamo M, Gardner RS, Baumbach A, Böhm M, Burri H, Butler J, Čelutkienė J, Chioncel O et al: 2021 ESC Guidelines for the diagnosis and treatment of acute and chronic heart failure: developed by the Task Force for the diagnosis and treatment of acute and chronic heart failure of the European Society of Cardiology (ESC). With the special contribution of the Heart Failure Association (HFA) of the ESC. Eur J Heart Fail 2022, 24(1):4-131.

Armenian SH, Lacchetti C, Lenihan D. Prevention and monitoring of cardiac dysfunction in survivors of adult cancers: American Society of Clinical Oncology Clinical Practice Guideline Summary. J Oncol Pract. 2017;13(4):270–5.

Lanza O, Ferrera A, Reale S, Solfanelli G, Petrungaro M, Tini Melato G, Volpe M, Battistoni A: New insights on the toxicity on heart and vessels of breast cancer therapies. Med Sci (Basel) 2022, 10(2).

Runowicz CD, Leach CR, Henry NL, Henry KS, Mackey HT, Cowens-Alvarado RL, Cannady RS, Pratt-Chapman ML, Edge SB, Jacobs LA, et al. American Cancer Society/American Society of Clinical Oncology Breast Cancer Survivorship Care Guideline. CA Cancer J Clin. 2016;66(1):43–73.

Lee GA, Aktaa S, Baker E, Gale CP, Yaseen IF, Gulati G, Asteggiano R, Szmit S, Cohen-Solal A, Abdin A, et al. European Society of Cardiology quality indicators for the prevention and management of cancer therapy-related cardiovascular toxicity in cancer treatment. Eur Heart J Qual Care Clin Outcomes. 2022;9(1):1–7.

Alexandraki A, Papageorgiou E, Zacharia M, Keramida K, Papakonstantinou A, Cipolla CM, Tsekoura D, Naka K, Mazzocco K, Mauri D et al: New insights in the era of clinical biomarkers as potential predictors of systemic therapy-induced cardiotoxicity in women with breast cancer: a systematic review. Cancers (Basel) 2023, 15(13).

Di Lisi D, Manno G, Madaudo C, Filorizzo C, Intravaia RCM, Galassi AR, Incorvaia L, Russo A, Novo G: Chemotherapy-related cardiac dysfunction: the usefulness of myocardial work indices. Int J Cardiovasc Imaging 2023.

Kar J, Cohen MV, McQuiston SA, Malozzi CM. Can global longitudinal strain (GLS) with magnetic resonance prognosticate early cancer therapy-related cardiac dysfunction (CTRCD) in breast cancer patients, a prospective study? Magn Reson Imaging. 2023;97:68–81.

Article   CAS   PubMed   Google Scholar  

Lim A, Jang H, Jeon M, Fadol AP, Kim S. Cancer treatment-related cardiac dysfunction in breast cancer survivors: a retrospective descriptive study using electronic health records from a Korean tertiary hospital. Eur J Oncol Nurs. 2022;59: 102163.

Liu W, Li W, Li H, Li Z, Zhao P, Guo Z, Liu C, Sun L, Wang Z. Two-dimensional speckle tracking echocardiography help identify breast cancer therapeutics-related cardiac dysfunction. BMC Cardiovasc Disord. 2022;22(1):548.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mauro C, Capone V, Cocchia R, Cademartiri F, Riccardi F, Arcopinto M, Alshahid M, Anwar K, Carafa M, Carbone A et al: Cardiovascular side effects of anthracyclines and HER2 inhibitors among patients with breast cancer: a multidisciplinary stepwise approach for prevention, early detection, and treatment. J Clin Med 2023, 12(6).

Okushi Y, Saijo Y, Yamada H, Toba H, Zheng R, Seno H, Takahashi T, Ise T, Yamaguchi K, Yagi S et al: Effectiveness of surveillance by echocardiography for cancer therapeutics-related cardiac dysfunction of patients with breast cancer. J Cardiol 2023.

Ositelu K, Trevino A, Tong A, Chen MH, Akhter N: Challenges in cardiovascular imaging in women with breast cancer. Curr Cardiol Rep 2023.

Terui Y, Sugimura K, Ota H, Tada H, Nochioka K, Sato H, Katsuta Y, Fujiwara J, Harada-Shoji N, Sato-Tadano A, et al. Usefulness of cardiac magnetic resonance for early detection of cancer therapeutics-related cardiac dysfunction in breast cancer patients. Int J Cardiol. 2023;371:472–9.

Thavendiranathan P, Shalmon T, Fan CS, Houbois C, Amir E, Thevakumaran Y, Somerset E, Malowany JM, Urzua-Fresno C, Yip P, et al. Comprehensive cardiovascular magnetic resonance tissue characterization and cardiotoxicity in women with breast cancer. JAMA Cardiol. 2023;8(6):524–34.

Trotti A, Colevas AD, Setser A, Basch E. Patient-reported outcomes and the evolution of adverse event reporting in oncology. J Clin Oncol. 2007;25(32):5121–7.

White J, Byles J, Williams T, Untaru R, Ngo DTM, Sverdlov AL. Early access to a cardio-oncology clinic in an Australian context: a qualitative exploration of patient experiences. Cardiooncology. 2022;8(1):14.

PubMed   PubMed Central   Google Scholar  

Trotti A, Colevas AD, Setser A, Rusch V, Jaques D, Budach V, Langer C, Murphy B, Cumberlin R, Coleman CN, Rubin P: CTCAE v3.0: development of a comprehensive grading system for the adverse effects of cancer treatment. Semin Radiat Oncol 2003, 13(3):176-181.

Basch E, Reeve BB, Mitchell SA, Clauser SB, Minasian LM, Dueck AC, Mendoza TR, Hay J, Atkinson TM, Abernethy AP et al: Development of the National Cancer Institute’s patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). J Natl Cancer Inst 2014, 106(9).

Kluetz PG, Chingos DT, Basch EM, Mitchell SA. Patient-reported outcomes in cancer clinical trials: measuring symptomatic adverse events with the National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Am Soc Clin Oncol Educ Book. 2016;36:67–73.

Article   Google Scholar  

Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, Schrag D. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA. 2017;318(2):197–8.

Liu L, Suo T, Shen Y, Geng C, Song Z, Liu F, Wang J, Xie Y, Zhang Y, Tang T, et al. Clinicians versus patients subjective adverse events assessment: based on patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). Qual Life Res. 2020;29(11):3009–15.

Munn Z, Pollock D, Khalil H, Alexander L, Mclnerney P, Godfrey CM, Peters M, Tricco AC. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evidence Synthesis. 2022;20(4):950–2.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1):19–32.

Peters M, Godfrey C, McInerney P, Munn Z, Tricco A, Khalil H: Chapter 11: scoping reviews (2020 version). In: JBI Manual for Evidence Synthesis. edn. Edited by Aromataris E MZ: JBI; 2020.

Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC medical research methodology. 2018;18:1–7.

Peters MDJ, Godfrey C, McInerney P, Khalil H, Larsen P, Marnie C, Pollock D, Tricco AC, Munn Z. Best practice guidance and reporting items for the development of scoping review protocols. JBI Evid Synth. 2022;20(4):953–68.

Bohdan M, Kowalczys A, Mickiewicz A, Gruchala M, Lewicka E: Cancer therapy-related cardiovascular complications in clinical practice: current perspectives. J Clin Med 2021, 10(8).

Priori SG, Wilde AA, Horie M, Cho Y, Behr ER, Berul C, Blom N, Brugada J, Chiang CE, Huikuri H, et al. HRS/EHRA/APHRS expert consensus statement on the diagnosis and management of patients with inherited primary arrhythmia syndromes: document endorsed by HRS, EHRA, and APHRS in May 2013 and by ACCF, AHA, PACES, and AEPC in June 2013. Heart Rhythm. 2013;10(12):1932–63.

Bozkurt B, Coats A, Tsutsui H: Universal definition and classification of heart failure. J Card Fail 2021.

Aldiab A. Cardiotoxicity with adjuvant trastuzumab use in breast cancer: a single institution»s experience. J Saudi Heart Assoc. 2010;22(3):133–6.

Russell SD, Blackwell KL, Lawrence J, Pippen JE Jr, Roe MT, Wood F, Paton V, Holmgren E, Mahaffey KW. Independent adjudication of symptomatic heart failure with the use of doxorubicin and cyclophosphamide followed by trastuzumab adjuvant therapy: a combined review of cardiac data from the National Surgical Adjuvant Breast and Bowel Project B-31 and the North Central Cancer Treatment Group N9831 clinical trials. J Clin Oncol. 2010;28(21):3416–21.

Polk A, Shahmarvand N, Vistisen K, Vaage-Nilsen M, Larsen FO, Schou M, Nielsen DL: Incidence and risk factors for capecitabine-induced symptomatic cardiotoxicity: a retrospective study of 452 consecutive patients with metastatic breast cancer. BMJ Open 2016, 6(10).

Clark RA, Marin TS, McCarthy AL, Bradley J, Grover S, Peters R, Karapetis CS, Atherton JJ, Koczwara B. Cardiotoxicity after cancer treatment: a process map of the patient treatment journey. Cardiooncology. 2019;5:14.

Anjos M, Fontes-Oliveira M, Costa VM, Santos M, Ferreira R. An update of the molecular mechanisms underlying doxorubicin plus trastuzumab induced cardiotoxicity. Life Sci. 2021;280: 119760.

Malik A, Brito D, Vaqar S, Chhabra L: Congestive heart failure. In: StatPearls. edn. Treasure Island (FL): StatPearls Publishing. Copyright © 2023, StatPearls Publishing LLC.; 2023.

Attin M, Reifenstein K, Mehta S, Arcoleo K, Lin CD, Storozynsky E. Reported signs, symptoms, and diagnostic tests before cardiotoxicity among women with breast cancer: a pilot study. J Cardiovasc Nurs. 2022;37(2):104–11.

Huang P, Dai S, Ye Z, Liu Y, Chen Z, Zheng Y, Shao X, Lei L, Wang X. Long-term tolerance and cardiac function in breast cancer patients receiving trastuzumab therapy. Oncotarget. 2017;8(2):2069–75.

Salyer J, Flattery M, Lyon DE. Heart failure symptom clusters and quality of life. Heart Lung. 2019;48(5):366–72.

Padegimas A, Carver JR. How to diagnose and manage patients with fluoropyrimidine-induced chest pain: a single center approach. JACC CardioOncol. 2020;2(4):650–4.

Sheth MA, Widmer RJ, Dandapantula HK. Pathobiology and evolving therapies of coronary artery vasospasm. Proc (Bayl Univ Med Cent). 2021;34(3):352–60.

PubMed   Google Scholar  

Hokimoto S, Kaikita K, Yasuda S, Tsujita K, Ishihara M, Matoba T, Matsuzawa Y, Mitsutake Y, Mitani Y, Murohara T, et al. JCS/CVIT/JCC 2023 guideline focused update on diagnosis and treatment of vasospastic angina (coronary spastic angina) and coronary microvascular dysfunction. J Cardiol. 2023;82(4):293–341.

Kanduri J, More LA, Godishala A, Asnani A. Fluoropyrimidine-associated cardiotoxicity. Cardiol Clin. 2019;37(4):399–405.

Garbis K, Rafiee MJ, Luu J. 5-fluorouracil-induced coronary vasospasm: a cardiovascular magnetic resonance imaging case report. Glob Cardiol Sci Pract. 2023;2023(3): e202316.

Dyhl-Polk A, Vaage-Nilsen M, Schou M, Vistisen KK, Lund CM, Kümler T, Appel JM, Nielsen DL. Incidence and risk markers of 5-fluorouracil and capecitabine cardiotoxicity in patients with colorectal cancer. Acta Oncol. 2020;59(4):475–83.

Becker K, Erckenbrecht JF, Häussinger D, Fueling T. Cardiotoxicity of the antiprolif erative compound fluorouracil. Drugs. 1999;57(4):475–84.

Lestuzzi C, Vaccher E, Talamini R, Lleshi A, Meneguzzo N, Viel E, Scalone S, Tartuferi L, Buonadonna A, Ejiofor L, Schmoll HJ. Effort myocardial ischemia during chemotherapy with 5-fluorouracil: an underestimated risk. Ann Oncol. 2014;25(5):1059–64.

Muco E, Patail H, Shaik A, McMahon S. Capecitabine-associated coronary vasospasm and cardiac arrest. Cureus. 2022;14(8): e28184.

Montemurro F, Mittica G, Cagnazzo C, Longo V, Berchialla P, Solinas G, Culotta P, Martinello R, Foresto M, Gallizioli S, et al. Self-evaluation of adjuvant chemotherapy-related adverse effects by patients with breast cancer. JAMA Oncol. 2016;2(4):445–52.

Atkinson TM, Ryan SJ, Bennett AV, Stover AM, Saracino RM, Rogak LJ, Jewell ST, Matsoukas K, Li Y, Basch E. The association between clinician-based common terminology criteria for adverse events (CTCAE) and patient-reported outcomes (PRO): a systematic review. Support Care Cancer. 2016;24(8):3669–76.

Galizia D, Milani A, Geuna E, Martinello R, Cagnazzo C, Foresto M, Longo V, Berchialla P, Solinas G, Calori A, et al. Self-evaluation of duration of adjuvant chemotherapy side effects in breast cancer patients: a prospective study. Cancer Med. 2018;7(9):4339–44.

Gavila J, Seguí M, Calvo L, López T, Alonso JJ, Farto M. Sánchez-de la Rosa R: Evaluation and management of chemotherapy-induced cardiotoxicity in breast cancer: a Delphi study. Clin Transl Oncol. 2017;19(1):91–104.

Leong DP, Lenihan DJ. Clinical practice guidelines in cardio-oncology. Heart Fail Clin. 2022;18(3):489–501.

Munn Z, Pollock D, Khalil H, Alexander L, McLnerney P, Godfrey CM, Peters M, Tricco AC. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022;20(4):950–2.

Li YR, Jia Z, Zhu H. Understanding the value of case reports and studies in the context of clinical research, research design and evidence-based practice. J Case Reports and Studies. 2013;1(2):1–4.

Conn VS, Valentine JC, Cooper HM, Rantz MJ. Grey literature in meta-analyses. Nurs Res. 2003;52(4):256–61.

Download references

Acknowledgements

The authors thank Nawon Kim, a librarian at the Yonsei University Medical Library, for building search terms and guiding the database searches.

This research is supported by the Brain Korea 21 FOUR Project founded by the National Research Foundation (NRF) of Korea, Yonsei University College of Nursing.

Author information

Authors and affiliations.

Graduate School, College of Nursing, Yonsei University, 50 Yonsei-ro, Seoul, South Korea

Hyunjoo Kim

Severance Cardiovascular Hospital, Yonsei University, 50-1 Yonsei-ro, Seoul, South Korea

College of Nursing and Brain Korea 21 FOUR Project, Yonsei University, 50-1 Yonsei-ro, Seoul, South Korea

College of Nursing and Mo-Im Kim Nursing Research Institute, Yonsei University, 50-1 Yonsei-ro, Seoul, South Korea

Sanghee Kim & Jeongok Park

Department of Internal Medicine, Division of Cardiology, Heart Failure Center, Severance Hospital, Yonsei University, 50-1 Yonsei-ro, Seoul, South Korea

Seok-Min Kang

You can also search for this author in PubMed   Google Scholar

Contributions

HK, BH, SK, and JP contributed to the study conception and design. The literature search and record screening were performed by HK and BH under the supervision of JP. Material preparation, data collection, and analysis were performed by HK, BH, and JP. The first draft of the manuscript was written by HK and JP commented on each updated version of the manuscript. The tables and figures were prepared by BH under the instruction of JP. SK helped to interpret the data and provided critical feedback on the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jeongok Park .

Ethics declarations

Ethics approval and consent to participate.

This study was exempted from ethical approval by the Yonsei University Institute Review Board, and consent to participate was not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kim, H., Hong, B., Kim, S. et al. Chemotherapy-related cardiotoxicity and its symptoms in patients with breast cancer: a scoping review. Syst Rev 13 , 167 (2024). https://doi.org/10.1186/s13643-024-02588-z

Download citation

Received : 25 December 2023

Accepted : 14 June 2024

Published : 27 June 2024

DOI : https://doi.org/10.1186/s13643-024-02588-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Breast cancer
• Chemotherapy
• Cardiotoxicity

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

The role of bim in integrating digital twin in building construction: a literature review.

1. Introduction

2. background, 2.1. concept of bim, 2.2. concept of digital twin, 2.3. advancement of bim to digital twin, 3. methodology, 4. literature review, 4.1. discussion on available research on digital twin with bim.

• Integration of BIM and DT: Douglas et al. [ 35 ] focused on using real time data from sensors and other sources to enhance the DT, as well as using data analytics and machine learning algorithms to analyze these data and make predictions about building performance;
• Real time data analysis: Opoku et al. [ 27 ] and Deng et al. [ 11 ] focused on using real time data from sensors and other sources to enhance the DT, as well as using data analytics and machine learning algorithms to analyze these data and make predictions about building performance;
• Simulation and visualization: there has been research on using simulation and visualization technologies to enhance the DT and improve decision-making in the construction and engineering industries [ 21 , 27 ];
• Cost and resource optimization: DT and BIM potentially reduce costs, improve resource allocation, and increase overall efficiency in the building construction process [ 33 , 40 ];
• BIM/DT in the context of sustainability: the integration of BIM and DT support sustainable design and construction practices by incorporating data on energy efficiency [ 21 ], material usage [ 26 ], and environmental impact [ 18 ]; it integrates real-time data from sensors and IoT devices [ 21 ], enabling continuous monitoring [ 5 ], analysis, and proactive maintenance [ 34 ] for sustainable practices.

4.2. Evolution of Digital Twin from BIM

4.3. current study to compare digital twin with bim.

• Concept Origin: technology’s origin is its history, goals, and principles. Understanding the concept helps researchers evaluate their strengths, weaknesses, and applications. The concept’s origin can also indicate which technological parts are more developed or need more research.
• Purpose: to define each technology’s scope and goals. This criterion helps determine their complementary roles and the best integration strategies to improve building design, construction, and operation.
• Application focus: It highlights each technology’s primary focus. It also shows each technology’s pros and cons to guide future improvements. It is crucial to choose the right technology for a project or application.
• Features: They are an essential aspect of the scientific comparison between BIM and DT, as they help understand each technology’s capabilities and limitations and their potential for integration and interoperability.
• Level of Details: We can assess the pros and cons of integrating these technologies into building projects.
• Scalability: allows for evaluating their ability to handle different types of projects and their potential limitations regarding resource requirements and integration with other technologies.
• Main Users: Identify each technology’s primary users and how it meets their needs. This information can help stakeholders choose technology based on project needs and team expertise.
• Interoperability: enables these technologies to be integrated with other systems and software, leading to greater efficiencies and improved outcomes in the building lifecycle management process.
• Application interface: evaluates the usability and effectiveness of the software for different users and applications.
• Building life cycle stage: compares BIM and DT in building construction, as it can help determine which technology is more suitable for a given project.

4.3.1. Concept Origin

4.3.2. purposes, 4.3.3. application focus, 4.3.4. features, 4.3.5. level of details (lod), 4.3.6. scalability, 4.3.7. main users, 4.3.8. interoperability, 4.3.9. application interface, 4.3.10. characteristics, 4.4. advancement of bim to improve digital twin in building construction.

• Increased interoperability: BIM technology has become more interoperable, allowing seamless data exchange between platforms and systems [ 7 ]. It makes creating and updating DT easier with real time data from sensors and other sources.
• Improved data accuracy: BIM technology can offer precise and comprehensive insights into a building’s blueprint, building process, and maintenance, all of which can contribute to developing a more precise DT [ 12 ].
• Increased collaboration: BIM enables collaboration among architects, engineers, and construction professionals, leading to better decision-making and improved overall outcomes [ 25 ]. When this collaboration is applied to creating a DT, it can result in a more comprehensive and effective virtual representation of the building.
• Better visualization: BIM technology has advanced to include more realistic and interactive visualizations [ 40 ], making it easier to understand and analyze the building’s performance through the DT [ 11 ].
• More advanced simulation: BIM has also advanced to include more advanced simulation capabilities, allowing for the simulation of complex systems and analyzing building performance in real time [ 40 ].

5. Result and Discussion

5.1. result and discussion, 5.2. limitation, 5.3. future study, 6. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Zhang, J.; Cheng, J.C.P.; Chen, W.; Chen, K. Digital Twins for Construction Sites: Concepts, LoD Definition, and Applications. J. Manag. Eng. 2022 , 38 , 04021094. [ Google Scholar ] [ CrossRef ]
• Lu, Q.; Xie, X.; Heaton, J.; Parlikad, A.K.; Schooling, J. From BIM towards Digital Twin: Strategy and Future Development for Smart Asset Management. In Service Oriented, Holonic and Multi-Agent Manufacturing Systems for Industry of the Future ; Borangiu, T., Trentesaux, D., Leitão, P., Giret Boggino, A., Botti, V., Eds.; Springer International Publishing: Cham, Switzerland, 2020; pp. 392–404. [ Google Scholar ]
• Grieves, M. Digital Twin: Manufacturing Excellence through Virtual Factory Replication ; Michael W. Grieves, LLC: Cocoa Beach, FL, USA, 2015. [ Google Scholar ]
• O’Gorman, T. How Digital Twins Optimize the Performance of Your Assets in a Sustainable Way. IBM Blog . 2023. Available online: https://www.ibm.com/blog/how-digital-twins-optimize-the-performance-of-your-assets-in-a-sustainable-way (accessed on 22 May 2023).
• Sacks, R.; Brilakis, I.; Pikas, E.; Xie, H.S.; Girolami, M. Construction with Digital Twin Information Systems. Data Cent. Eng. 2020 , 1 , e14. [ Google Scholar ] [ CrossRef ]
• Autodesk Autodesk. Available online: https://www.autodesk.com/solutions/digital-twin/architecture-engineering-construction (accessed on 15 February 2022).
• Boje, C.; Guerriero, A.; Kubicki, S.; Rezgui, Y. Towards a Semantic Construction Digital Twin: Directions for Future Research. Autom. Constr. 2020 , 114 , 103179. [ Google Scholar ] [ CrossRef ]
• Chen, Y.; Kamara, J.M. A Framework for Using Mobile Computing for Information Management on Construction Sites. Autom. Constr. 2011 , 20 , 776–788. [ Google Scholar ] [ CrossRef ]
• Fischer, M.; Ashcraft, H.W.; Reed, D.; Khanzode, A. Atul Khanzode Integrating Project Delivery ; Wiley: Hoboken, NJ, USA, 2017; ISBN 978-0470587355. Available online: https://www.amazon.com/Integrating-Project-Delivery-Martin-Fischer/dp/0470587350 (accessed on 29 October 2022).
• Eastman, C.M.; Eastman, C.; Teicholz, P.; Sacks, R.; Liston, K. BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers and Contractors ; John Wiley & Sons: Hoboken, NJ, USA, 2011; ISBN 978-0-470-54137-1. [ Google Scholar ]
• Deng, M.; Menassa, C.C.; Kamat, V.R. From BIM to Digital Twins: A Systematic Review of the Evolution of Intelligent Building Representations in the AEC-FM Industry. ITcon 2021 , 26 , 58–83. [ Google Scholar ] [ CrossRef ]
• Akanmu, A.A.; Anumba, C.J.; Ogunseiju, O.O. Towards next Generation Cyber-Physical Systems and Digital Twins for Construction. ITcon 2021 , 26 , 505–525. [ Google Scholar ] [ CrossRef ]
• Building Digital Twin Association Publications: Antwerp, Belgium, 2019. White Paper, Q4 2019. p. 9. Available online: https://buildingdigitaltwin.org/wp-content/uploads/2022/02/WhitePaper1-en.pdf (accessed on 22 May 2023).
• Adhikari, S.; Collins, J.; Loreto, G.; Nguyen, T.D. The Use of Parametric Modeling to Enhance the Understanding of Concrete Formwork Structures. In Proceedings of the 2021 ASEE Virtual Annual Conference Content Access, Virtual, 26–29 July 2021. [ Google Scholar ]
• Grieves, M.W. Product Lifecycle Management: The New Paradigm for Enterprises. Int. J. Prod. Dev. 2005 , 2 , 71–84. [ Google Scholar ] [ CrossRef ]
• Grieves, M. Product Lifecycle Management: Driving the next Generation of Lean Thinking ; McGraw-Hill: New York, NY, USA, 2006; ISBN 978-0-07-145230-4. [ Google Scholar ]
• Grieves, M. Virtually Perfect: Driving Innovative and Lean Products through Product Lifecycle Management ; Space Coast Press: Cocoa Beach, FL, USA, 2011; ISBN 978-0-9821380-0-7. [ Google Scholar ]
• Seaton, H.; Savian, C.; Sepasgozar, S.; Sawhney, A. Digital Twins from Design to Handover of Constructed Assets ; Royal Institution of Chartered Surveyors: London, UK, 2022. [ Google Scholar ]
• Hardin: BIM and Construction Management: Proven Tools. Available online: https://scholar.google.com/scholar_lookup?title=BIM%20and%20Construction%20Management%3A%20Proven%20Tools%2C%20Methods&author=B.%20Hardin&publication_year=2009 (accessed on 10 May 2023).
• Zhou, L.; An, C.; Shi, J.; Lv, Z.; Liang, H. Design and Construction Integration Technology Based on Digital Twin. In Proceedings of the 2021 Power System and Green Energy Conference (PSGEC), Shanghai, China, 20–22 August 2021; pp. 7–11. [ Google Scholar ]
• Khajavi, S.H.; Motlagh, N.H.; Jaribion, A.; Werner, L.C.; Holmström, J. Digital Twin: Vision, Benefits, Boundaries, and Creation for Buildings. IEEE Access 2019 , 7 , 147406–147419. [ Google Scholar ] [ CrossRef ]
• Jones, D.; Snider, C.; Nassehi, A.; Yon, J.; Hicks, B. Characterising the Digital Twin: A Systematic Literature Review. CIRP J. Manuf. Sci. Technol. 2020 , 29 , 36–52. [ Google Scholar ] [ CrossRef ]
• Sepasgozar, S.M.E. Differentiating Digital Twin from Digital Shadow: Elucidating a Paradigm Shift to Expedite a Smart, Sustainable Built Environment. Buildings 2021 , 11 , 151. [ Google Scholar ] [ CrossRef ]
• El Jazzar, M.; Piskernik, M.; Nassereddine, H. Digital Twin in Construction: An Empirical Analysis. In Proceedings of the EG-ICE 2020 Workshop on Intelligent Computing in Engineering, Proceedings, Online, 1–4 July 2020. [ Google Scholar ]
• Shahzad, M.; Shafiq, M.T.; Douglas, D.; Kassem, M. Digital Twins in Built Environments: An Investigation of the Characteristics, Applications, and Challenges. Buildings 2022 , 12 , 120. [ Google Scholar ] [ CrossRef ]
• Alonso, R.; Borras, M.; Koppelaar, R.H.E.M.; Lodigiani, A.; Loscos, E.; Yöntem, E. SPHERE: BIM Digital Twin Platform. Proceedings 2019 , 20 , 9. [ Google Scholar ] [ CrossRef ]
• Opoku, D.-G.J.; Perera, S.; Osei-Kyei, R.; Rashidi, M. Digital Twin Application in the Construction Industry: A Literature Review. J. Build. Eng. 2021 , 40 , 102726. [ Google Scholar ] [ CrossRef ]
• Sawhney, A.; Riley, M.; Irizarry, J.; Pérez, C.T. A Proposed Framework for Construction 4.0 Based on a Review of Literature. In EPiC Series in Built Environment ; EasyChair: Liverpool, UK, 2020; Volume 1, pp. 301–309. [ Google Scholar ]
• Madubuike, O.C.; Anumba, C.J.; Khallaf, R. A Review of Digital Twin Applications in Construction. ITcon 2022 , 27 , 145–172. [ Google Scholar ] [ CrossRef ]
• Feng, H.; Chen, Q.; García de Soto, B. Application of Digital Twin Technologies in Construction: An Overview of Opportunities and Challenges. In Proceedings of the 38th International Symposium on Automation and Robotics in Construction (ISARC 2021), Dubai, United Arab Emirates, 4 August 2021. [ Google Scholar ]
• Zhang, H.; Zhou, Y.; Zhu, H.; Sumarac, D.; Cao, M. Digital Twin-Driven Intelligent Construction: Features and Trends. Struct. Durab. Health Monit. 2021 , 15 , 183–206. [ Google Scholar ] [ CrossRef ]
• Grieves, M.W. Virtually Intelligent Product Systems: Digital and Physical Twins. In Complex Systems Engineering: Theory and Practice ; Flumerfelt, S., Schwartz, K.G., Mavris, D., Briceno, S., Eds.; American Institute of Aeronautics and Astronautics, Inc.: Reston, VA, USA, 2019; pp. 175–200. ISBN 978-1-62410-564-7. [ Google Scholar ]
• Zhang, X.; Shen, J.; Saini, P.K.; Lovati, M.; Han, M.; Huang, P.; Huang, Z. Digital Twin for Accelerating Sustainability in Positive Energy District: A Review of Simulation Tools and Applications. Front. Sustain. Cities 2021 , 3 , 663269. [ Google Scholar ] [ CrossRef ]
• Hosamo, H.H.; Imran, A.; Cardenas-Cartagena, J.; Svennevig, P.R.; Svidt, K.; Nielsen, H.K. A Review of the Digital Twin Technology in the AEC-FM Industry. Adv. Civ. Eng. 2022 , 2022 , e2185170. [ Google Scholar ] [ CrossRef ]
• Douglas, D.; Kelly, G.; Kassem, K. BIM, Digital Twin and Cyber-Physical Systems: Crossing and Blurring Boundaries. In Proceedings of the 2021 European Conference on Computing in Construction, Rhodes, Greece, 26 July 2021; pp. 204–211. [ Google Scholar ]
• Gorse, C.; Booth, C.; Scott, L.; Dastbaz, M. Climate Emergency—Managing, Building, and Delivering the Sustainable Development Goals: Selected Proceedings from the International Conference of Sustainable Ecological Engineering Design for Society (SEEDS) 2020 ; Springer International Publishing: Berlin/Heidelberg, Germany, 2021; ISBN 978-3-030-79450-7. [ Google Scholar ]
• Sobhkhiz, S.; El-Diraby, T. Developing BIM-Based Linked Data Digital Twin Architecture to Address a Key Missing Factor: Occupants ; ASCE Library: Reston, VA, USA, 2022; pp. 11–20. [ Google Scholar ] [ CrossRef ]
• Almatared, M.; Liu, H.; Tang, S.; Sulaiman, M.; Lei, Z.; Li, H.X. Digital Twin in the Architecture, Engineering, and Construction Industry: A Bibliometric Review ; ASCE Library: Reston, VA, USA, 2022; p. 678. [ Google Scholar ]
• Quirk, D.; Lanni, J.; Chauhan, N. Digital Twins: Details Of Implementation. ASHRAE J. 2020 , 62 , 20–24. [ Google Scholar ]
• Bolpagni, M.; Gavina, R.; Ribeiro, D. Industry 4.0 for the Built Environment: Methodologies, Technologies and Skills ; Springer Nature: Berlin/Heidelberg, Germany, 2021; ISBN 978-3-030-82430-3. [ Google Scholar ]
• Daniotti, B.; Masera, G.; Bolognesi, C.M.; Lupica Spagnolo, S.; Pavan, A.; Iannaccone, G.; Signorini, M.; Ciuffreda, S.; Mirarchi, C.; Lucky, M.; et al. The Development of a BIM-Based Interoperable Toolkit for Efficient Renovation in Buildings: From BIM to Digital Twin. Buildings 2022 , 12 , 231. [ Google Scholar ] [ CrossRef ]
• Baghalzadeh Shishehgarkhaneh, M.; Keivani, A.; Moehler, R.C.; Jelodari, N.; Roshdi Laleh, S. Internet of Things (IoT), Building Information Modeling (BIM), and Digital Twin (DT) in Construction Industry: A Review, Bibliometric, and Network Analysis. Buildings 2022 , 12 , 1503. [ Google Scholar ] [ CrossRef ]
• Kaur, M.J.; Mishra, V.P.; Maheshwari, P. The Convergence of Digital Twin, IoT, and Machine Learning: Transforming Data into Action ; Springer: Cham, Switzerland, 2020; Available online: https://link.springer.com/chapter/10.1007/978-3-030-18732-3_1 (accessed on 14 May 2023).
• Schleich, B.; Anwer, N.; Mathieu, L.; Wartzack, S. Shaping the Digital Twin for Design and Production Engineering. CIRP Ann. 2017 , 66 , 141–144. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Qi, Q.; Tao, F. Digital Twin and Big Data Towards Smart Manufacturing and Industry 4.0: 360 Degree Comparison. IEEE Access 2018 , 6 , 3585–3593. [ Google Scholar ] [ CrossRef ]
• Carlsén, A.; Elfstrand, O. Augmented Construction: Developing a Framework for Implementing Building Information Modeling through Augmented Reality at Construction Sites ; Semantic Scholar: Stockholm, Sweden, 2018. [ Google Scholar ]
• Gubbi, J.; Buyya, R.; Marusic, S.; Palaniswami, M. Internet of Things (IoT): A Vision, Architectural Elements, and Future Directions. Future Gener. Comput. Syst. 2013 , 29 , 1645–1660. [ Google Scholar ] [ CrossRef ] [ Green Version ]
• Tang, S.; Shelden, D.R.; Eastman, C.M.; Pishdad-Bozorgi, P.; Gao, X. A Review of Building Information Modeling (BIM) and the Internet of Things (IoT) Devices Integration: Present Status and Future Trends. Autom. Constr. 2019 , 101 , 127–139. [ Google Scholar ] [ CrossRef ]
• Lee, D.; Cha, G.; Park, S. A Study on Data Visualization of Embedded Sensors for Building Energy Monitoring Using BIM. Int. J. Precis. Eng. Manuf. 2016 , 17 , 807–814. [ Google Scholar ] [ CrossRef ]
• IoT for All. ThoughtWire Blog. Digital Twins vs. Building Information Modeling (BIM). 2020. Available online: https://www.iotforall.com/digital-twin-vs-bim?ss360SearchTerm=Digital%20Twin (accessed on 16 August 2022).
• Rolle, R.; Martucci, V.; Godoy, E. Architecture for Digital Twin Implementation Focusing on Industry 4.0. IEEE Lat. Am. Trans. 2020 , 18 , 889–898. [ Google Scholar ] [ CrossRef ]
• Kritzinger, W.; Karner, M.; Traar, G.; Henjes, J.; Sihn, W. Digital Twin in Manufacturing: A Categorical Literature Review and Classification. IFAC Pap. 2018 , 51 , 1016–1022. [ Google Scholar ] [ CrossRef ]
• Lavrentyeva, A.V.; Dzikia, A.A.; Kalinina, A.E.; Frolov, D.P.; Akhverdiev, E.A.; Barakova, A.S. Artificial Intelligence and Digital Transformations in the Society. IOP Conf. Ser. Mater. Sci. Eng. 2019 , 483 , 012019. [ Google Scholar ] [ CrossRef ] [ Green Version ]

Click here to enlarge figure

Share and Cite

Nguyen, T.D.; Adhikari, S. The Role of BIM in Integrating Digital Twin in Building Construction: A Literature Review. Sustainability 2023 , 15 , 10462. https://doi.org/10.3390/su151310462

Nguyen TD, Adhikari S. The Role of BIM in Integrating Digital Twin in Building Construction: A Literature Review. Sustainability . 2023; 15(13):10462. https://doi.org/10.3390/su151310462

Nguyen, Tran Duong, and Sanjeev Adhikari. 2023. "The Role of BIM in Integrating Digital Twin in Building Construction: A Literature Review" Sustainability 15, no. 13: 10462. https://doi.org/10.3390/su151310462

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals