literature review example in computer science

Computer Science

• Key resources
• Databases and journal articles
• Standards and patents

What is a literature review?

Finding the gaps and advances in your area, literature review resources, annotated bibliography.

A literature review is a survey and critical analysis of what has been written on a particular topic, theory, question or method.

What is its purpose?

• justify your research
• provide context for your research
• ensure that the research has not been done before
• highlight flaws in previous research
• identify new ways, to interpret and highlight gaps in previous research
• signpost a way forward for further research
• show where your research fits into the existing literature
• Literature reviews guide The Literature reviews guide contains detailed information on the process of searching for and producing literature reviews.
• Scopus A multidisciplinary abstract and citation database of peer reviewed literature, book reviews and conference proceedings.
• Web of Science A collection of citation databases and citation analysis tools covering the sciences, social sciences, arts and humanities.
• IEEE Xplore digital library Provides full-text access to IEEE and IEE transactions, journals, magazines and conference proceedings published since 1988 and current IEEE Standards.
• ACM digital library publications Full text of every article published by the Association for Computing Machinery (ACM) and bibliographic citations from major publishers in computing
• Annual reviews online This database provides review journals from across the sciences, with articles that review significant primary research literature.
• Cited reference searching

Books and other resources for approaches and methods on doing a literature review. See the Literature reviews guide .

An annotated bibliography provides:

• a list of references presenting a brief summary of the main arguments or ideas of each resource.
• a critique or evaluation of the resource's usefulness, reliability, objectivity or bias
• a reflection on how the resource fits into your research.
• << Previous: Multimedia
• Last Updated: Nov 29, 2023 12:58 PM
• URL: https://guides.library.uq.edu.au/computer-science

Computer Science and Engineering

• Getting Started
• Keeping Current

Introduction

Gather Your Tools

Determine the Project's Scope

Create the Search Strategy

Determine What Resources to Use

Search, read, refine, repeat.

Saved Searches, Alerts and Feeds

• STEM Biographies & Info
• Writing & Citing

This page focuses on how to do an in-depth literature review for a dissertation, thesis, grant application or lengthy term paper in electrical engineering.  

• For a more general description of what an in-depth literature review is and how it looks, see our guide on " Literature Reviews and Annotated Bibliographies " created by Ed Oetting, history and political science librarian.
• For lower-level engineering undergraduate students who are doing a short term paper, the " How to Research a Topic " page on the " Engineerng Basics" guide may be more applicable.

Library Account Is your library account clear of fines?   If not, you may not be allowed to check out more books nor renew books you already have.  All library notices are sent via email to your "asu.edu" address; if you prefer to receive email at a different address make sure you have forwarded your asu.edu correctly.  Also, make sure that your spam filter allows the library email to come through. 

Illiad (Interlibrary Loan) Account   If you don't already have an ILLiad account, please register for one.  Interlibrary loan services will get you material not available at the ASU Library and also scan or deliver materials from the libraries on the other ASU campuses.

Determine the Project's Scope.

Do you know what you are looking for?  Can you describe your project using one simple sentence or can you phrase the project as a question?  Without a clear idea of the project, you may not be able to determine which are the best resources to search, what terminology should be used in those resources, and if the results are appropriate and sufficient.    

If you're having difficulty getting your project described succinctly, try using a PICO chart to identify the concepts involved:

• P is the popluation, problem, predicament or process
• I is the intervention or improvement
• C is what you'll compare your intervention/improvment to, and
• O is the outcome (or results of the comparison of I and C ) 

For example: 

Your client, the owner of a nuclear power generating facility, has had several less than optimal safety inspections recently.  The inspectors have singled out operator error as a major concern and have required changes in employee training.  But is more training the solution?  The employees complain that the plant's poorly designed control room hampers their ability to respond to non-standard situations.  Could a redesign improve performance and decrease the occurance of unsafe events?   Your client wants more than just your opinion, he wants to see the data to back it up.   So, what can you find in the literature?

Here's one way that the PICO chart could be filled out:     

• P =   nuclear power safety  
• I  = human factors engineering
• C  =  additional training; little or no human factors engineering used  
• O = accident rate or safety inspection comparison

And here are examples of possible search statements:  

• I am looking for ways that human factors engineering can improve safety in the nuclear power industry.
• Is additional training or employing human factors engineering the better method for reducing safety violations in a nuclear power plant? 

Your research will always start with a " P AND I " search; those are the most important pieces of the puzzle.  However, once you have the results from that search, you'll need to know where you want to go with those results; that's when the C and O concepts need to be considered.  

 Also, don't forget --- determine if your project has limits.  For example:

• Are you reviewing the literature only within a specific time frame?
• Are you looking at English-language material only?
• Are you considering research from just the United States or worldwide?
• Are there types of material you won't be covering (trade magazines, patents, technical reports, etc.)?  

Take the simple sentence or question that describes what you are looking for.  What are the concepts in the sentence? Are there synonyms that describe the same concept?   If you filled out a PICO chart, concentrate on the  P (problem) and the  I (intervention) for the concept chart.  

Concept Chart:

Concept 1:   _______  OR _______  OR _______  AND Concept 2:   _______   OR   _______  OR   _______  AND Concept 3:  _______  OR  _______  OR   _______ 

  Example:  

I am looking for ways that human factors engineering can improve safety in the nuclear power industry. 

Concept 1:   nuclear power    OR _ nuclear industry _____   AND Concept 2:   _safety___  OR   _accident prevention____   AND Concept 3: _ human factors engineering ___   

What resources you'll use for your literature review depends on what types of materials you want to find.  

• Background Information The more you know about a topic, the better you'll be able to research it.  You'll be familiar with the terminology, understand the underlining science/technology and be aware of the issues in the field. Most importantly, you'll be able to understand what you've retrieved from your search.  But no matter how much you know before hand you'll likely run across terms and concepts with which you're unfamiliar.    Materials such as encyclopedias, dictionaries and handbooks will not only help you learn about the basics of your topic before you begin your search but they'll also help you understand the terminology used in the documents you found from your literature review.    You'll find these types of resources listed on the Dictionaries and Handbooks pages on this guide.
• Books The large size of books (usually 100-500 pages) allows a topic to be studied braodly, covering many different issues.  Conversely, the large size also allows for a specific aspect of the topic to be covered in great detail.  Because of the time it takes to publish, sci-tech books generally do not contain the most current information. To find print and online books from both the ASU Library as well as in other libraries, see the Books page on this guide.
• Conference Papers Scientists and engineers frequently present new findings at conferences before these findings are written up in journal articles or books.  Not every conference, however, publishes it proceedings.  In some cases, conferences publish only a few of the papers presented but not all.   Many resources that help you find journal articles, may also be used to find conference papers, see the Articles page on this guide.
• Journal and Trade Magazine Articles Articles in journals (also called magazines) are short, usually 5-20 pages in length and cover a specific finding, experiment or project.  Articles in scholary journals are usually written by academics or professional scientists/engineers and are aimed at others at the same level.   Articles in trade journals/magazines are written by the journal staff and report on industry news suchs as sales, mergers, prices, etc.   To find journal and trade magazine articles, use the resources listed on the Articles page on this guide. 
• Patents Patents are grants from governments that gives the inventor certain rights of manufacture.  Patents provide a wealth of information for how a technology is being advanced and by which companies.  It is frequently stated that 80% of the information in patents never appears elsewhere in the literature. 
• To identify patents granted in the U.S. and internationally see the " Searching for Patents " guide.
• To see statistical information for U.S. patents by technology class see the US Patent and Trademark's website.
• Technical Reports Technical reports are part of the "gray literature";  gray literature refers to documents that are not published commercially, hence they are difficult to both identify and find.  Technical reports focus on a specific experiment or research project and are meant to convey the results of the experiment or project back to the funding organization.  In the United States, common sources of technical reports are the government agencies that sponsor research projects.  Reports generated within a private corporation and funded soley by that corporation are seldom ever available to anyone outside of the company.      To find technical reports, use the resources listed on the Technical Reports page on this guide.     

Search, Read, Refine and Repeat

Now it's time to apply your search strategy in the resources you've decided to use.

• Use the Advanced Search feature (or whatever search is set up with the 3 lines of boxes) and enter your search strategy just as you recorded in your search strategy chart.  Don't forget to set your limits.   If the resource only provides a single search box, rearrange your chart from vertical into horizontal so that the search statement looks like this:   (Concept#1 OR synonym) AND (Concept#2 OR synonym) AND (Concept#3 OR synonym) Example: (nuclear power OR nuclear industry) AND (safety OR accident prevention) AND (human factors engineering)
• Examine the results to find the most appropriate items.  Keep your one-sentence project description (and/or your PICO chart) in mind to help you stay on track.
• Export the records/citations you want to keep into a citation manager.
• If there are subjects (may also be called subject headings, index terms, descriptors or controlled vocabulary) assigned to each item, make sure that those also transferred into a citation manager.  If not, add them manually.
• Get the full text of the items 
• Read the full text of the items and look at the subjects assigned to the item and consider:
• Do I have to change (narrow) my topic to something more specific because I'm finding way too much? 
• Do I have to change (broaden) my topic because I can't find enough about it? 
• Is there additional terminology for my topic/concepts that I hadn't included in my original search?
• Redo your search strategy according to what you found in step #6 and rerun the search in the resources again.
• You may need to repeat this cycle several times before you are able to identify the best terminology to use in each resource. 

If there will be several months in between when you search the literature and when you turn in the paper, consider setting up alerts and feeds so that you are notified should new items about your topic appear.   How you set up an alert or feed will vary.  In most cases you'll be required to set up a personal account or profile with the journal or database --- there is no charge for this but you will have to identify yourself and provide an email address.  

For instructions on setting up alerts and feeds, see the " Keep Current " page.

• << Previous: Keeping Current
• Next: Resources >>
• Last updated: Jan 2, 2024 8:27 AM
• URL: https://libguides.asu.edu/computerscience

The ASU Library acknowledges the twenty-three Native Nations that have inhabited this land for centuries. Arizona State University's four campuses are located in the Salt River Valley on ancestral territories of Indigenous peoples, including the Akimel O’odham (Pima) and Pee Posh (Maricopa) Indian Communities, whose care and keeping of these lands allows us to be here today. ASU Library acknowledges the sovereignty of these nations and seeks to foster an environment of success and possibility for Native American students and patrons. We are advocates for the incorporation of Indigenous knowledge systems and research methodologies within contemporary library practice. ASU Library welcomes members of the Akimel O’odham and Pee Posh, and all Native nations to the Library.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

Brazil’s plummeting graduate enrolments hint at declining interest in academic science careers

Career News 21 MAY 24

How to set up your new lab space

Career Column 20 MAY 24

How religious scientists balance work and faith

Career Feature 20 MAY 24

Pay researchers to spot errors in published papers

World View 21 MAY 24

US halts funding to controversial virus-hunting group: what researchers think

News 16 MAY 24

Harassment of scientists is surging — institutions aren’t sure how to help

News Feature 21 MAY 24

I’m worried I’ve been contacted by a predatory publisher — how do I find out?

Career Feature 15 MAY 24

Vice President, Nature Communications Portfolio

This is an exciting opportunity to play a key leadership role in the market-leading journal Nature Portfolio and help drive its overall contribution.

New York City, New York (US), Berlin, or Heidelberg

Springer Nature Ltd

Senior Postdoctoral Research Fellow

Senior Postdoctoral Research Fellow required to lead exciting projects in Cancer Cell Cycle Biology and Cancer Epigenetics.

Melbourne University, Melbourne (AU)

University of Melbourne & Peter MacCallum Cancer Centre

Overseas Talent, Embarking on a New Journey Together at Tianjin University

We cordially invite outstanding young individuals from overseas to apply for the Excellent Young Scientists Fund Program (Overseas).

Tianjin, China

Tianjin University (TJU)

Chair Professor Positions in the School of Pharmaceutical Science and Technology

SPST seeks top Faculty scholars in Pharmaceutical Sciences.

Chair Professor Positions in the School of Precision Instruments and Optoelectronic Engineering

We are committed to accomplishing the mission of achieving a world-top-class engineering school.

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

The Sheridan Libraries

• Computer Science and Information Security
• Sheridan Libraries

Literature Reviews

• How to Access Full Text
• Background: Books and Review Articles
• Articles, Conferences, More
• Google Scholar and Google Books
• Information Security
• LaTeX, MATLAB, Open Source, Overleaf, More
• Problems and Solutions
• Technical Reports
• Avoiding Plagiarism
• Citing Sources This link opens in a new window
• Copyright This link opens in a new window
• Evaluating Information This link opens in a new window
• RefWorks Guide and Help This link opens in a new window
• CAPSTONE HELP
• Organizing/Synthesizing
• Peer Review
• Ulrich's -- One More Way To Find Peer-reviewed Papers

"Literature review," "systematic literature review," "integrative literature review" -- these are terms used in different disciplines for basically the same thing -- a rigorous examination of the scholarly literature about a topic (at different levels of rigor, and with some different emphases).  

1. Our library's guide to Writing a Literature Review

2. Other helpful sites

• Writing Center at UNC (Chapel Hill) -- A very good guide about lit reviews and how to write them
• Literature Review: Synthesizing Multiple Sources (LSU, June 2011 but good; PDF) -- Planning, writing, and tips for revising your paper

3. Welch Library's list of the types of expert reviews

Doing a good job of organizing your information makes writing about it a lot easier.

You can organize your sources using a citation manager, such as refworks , or use a matrix (if you only have a few references):.

• Use Google Sheets, Word, Excel, or whatever you prefer to create a table
• The column headings should include the citation information, and the main points that you want to track, as shown

Synthesizing your information is not just summarizing it. Here are processes and examples about how to combine your sources into a good piece of writing:

• Purdue OWL's Synthesizing Sources
• Synthesizing Sources (California State University, Northridge)

Annotated Bibliography  

An "annotation" is a note or comment. An "annotated bibliography" is a "list of citations to books, articles, and [other items]. Each citation is followed by a brief...descriptive and evaluative paragraph, [whose purpose is] to inform the reader of the relevance, accuracy, and quality of the sources cited."*

• Sage Research Methods (database) --> Empirical Research and Writing (ebook) -- Chapter 3: Doing Pre-research  
• Purdue's OWL (Online Writing Lab) includes definitions and samples of annotations  
• Cornell's guide * to writing annotated bibliographies  

* Thank you to Olin Library Reference, Research & Learning Services, Cornell University Library, Ithaca, NY, USA https://guides.library.cornell.edu/annotatedbibliography

What does "peer-reviewed" mean?

• If an article has been peer-reviewed before being published, it means that the article has been read by other people in the same field of study ("peers").
• The author's reviewers have commented on the article, not only noting typos and possible errors, but also giving a judgment about whether or not the article should be published by the journal to which it was submitted.

How do I find "peer-reviewed" materials?

• Most of the the research articles in scholarly journals are peer-reviewed.
• Many databases allow you to check a box that says "peer-reviewed," or to see which results in your list of results are from peer-reviewed sources. Some of the databases that provide this are Academic Search Ultimate, CINAHL, PsycINFO, and Sociological Abstracts.

What kinds of materials are *not* peer-reviewed?

• open web pages
• most newspapers, newsletters, and news items in journals
• letters to the editor
• press releases
• columns and blogs
• book reviews
• anything in a popular magazine (e.g., Time, Newsweek, Glamour, Men's Health)

If a piece of information wasn't peer-reviewed, does that mean that I can't trust it at all?

No; sometimes you can. For example, the preprints submitted to well-known sites such as  arXiv  (mainly covering physics) and  CiteSeerX (mainly covering computer science) are probably trustworthy, as are the databases and web pages produced by entities such as the National Library of Medicine, the Smithsonian Institution, and the American Cancer Society.

Is this paper peer-reviewed? Ulrichsweb will tell you.

1) On the library home page , choose "Articles and Databases" --> "Databases" --> Ulrichsweb

2) Put in the title of the JOURNAL (not the article), in quotation marks so all the words are next to each other

3) Mouse over the black icon, and you'll see that it means "refereed" (which means peer-reviewed, because it's been looked at by referees or reviewers). This journal is not peer-reviewed, because none of the formats have a black icon next to it:

• << Previous: Evaluating Information
• Next: RefWorks Guide and Help >>
• Last Updated: May 13, 2024 10:04 AM
• URL: https://guides.library.jhu.edu/computer

Computer Science (UNH Durham): Literature Reviews

• Books & E-Books
• Find Articles
• Reference Shelf
• Literature Reviews

Technical Reports

• Tools & Tutorials

Bibliographic Management Software

Zotero is a free, open source bibliographic management tool that operates as an extension of the Firefox browser.   Zotero allows the collection of citations to any kind of material and automatically formats bibliographies in almost any style.  Zotero also has many search, tagging, and note taking features.  Visit the Zotero quick start guide to learn more. 

• Electronic Colloquium on Computational Complexity Full text collection of reports, papers, and notes searchable by keyword or author. Includes a browsable keyword list.
• National Technical Information Service (NTIS) Collection of citations to reports on government funded research. If you need materials cited on the NTIS site please check WorldCat or talk to your librarian.
• Science.gov Allows searches across many government databases and websites. Content is not limited to technical reports. Often full text of reports is available.
• Virtual Technical Reports Center Links to hundreds of university, government, and project websites containing technical reports and other resources. more... less... Maintained by the University of Maryland Libraries.

Getting Started

The goal of a literature review is to find all the relevant publications on a topic and to then summarize and synthesize that information. A literature review can help you find areas where further research is needed, narrow a research topic, or determine if a thesis question is unique. Talk to your advisor for help defining your research question. For help with library resources, talk to your librarian; they can help you find a combination of resources that will result in a comprehensive search.

Indexes and Databases

• ACM Digital Library This link opens in a new window Provides access to ACM journals, newsletters, and conference proceedings. Includes bibliographic information, abstracts, reviews, and the full text for articles along with selected works published by affiliated organizations. Dates of coverage: 1947-current
• CiteSeerX Indexes computer and information science literature and includes full text of articles and citation statistics. Allows the full text of articles to be searched.
• IEEE Xplore This link opens in a new window A digital library providing full text access to the world's highest quality technical literature in electrical engineering, computer science, and electronics. It contains full text documents from IEEE journals, transactions, magazines, letters, conference proceedings, standards, and IET (Institution of Engineering and Technology) Conferences. UNH no longer has access to IET Journals.
• Web of Science This link opens in a new window Facilitates research-level interdisciplinary search. Use the Basic Search to locate articles in high-impact scholarly journals in science, social science, arts, and humanities. Use the unique Cited Reference search for articles that cite an article you already know of, so you can track citations forward in time.
• INSPEC This link opens in a new window The world's leading resource for coverage of research literature in physics, including astronomy and astrophysics. Its other subject strengths are electrical engineering; control, and computing, including information technology. International journals and conference proceedings are covered, with selected indexing of books and reports. This resource allows searching by specialized fields. Dates of coverage: 1896-current
• Computing Research Repository (CoRR) Archives computer science papers with some coverage back to 1993. Includes full text of all papers. Frequently updated. more... less... Sponsored by ACM, the arXiv.org e-Print archive, NCSTRL (Networked Computer Science Technical Reference Library), and AAAI.
• DBLP Computer Science Bibliography Provides citations to articles in major computer science journals and conference proceedings and in some cases links to full-text versions of articles (look for the words "Electronic Editions.") Search by author or keyword, or browse by journal, conference proceeding, or broad subject area.

Finding Dissertations and Theses

• Dissertations & Theses Global This link opens in a new window Comprehensive collection of full-text dissertations and theses. Official digital dissertations archive for the Library of Congress and considered the database of record for graduate research. Note: Full text for certain publications is subject to market availability. Dates of coverage: 1861-current
• Networked Digital Library of Theses and Dissertations Searches almost 4 million dissertations and theses from around the world. To focus your search, try the "advanced search tips" and use the facets on the left when viewing your search results.
• OpenDOAR Allows searches of the contents of institutional repositories which may contain dissertations that are not available through the Digital Dissertations database. Repositories contents are not limited to dissertations; to limit your search add "dissertation" or "thesis" to your keywords in the search box.
• TEL (thèses-EN-ligne) Part of HAL , this database is a multidisciplinary collection of self-submitted theses and dissertations.

Engineering & Physical Sciences Librarian

Print Indexes

• ACM Guide to Computing Literature by Association for Computing Machinery Call Number: Eng/Math/CS Library QA76 .A8 Index to books, papers, reports, articles in major journals of computing and related fields, conferences and symposia. Available online from 1985 to present at the ACM website. Available only in print from 1977 to 1984.
• << Previous: Reference Shelf
• Next: Technical Reports >>
• Last Updated: May 2, 2024 2:55 PM
• URL: https://libraryguides.unh.edu/computerscience

University Libraries      University of Nevada, Reno

• Skill Guides
• Subject Guides

Computer Science: Systematic Reviews

• Articles, eBooks, & Databases
• Technical Reports & Standards
• Software & Programming Languages
• Systematic Reviews

What Is a Systematic Review?

Regular literature reviews are simply summaries of the literature on a particular topic. A systematic review, however, is a comprehensive literature review conducted to answer a specific research question. Authors of a systematic review aim to find, code, appraise, and synthesize all of the previous research on their question in an unbiased and well-documented manner. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) outline the minimum amount of information that needs to be reported at the conclusion of a systematic review project. 

Other types of what are known as "evidence syntheses," such as scoping, rapid, and integrative reviews, have varying methodologies. While systematic reviews originated with and continue to be a popular publication type in medicine and other health sciences fields, more and more researchers in other disciplines are choosing to conduct evidence syntheses. 

This guide will walk you through the major steps of a systematic review and point you to key resources including Covidence, a systematic review project management tool. For help with systematic reviews and other major literature review projects, please send us an email at  [email protected] .

Getting Help with Reviews

Organization such as the Institute of Medicine recommend that you consult a librarian when conducting a systematic review. Librarians at the University of Nevada, Reno can help you:

• Understand best practices for conducting systematic reviews and other evidence syntheses in your discipline
• Choose and formulate a research question
• Decide which review type (e.g., systematic, scoping, rapid, etc.) is the best fit for your project
• Determine what to include and where to register a systematic review protocol
• Select search terms and develop a search strategy
• Identify databases and platforms to search
• Find the full text of articles and other sources
• Become familiar with free citation management (e.g., EndNote, Zotero)
• Get access to you and help using Covidence, a systematic review project management tool

Doing a Systematic Review

• Plan - This is the project planning stage. You and your team will need to develop a good research question, determine the type of review you will conduct (systematic, scoping, rapid, etc.), and establish the inclusion and exclusion criteria (e.g., you're only going to look at studies that use a certain methodology). All of this information needs to be included in your protocol. You'll also need to ensure that the project is viable - has someone already done a systematic review on this topic? Do some searches and check the various protocol registries to find out. 
• Identify - Next, a comprehensive search of the literature is undertaken to ensure all studies that meet the predetermined criteria are identified. Each research question is different, so the number and types of databases you'll search - as well as other online publication venues - will vary. Some standards and guidelines specify that certain databases (e.g., MEDLINE, EMBASE) should be searched regardless. Your subject librarian can help you select appropriate databases to search and develop search strings for each of those databases.  
• Evaluate - In this step, retrieved articles are screened and sorted using the predetermined inclusion and exclusion criteria. The risk of bias for each included study is also assessed around this time. It's best if you import search results into a citation management tool (see below) to clean up the citations and remove any duplicates. You can then use a tool like Rayyan (see below) to screen the results. You should begin by screening titles and abstracts only, and then you'll examine the full text of any remaining articles. Each study should be reviewed by a minimum of two people on the project team. 
• Collect - Each included study is coded and the quantitative or qualitative data contained in these studies is then synthesized. You'll have to either find or develop a coding strategy or form that meets your needs. 
• Explain - The synthesized results are articulated and contextualized. What do the results mean? How have they answered your research question?
• Summarize - The final report provides a complete description of the methods and results in a clear, transparent fashion. 

Adapted from

Types of reviews, systematic review.

These types of studies employ a systematic method to analyze and synthesize the results of numerous studies. "Systematic" in this case means following a strict set of steps - as outlined by entities like PRISMA and the Institute of Medicine - so as to make the review more reproducible and less biased. Consistent, thorough documentation is also key. Reviews of this type are not meant to be conducted by an individual but rather a (small) team of researchers. Systematic reviews are widely used in the health sciences, often to find a generalized conclusion from multiple evidence-based studies. 

Meta-Analysis

A systematic method that uses statistics to analyze the data from numerous studies. The researchers combine the data from studies with similar data types and analyze them as a single, expanded dataset. Meta-analyses are a type of systematic review.

Scoping Review

A scoping review employs the systematic review methodology to explore a broader topic or question rather than a specific and answerable one, as is generally the case with a systematic review. Authors of these types of reviews seek to collect and categorize the existing literature so as to identify any gaps.

Rapid Review

Rapid reviews are systematic reviews conducted under a time constraint. Researchers make use of workarounds to complete the review quickly (e.g., only looking at English-language publications), which can lead to a less thorough and more biased review. 

Narrative Review

A traditional literature review that summarizes and synthesizes the findings of numerous original research articles. The purpose and scope of narrative literature reviews vary widely and do not follow a set protocol. Most literature reviews are narrative reviews. 

Umbrella Review

Umbrella reviews are, essentially, systematic reviews of systematic reviews. These compile evidence from multiple review studies into one usable document. 

Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal , vol. 26, no. 2, 2009, pp. 91-108. doi: 10.1111/j.1471-1842.2009.00848.x .

• << Previous: Software & Programming Languages

Engineering -- Computer Science

• Computing Reviews
• Find Articles
• Web Resources
• Citation Help
• Research help step-by-step
• Find Theses

Types of Literature Reviews

How to write a literature review, organizing the literature review.

• Associations
• Related Guides
• Career Resources

Literature Reviews!

• Traditional/narrative .  This type of review critiques and summaries a body of literature and draws conclusions about the topic in question. Cronin, P. BJN, 2008, vol 17, No 1 p. 38-39. 
• Systematic literature review .  These reviews use a rigorous and well-defined approach to reviewing the literature in a specific subject area. Cronin, P. BJN, 2008, Vol 17, No 1 p.42
• Scoping reviews aim to map the key concepts underpinning a research area and the main sources and types of evidence available. They provide wide coverage of the literature but can vary enormously in the degree to which they extract, analyse and represent the available evidence. Arksey, H.& O'Malley, 2005, "Scoping studies: towards a methodological framework," http://eprints.whiterose.ac.uk/1618/1/Scopingstudies.pdf.
• Meta-analysis .  A form of systematic review which is largely a statistical technique.  It takes findings from several studies on the same subject and analyses them using stardardized statistical procedures. Cronin, P. BJN, 2008, Vol 17, No 1 p.42
• Meta-synthesis .  A non-statistical technique used to integrate, evaluate and interpret the findings of multiple qualitative research studies. Cronin, P. BJN, 2008, Vol 17, No 1 p.42
• Rapid evidence reviews are used to summarise the available research within the constraints of a given timetable, typically three months or less.  Consequently there are limitations on the extent of the searches and other review activities and should be viewed as provisional appraisals rather than full systematic reviews. Centre for Reviews and Dissemination, 2009,"Systematic Reviews" http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf
• Cochrane Handbook for Systematic Reviews of Interventions Chapter 5: Defining the review question and developing criteria for including studies.
• How to Write a Literature Review University of North Carolina
• Learn How to Write a Literature Review University of Wisconson
• The Literature Review: A Few Tips On Conducting It University of Toronto

Various approaches can be used to organize the review.

• Themes or Categories - this is popular and allows integration of theoretical and empirical(research) literature.
• Chronological - can demostrate the emergence of a topic over time.
• Theoretical/Methodological - this method allows for discussion of the theoretical literatures followed by exploration of methodological literature to indicate why a particle research design might be appropriate for the topic.  Useful when there is little empirical(research) literature. 
• Theoretical and Methodological in two sections - allows both types to be discussed separately.  May be more of a descriptive rather than a critical review.

Adapted from Cronin, P. BJN, 2008, Vol 17, No 1 p.42

• << Previous: Find Theses
• Next: Associations >>
• Last Updated: Sep 11, 2023 12:10 PM
• URL: https://libguides.uvic.ca/compsci

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability