peer reviewed articles critical thinking

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Critical Thinking: A Model of Intelligence for Solving Real-World Problems

Diane f. halpern.

1 Department of Psychology, Claremont McKenna College, Emerita, Altadena, CA 91001, USA

Dana S. Dunn

2 Department of Psychology, Moravian College, Bethlehem, PA 18018, USA; ude.naivarom@nnud

Most theories of intelligence do not directly address the question of whether people with high intelligence can successfully solve real world problems. A high IQ is correlated with many important outcomes (e.g., academic prominence, reduced crime), but it does not protect against cognitive biases, partisan thinking, reactance, or confirmation bias, among others. There are several newer theories that directly address the question about solving real-world problems. Prominent among them is Sternberg’s adaptive intelligence with “adaptation to the environment” as the central premise, a construct that does not exist on standardized IQ tests. Similarly, some scholars argue that standardized tests of intelligence are not measures of rational thought—the sort of skill/ability that would be needed to address complex real-world problems. Other investigators advocate for critical thinking as a model of intelligence specifically designed for addressing real-world problems. Yes, intelligence (i.e., critical thinking) can be enhanced and used for solving a real-world problem such as COVID-19, which we use as an example of contemporary problems that need a new approach.

1. Introduction

The editors of this Special Issue asked authors to respond to a deceptively simple statement: “How Intelligence Can Be a Solution to Consequential World Problems.” This statement holds many complexities, including how intelligence is defined and which theories are designed to address real-world problems.

2. The Problem with Using Standardized IQ Measures for Real-World Problems

For the most part, we identify high intelligence as having a high score on a standardized test of intelligence. Like any test score, IQ can only reflect what is on the given test. Most contemporary standardized measures of intelligence include vocabulary, working memory, spatial skills, analogies, processing speed, and puzzle-like elements (e.g., Wechsler Adult Intelligence Scale Fourth Edition; see ( Drozdick et al. 2012 )). Measures of IQ correlate with many important outcomes, including academic performance ( Kretzschmar et al. 2016 ), job-related skills ( Hunter and Schmidt 1996 ), reduced likelihood of criminal behavior ( Burhan et al. 2014 ), and for those with exceptionally high IQs, obtaining a doctorate and publishing scholarly articles ( McCabe et al. 2020 ). Gottfredson ( 1997, p. 81 ) summarized these effects when she said the “predictive validity of g is ubiquitous.” More recent research using longitudinal data, found that general mental abilities and specific abilities are good predictors of several work variables including job prestige, and income ( Lang and Kell 2020 ). Although assessments of IQ are useful in many contexts, having a high IQ does not protect against falling for common cognitive fallacies (e.g., blind spot bias, reactance, anecdotal reasoning), relying on biased and blatantly one-sided information sources, failing to consider information that does not conform to one’s preferred view of reality (confirmation bias), resisting pressure to think and act in a certain way, among others. This point was clearly articulated by Stanovich ( 2009, p. 3 ) when he stated that,” IQ tests measure only a small set of the thinking abilities that people need.”

3. Which Theories of Intelligence Are Relevant to the Question?

Most theories of intelligence do not directly address the question of whether people with high intelligence can successfully solve real world problems. For example, Grossmann et al. ( 2013 ) cite many studies in which IQ scores have not predicted well-being, including life satisfaction and longevity. Using a stratified random sample of Americans, these investigators found that wise reasoning is associated with life satisfaction, and that “there was no association between intelligence and well-being” (p. 944). (critical thinking [CT] is often referred to as “wise reasoning” or “rational thinking,”). Similar results were reported by Wirthwein and Rost ( 2011 ) who compared life satisfaction in several domains for gifted adults and adults of average intelligence. There were no differences in any of the measures of subjective well-being, except for leisure, which was significantly lower for the gifted adults. Additional research in a series of experiments by Stanovich and West ( 2008 ) found that participants with high cognitive ability were as likely as others to endorse positions that are consistent with their biases, and they were equally likely to prefer one-sided arguments over those that provided a balanced argument. There are several newer theories that directly address the question about solving real-world problems. Prominent among them is Sternberg’s adaptive intelligence with “adaptation to the environment” as the central premise, a construct that does not exist on standardized IQ tests (e.g., Sternberg 2019 ). Similarly, Stanovich and West ( 2014 ) argue that standardized tests of intelligence are not measures of rational thought—the sort of skill/ability that would be needed to address complex real-world problems. Halpern and Butler ( 2020 ) advocate for CT as a useful model of intelligence for addressing real-world problems because it was designed for this purpose. Although there is much overlap among these more recent theories, often using different terms for similar concepts, we use Halpern and Butler’s conceptualization to make our point: Yes, intelligence (i.e., CT) can be enhanced and used for solving a real-world problem like COVID-19.

4. Critical Thinking as an Applied Model for Intelligence

One definition of intelligence that directly addresses the question about intelligence and real-world problem solving comes from Nickerson ( 2020, p. 205 ): “the ability to learn, to reason well, to solve novel problems, and to deal effectively with novel problems—often unpredictable—that confront one in daily life.” Using this definition, the question of whether intelligent thinking can solve a world problem like the novel coronavirus is a resounding “yes” because solutions to real-world novel problems are part of his definition. This is a popular idea in the general public. For example, over 1000 business managers and hiring executives said that they want employees who can think critically based on the belief that CT skills will help them solve work-related problems ( Hart Research Associates 2018 ).

We define CT as the use of those cognitive skills or strategies that increase the probability of a desirable outcome. It is used to describe thinking that is purposeful, reasoned, and goal directed--the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions, when the thinker is using skills that are thoughtful and effective for the particular context and type of thinking task. International surveys conducted by the OECD ( 2019, p. 16 ) established “key information-processing competencies” that are “highly transferable, in that they are relevant to many social contexts and work situations; and ‘learnable’ and therefore subject to the influence of policy.” One of these skills is problem solving, which is one subset of CT skills.

The CT model of intelligence is comprised of two components: (1) understanding information at a deep, meaningful level and (2) appropriate use of CT skills. The underlying idea is that CT skills can be identified, taught, and learned, and when they are recognized and applied in novel settings, the individual is demonstrating intelligent thought. CT skills include judging the credibility of an information source, making cost–benefit calculations, recognizing regression to the mean, understanding the limits of extrapolation, muting reactance responses, using analogical reasoning, rating the strength of reasons that support and fail to support a conclusion, and recognizing hindsight bias or confirmation bias, among others. Critical thinkers use these skills appropriately, without prompting, and usually with conscious intent in a variety of settings.

One of the key concepts in this model is that CT skills transfer in appropriate situations. Thus, assessments using situational judgments are needed to assess whether particular skills have transferred to a novel situation where it is appropriate. In an assessment created by the first author ( Halpern 2018 ), short paragraphs provide information about 20 different everyday scenarios (e.g., A speaker at the meeting of your local school board reported that when drug use rises, grades decline; so schools need to enforce a “war on drugs” to improve student grades); participants provide two response formats for every scenario: (a) constructed responses where they respond with short written responses, followed by (b) forced choice responses (e.g., multiple choice, rating or ranking of alternatives) for the same situations.

There is a large and growing empirical literature to support the assertion that CT skills can be learned and will transfer (when taught for transfer). See for example, Holmes et al. ( 2015 ), who wrote in the prestigious Proceedings of the National Academy of Sciences , that there was “significant and sustained improvement in students’ critical thinking behavior” (p. 11,199) for students who received CT instruction. Abrami et al. ( 2015, para. 1 ) concluded from a meta-analysis that “there are effective strategies for teaching CT skills, both generic and content specific, and CT dispositions, at all educational levels and across all disciplinary areas.” Abrami et al. ( 2008, para. 1 ), included 341 effect sizes in a meta-analysis. They wrote: “findings make it clear that improvement in students’ CT skills and dispositions cannot be a matter of implicit expectation.” A strong test of whether CT skills can be used for real-word problems comes from research by Butler et al. ( 2017 ). Community adults and college students (N = 244) completed several scales including an assessment of CT, an intelligence test, and an inventory of real-life events. Both CT scores and intelligence scores predicted individual outcomes on the inventory of real-life events, but CT was a stronger predictor.

Heijltjes et al. ( 2015, p. 487 ) randomly assigned participants to either a CT instruction group or one of six other control conditions. They found that “only participants assigned to CT instruction improved their reasoning skills.” Similarly, when Halpern et al. ( 2012 ) used random assignment of participants to either a learning group where they were taught scientific reasoning skills using a game format or a control condition (which also used computerized learning and was similar in length), participants in the scientific skills learning group showed higher proportional learning gains than students who did not play the game. As the body of additional supportive research is too large to report here, interested readers can find additional lists of CT skills and support for the assertion that these skills can be learned and will transfer in Halpern and Dunn ( Forthcoming ). There is a clear need for more high-quality research on the application and transfer of CT and its relationship to IQ.

5. Pandemics: COVID-19 as a Consequential Real-World Problem

A pandemic occurs when a disease runs rampant over an entire country or even the world. Pandemics have occurred throughout history: At the time of writing this article, COVID-19 is a world-wide pandemic whose actual death rate is unknown but estimated with projections of several million over the course of 2021 and beyond ( Mega 2020 ). Although vaccines are available, it will take some time to inoculate most or much of the world’s population. Since March 2020, national and international health agencies have created a list of actions that can slow and hopefully stop the spread of COVID (e.g., wearing face masks, practicing social distancing, avoiding group gatherings), yet many people in the United States and other countries have resisted their advice.

Could instruction in CT encourage more people to accept and comply with simple life-saving measures? There are many possible reasons to believe that by increasing citizens’ CT abilities, this problematic trend can be reversed for, at least, some unknown percentage of the population. We recognize the long history of social and cognitive research showing that changing attitudes and behaviors is difficult, and it would be unrealistic to expect that individuals with extreme beliefs supported by their social group and consistent with their political ideologies are likely to change. For example, an Iranian cleric and an orthodox rabbi both claimed (separately) that the COVID-19 vaccine can make people gay ( Marr 2021 ). These unfounded opinions are based on deeply held prejudicial beliefs that we expect to be resistant to CT. We are targeting those individuals who beliefs are less extreme and may be based on reasonable reservations, such as concern about the hasty development of the vaccine and the lack of long-term data on its effects. There should be some unknown proportion of individuals who can change their COVID-19-related beliefs and actions with appropriate instruction in CT. CT can be a (partial) antidote for the chaos of the modern world with armies of bots creating content on social media, political and other forces deliberately attempting to confuse issues, and almost all media labeled “fake news” by social influencers (i.e., people with followers that sometimes run to millions on various social media). Here, are some CT skills that could be helpful in getting more people to think more critically about pandemic-related issues.

Reasoning by Analogy and Judging the Credibility of the Source of Information

Early communications about the ability of masks to prevent the spread of COVID from national health agencies were not consistent. In many regions of the world, the benefits of wearing masks incited prolonged and acrimonious debates ( Tang 2020 ). However, after the initial confusion, virtually all of the global and national health organizations (e.g., WHO, National Health Service in the U. K., U. S. Centers for Disease Control and Prevention) endorse masks as a way to slow the spread of COVID ( Cheng et al. 2020 ; Chu et al. 2020 ). However, as we know, some people do not trust governmental agencies and often cite the conflicting information that was originally given as a reason for not wearing a mask. There are varied reasons for refusing to wear a mask, but the one most often cited is that it is against civil liberties ( Smith 2020 ). Reasoning by analogy is an appropriate CT skill for evaluating this belief (and a key skill in legal thinking). It might be useful to cite some of the many laws that already regulate our behavior such as, requiring health inspections for restaurants, setting speed limits, mandating seat belts when riding in a car, and establishing the age at which someone can consume alcohol. Individuals would be asked to consider how the mandate to wear a mask compares to these and other regulatory laws.

Another reason why some people resist the measures suggested by virtually every health agency concerns questions about whom to believe. Could training in CT change the beliefs and actions of even a small percentage of those opposed to wearing masks? Such training would include considering the following questions with practice across a wide domain of knowledge: (a) Does the source have sufficient expertise? (b) Is the expertise recent and relevant? (c) Is there a potential for gain by the information source, such as financial gain? (d) What would the ideal information source be and how close is the current source to the ideal? (e) Does the information source offer evidence that what they are recommending is likely to be correct? (f) Have you traced URLs to determine if the information in front of you really came from the alleged source?, etc. Of course, not everyone will respond in the same way to each question, so there is little likelihood that we would all think alike, but these questions provide a framework for evaluating credibility. Donovan et al. ( 2015 ) were successful using a similar approach to improve dynamic decision-making by asking participants to reflect on questions that relate to the decision. Imagine the effect of rigorous large-scale education in CT from elementary through secondary schools, as well as at the university-level. As stated above, empirical evidence has shown that people can become better thinkers with appropriate instruction in CT. With training, could we encourage some portion of the population to become more astute at judging the credibility of a source of information? It is an experiment worth trying.

6. Making Cost—Benefit Assessments for Actions That Would Slow the Spread of COVID-19

Historical records show that refusal to wear a mask during a pandemic is not a new reaction. The epidemic of 1918 also included mandates to wear masks, which drew public backlash. Then, as now, many people refused, even when they were told that it was a symbol of “wartime patriotism” because the 1918 pandemic occurred during World War I ( Lovelace 2020 ). CT instruction would include instruction in why and how to compute cost–benefit analyses. Estimates of “lives saved” by wearing a mask can be made meaningful with graphical displays that allow more people to understand large numbers. Gigerenzer ( 2020 ) found that people can understand risk ratios in medicine when the numbers are presented as frequencies instead of probabilities. If this information were used when presenting the likelihood of illness and death from COVID-19, could we increase the numbers of people who understand the severity of this disease? Small scale studies by Gigerenzer have shown that it is possible.

Analyzing Arguments to Determine Degree of Support for a Conclusion

The process of analyzing arguments requires that individuals rate the strength of support for and against a conclusion. By engaging in this practice, they must consider evidence and reasoning that may run counter to a preferred outcome. Kozyreva et al. ( 2020 ) call the deliberate failure to consider both supporting and conflicting data “deliberate ignorance”—avoiding or failing to consider information that could be useful in decision-making because it may collide with an existing belief. When applied to COVID-19, people would have to decide if the evidence for and against wearing a face mask is a reasonable way to stop the spread of this disease, and if they conclude that it is not, what are the costs and benefits of not wearing masks at a time when governmental health organizations are making them mandatory in public spaces? Again, we wonder if rigorous and systematic instruction in argument analysis would result in more positive attitudes and behaviors that relate to wearing a mask or other real-world problems. We believe that it is an experiment worth doing.

7. Conclusions

We believe that teaching CT is a worthwhile approach for educating the general public in order to improve reasoning and motivate actions to address, avert, or ameliorate real-world problems like the COVID-19 pandemic. Evidence suggests that CT can guide intelligent responses to societal and global problems. We are NOT claiming that CT skills will be a universal solution for the many real-world problems that we confront in contemporary society, or that everyone will substitute CT for other decision-making practices, but we do believe that systematic education in CT can help many people become better thinkers, and we believe that this is an important step toward creating a society that values and practices routine CT. The challenges are great, but the tools to tackle them are available, if we are willing to use them.

Author Contributions

Conceptualization, D.F.H. and D.S.D.; resources, D.F.H.; data curation, writing—original draft preparation, D.F.H.; writing—review and editing, D.F.H. and D.S.D. All authors have read and agreed to the published version of the manuscript.

This research received no external funding.

Institutional Review Board Statement

No IRB Review.

Informed Consent Statement

No Informed Consent.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Abrami Philip C., Bernard Robert M., Borokhovski Evgueni, Wade C. Anne, Surkes Michael A., Tamim Rana, Zhang Dai. Instructional interventions affecting critical thinking skills and dispositions: A Stage 1 meta-analysis. Review of Educational Research. 2008; 78 :1102–34. doi: 10.3102/0034654308326084. [ CrossRef ] [ Google Scholar ]
• Abrami Philip C., Bernard Robert M., Borokhovski Evgueni, Waddington David I., Wade C. Anne. Strategies for teaching students to think critically: A meta-analysis. Review of Educational Research. 2015; 85 :275–341. doi: 10.3102/0034654314551063. [ CrossRef ] [ Google Scholar ]
• Burhan Nik Ahmad Sufian, Kurniawan Yohan, Sidek Abdul Halim, Mohamad Mohd Rosli. Crimes and the Bell curve: Th e role of people with high, average, and low intelligence. Intelligence. 2014; 47 :12–22. doi: 10.1016/j.intell.2014.08.005. [ CrossRef ] [ Google Scholar ]
• Butler Heather A., Pentoney Christopher, Bong Maebelle P. Predicting real-world outcomes: Critical thinking ability is a better predictor of life decisions than intelligence. Thinking Skills and Creativity. 2017; 25 :38–46. doi: 10.1016/j.tsc.2017.06.005. [ CrossRef ] [ Google Scholar ]
• Cheng Vincent Chi-Chung, Wong Shuk-Ching, Chuang Vivien Wai-Man, So Simon Yung-Chun, Chen Jonathan Hon-Kwan, Sridhar Sidharth, To Kelvin Kai-Wwang, Chan Jasper Fuk-Wu, Hung Ivan Fan-Ngai, Ho Pak-Leung, et al. The role of community-wide wearing of face mask for control of coronavirus disease 2019 (COVID-19) epidemic due to SARS-CoV-2. Journal of Infectious Disease. 2020; 81 :107–14. doi: 10.1016/j.jinf.2020.04.024. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chu Derek K., Aki Elie A., Duda Stephanie, Solo Karla, Yaacoub Sally, Schunemann Holger J. Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: A system atic review and meta-analysis. Lancet. 2020; 395 :1973–87. doi: 10.1016/S0140-6736(20)31142-9. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Donovan Sarah J., Guss C. Dominick, Naslund Dag. Improving dynamic decision-making through training and self-re flection. Judgment and Decision Making. 2015; 10 :284–95. [ Google Scholar ]
• Drozdick Lisa Whipple, Wahlstrom Dustin, Zhu Jianjun, Weiss Lawrence G. The Wechsler Adult Intelligence Scale—Fourth Edition and the Wechsler Memory Scale—Fourth Edition. In: Flanagan Dawn P., Harrison Patti L., editors. Contemporary Intellectual as Sessment: Theories, Tests, and Issues. The Guilford Press; New York: 2012. pp. 197–223. [ Google Scholar ]
• Gigerenzer Gerd. When all is just a click away: Is critical thinking obsolete in the digital age? In: Sternberg Robert J., Halpern Diane F., editors. Critical Thinking IN Psychology. 2nd ed. Cambridge University Press; Cambridge: 2020. pp. 197–223. [ Google Scholar ]
• Gottfredson Linda S. Why g matters: The complexity of everyday life. Intelligence. 1997; 24 :79–132. doi: 10.1016/S0160-2896(97)90014-3. [ CrossRef ] [ Google Scholar ]
• Grossmann Igor, Varnum Michael E. W., Na Jinkyung, Kitayama Shinobu, Nisbett Richard E. A route to well-being: Intelligence ver sus wise reasoning. Journal of Experimental Psychology: General. 2013; 142 :944–53. doi: 10.1037/a0029560. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Halpern Diane F. Halpern Critical Thinking Assessment. Schuhfried Test Publishers; Modling: 2018. [(accessed on 30 March 2021)]. Available online: www.schuhfried.com [ Google Scholar ]
• Halpern Diane F., Butler Heather A. Is critical thinking a better model of intelligence? In: Sternberg Robert J., editor. The nature of Intelligence. 2nd ed. Cambridge University Press; Cambridge: 2020. pp. 183–96. [ Google Scholar ]
• Halpern Diane F., Dunn Dana S. Thought and Knowledge: An Introduction to Critical Thinking. 6th ed. Taylor & Francis; New York: Forthcoming. in press. [ Google Scholar ]
• Halpern Diane F., Millis Keith, Graesser Arthur, Butler Heather, Forsyth Carol, Cai Zhiqiang. Operation ARA: A computerized learn ing game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity. 2012; 7 :93–100. doi: 10.1016/j.tsc.2012.03.006. [ CrossRef ] [ Google Scholar ]
• Hart Research Associates [(accessed on 30 March 2021)]; Employers Express Confidence in Colleges and Universities: See College as Worth the Investment, New Research Finds. 2018 Aug 29; Available online: https://hartresearch.com/employers-express-confidence-in-colleges-and-universities-see-college-as-worth-the-investment-new-research-finds/
• Heijltjes Anita, Gog Tamara van, Lippink Jimmie, Paas Fred. Unraveling the effects of critical thinking instructions, practice, and self-explanation on students’ reasoning performance. Instructional Science. 2015; 43 :487–506. doi: 10.1007/s11251-015-9347-8. [ CrossRef ] [ Google Scholar ]
• Holmes Natasha G., Wieman Carl E., Bonn DougA. Teaching critical thinking. Proceedings of the National Academy of Sciences. 2015; 112 :11199–204. doi: 10.1073/pnas.1505329112. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hunter John E., Schmidt Frank L. Intelligence and job performance: Economic and social implications. Psychology, Public Policy, and Law. 1996; 2 :447–72. doi: 10.1037/1076-8971.2.3-4.447. [ CrossRef ] [ Google Scholar ]
• Kozyreva Anastasia, Lewandowsky Stephan, Hertwig Ralph. Citizens versus the internet: Confronting digital challenges with cognitive tools. [(accessed on 30 March 2021)]; Psychological Science in the Public Interest. 2020 21 doi: 10.1177/1529100620946707. Available online: https://www.psychologi calscience.org/publications/confronting-digital-challenges-with-cognitive-tools.html [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kretzschmar Andre, Neubert Jonas C., Wusternberg Sascha, Greiff Samuel. Construct validity of complex problem- solv ing: A comprehensive view on different facts of intelligence and school grades. Intelligence. 2016; 54 :55–69. doi: 10.1016/j.intell.2015.11.004. [ CrossRef ] [ Google Scholar ]
• Lang Jonas W.B., Kell Harrison J. General mental ability and specific abilities: Their relative importance for extrinsic career success. Journal of Applied Psychology. 2020; 105 :1047–61. doi: 10.1037/apl0000472. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lovelace Berkeley., Jr. Medical Historians Compare the Coronavirus to the 1918 Flu Pandemic: Both Were Highly Political. [(accessed on 30 March 2021)]; CNBC. 2020 Available online: https://www.cnbc.com/2020/09/28/comparing-1918-flu-vs-corona virus.html?fbclid=IwAR1RAVRUOIdN9qqvNnMPimf5Q4XfV-pn_qdC3DwcfnPu9kavwumDI2zq9Xs
• Marr Rhuaridh. Iranian Cleric Claims COVID-19 Vaccine Can Make People Gay. [(accessed on 30 March 2021)]; Metro Weekly. 2021 Available online: https://www.metroweekly.com/2021/02/iranian-cleric-claims-covid-19-vaccine-can-make-people-gay/
• McCabe Kira O., Lubinski David, Benbow Camilla P. Who shines most among the brightest?: A 25-year longitudinal study of elite STEM graduate students. Journal of Personality and Social Psychology. 2020; 119 :390–416. doi: 10.1037/pspp0000239. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Mega Emiliano R. COVID Has Killed more than One Million People. How Many more will Die? [(accessed on 30 March 2021)]; Nature. 2020 Available online: https://www.nature.com/articles/d41586-020-02762-y [ PubMed ]
• Nickerson Raymond S. Developing intelligence through instruction. In: Sternberg Robert J., editor. The Cambridge Handbook of Intelligence. 2nd ed. Cambridge University Press; Cambridge: 2020. pp. 205–37. [ Google Scholar ]
• OECD . The Survey of Adult Skills: Reader’s Companion. 3rd ed. OECD Publishing; Paris: 2019. OECD Skills Studies. [ CrossRef ] [ Google Scholar ]
• Smith Matthew. Why won’t Britons Wear Face Masks? [(accessed on 30 March 2021)]; YouGov. 2020 Available online: https://yougov.co.uk/topics/health/articles-reports/2020/07/15/why-wont-britons-wear-face-masks
• Stanovich Keith E. What Intelligence Tests Miss: The Psychology of Rational Thought. Yale University Press; New Haven: 2009. [ Google Scholar ]
• Stanovich Keith E., West Richard F. On the failure of cognitive ability to predict my-side bias and one-sided thinking biases. Thinking & Reasoning. 2008; 14 :129–67. doi: 10.1080/13546780701679764. [ CrossRef ] [ Google Scholar ]
• Stanovich Keith E., West Richard F. What intelligence tests miss. The Psychologist. 2014; 27 :80–83. doi: 10.5840/inquiryctnews201126216. [ CrossRef ] [ Google Scholar ]
• Sternberg Robert J. A theory of adaptive intelligence and its relation to general intelligence. Journal of Intelligence. 2019; 7 :23. doi: 10.3390/jintelligence7040023. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tang Julian W. COVID-19: Interpreting scientific evidence—Uncertainty, confusion, and delays. BMC Infectious Diseases. 2020; 20 :653. doi: 10.1186/s12879-020-05387-8. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Wirthwein Linda, Rost Detlef H. Giftedness and subjective well-being: A study with adults. Learning and Individuals Differences. 2011; 21 :182–86. doi: 10.1016/j.lindif.2011.01.001. [ CrossRef ] [ Google Scholar ]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• PubMed/Medline
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Predicting everyday critical thinking: a review of critical thinking assessments.

1. Introduction

2. how critical thinking impacts everyday life, 3. critical thinking: skills and dispositions.

“the use of those cognitive skills and abilities that increase the probability of a desirable outcome. It is used to describe thinking that is purposeful, reasoned, and goal directed—the kind of thinking involved in solving problems, formulating inferences, calculating likelihoods, and making decisions” ( Halpern 2014, p. 8 ).

4. Measuring Critical Thinking

4.1. practical challenges, 4.2. critical thinking assessments, 4.2.1. california critical thinking dispositions inventory (cctdi; insight assessment, inc. n.d. ), 4.2.2. california critical thinking skills test (cctst; insight assessment, inc. n.d. ), 4.2.3. cornell critical thinking test (cctt; the critical thinking company n.d. ), 4.2.4. california measure of mental motivation (cm3; insight assessment, inc. n.d. ), 4.2.5. ennis–weir critical thinking essay test ( ennis and weir 2005 ), 4.2.6. halpern critical thinking assessment (hcta; halpern 2012 ), 4.2.7. test of everyday reasoning (ter; insight assessment, inc. n.d. ), 4.2.8. watson–glaser tm ii critical thinking appraisal (w-gii; ncs pearson, inc. 2009 ).

“Virtual employees, or employees who work from home via a computer, are an increasing trend. In the US, the number of virtual employees has increased by 39% in the last two years and 74% in the last five years. Employing virtual workers reduces costs and makes it possible to use talented workers no matter where they are located globally. Yet, running a workplace with virtual employees might entail miscommunication and less camaraderie and can be more time-consuming than face-to-face interaction”.

5. Conclusions

Institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Ali, Marium, and AJLabs. 2023. How Many Years Does a Typical User Spend on Social Media? Doha: Al Jazeera. Available online: https://www.aljazeera.com/news/2023/6/30/how-many-years-does-a-typical-user-spend-on-social-media (accessed on 13 November 2023).
• Arendasy, Martin, Lutz Hornke, Markus Sommer, Michaela Wagner-Menghin, Georg Gittler, Joachim Häusler, Bettina Bognar, and M. Wenzl. 2012. Intelligenz-Struktur-Batterie (Intelligence Structure Battery; INSBAT) . Mödling: Schuhfried GmbH. [ Google Scholar ]
• Arum, Richard, and Josipa Roksa. 2010. Academically Adrift . Chicago: The University of Chicago Press. [ Google Scholar ]
• Bakshy, Eytan, Solomon Messing, and Lada Adamic. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science 348: 1130–32. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Bart, William. 2010. The Measurement and Teaching of Critical Thinking Skills . Tokyo: Invited colloquium given at the Center for Research on Education Testing. [ Google Scholar ]
• Bruine de Bruin, Wandi, Andrew Parker, and Baruch Fischhoff. 2007. Individual differences in adult decision-making competence. Journal of Personality and Social Psychology 92: 938–56. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Butler, Heather. 2012. Halpern Critical Thinking Assessment predicts real-world outcomes of critical thinking. Applied Cognitive Psychology 26: 721–29. [ Google Scholar ] [ CrossRef ]
• Butler, Heather, and Diane Halpern. 2020. Critical Thinking Impacts Our Everyday Lives. In Critical Thinking in Psychology , 2nd ed. Edited by Robert Sternberg and Diane Halpern. Cambridge, UK: Cambridge University Press. [ Google Scholar ] [ CrossRef ]
• Butler, Heather, Chris Dwyer, Michael Hogan, Amanda Franco, Silvia Rivas, Carlos Saiz, and Leandro Almeida. 2012. Halpern Critical Thinking Assessment and real-world outcomes: Cross-national applications. Thinking Skills and Creativity 7: 112–21. [ Google Scholar ] [ CrossRef ]
• Butler, Heather, Chris Pentoney, and Mabelle Bong. 2017. Critical thinking ability is a better predictor of life decisions than intelligence. Thinking Skills and Creativity 24: 38–46. [ Google Scholar ] [ CrossRef ]
• Ennis, Robert. 2005. The Ennis-Weir Critical Thinking Essay Test . Urbana: The Illinois Critical Thinking Project. Available online: http://faculty.ed.uiuc.edu/rhennis/supplewmanual1105.htm (accessed on 22 October 2023).
• Ennis, Robert, and Eric Weir. 2005. Ennis-Weir Critical Thinking Essay Test . Seaside: The Critical Thinking Company. Available online: https://www.academia.edu/1847582/The_Ennis_Weir_Critical_Thinking_Essay_Test_An_Instrument_for_Teaching_and_Testing (accessed on 22 October 2023).
• Facione, Peter. 1990. California Critical Thinking Dispositions Inventory . Millbrae: The California Academic Press. [ Google Scholar ]
• Facione, Peter, Noreen Facione, and Kathryn Winterhalter. 2012. The Test of Everyday Reasoning—(TER): Test Manual . Millbrae: California Academic Press. [ Google Scholar ]
• Forsyth, Carol, Philip Pavlik, Arthur C. Graesser, Zhiqiang Cai, Mae-lynn Germany, Keith Millis, Robert P. Dolan, Heather Butler, and Diane Halpern. 2012. Learning gains for core concepts in a serious game on scientific reasoning. In Proceedings of the 5th International Conference on Educational Data Mining . Edited by Kalina Yacef, Osmar Zaïane, Arnon Hershkovitz, Michael Yudelson and John Stamper. Chania: International Educational Data Mining Society, pp. 172–75. [ Google Scholar ]
• French, Brian, Brian Hand, William Therrien, and Juan Valdivia Vazquez. 2012. Detection of sex differential item functioning in the Cornell Critical Thinking Test. European Journal of Psychological Assessment 28: 201–7. [ Google Scholar ] [ CrossRef ]
• Frenkel, Sheera, and Mike Isaac. 2018. Facebook ‘Better Prepared’ to Fight Election Interference, Mark Zuckerberg Says . Manhattan: New York Times. Available online: https://www.nytimes.com/2018/09/13/technology/facebook-elections-mark-zuckerberg.html (accessed on 22 October 2023).
• Gheorghia, Olimpiu. 2018. Romania’s Measles Outbreak Kills Dozens of Children: Some Doctors Complain They Don’t Have Sufficient Stock of Vaccines . New York: Associated Press. Available online: https://www.nbcnews.com/health/health-news/romania-s-measles-outbreak-kills-dozens-children-n882771 (accessed on 13 November 2023).
• Giancarlo, Carol, Stephen Bloom, and Tim Urdan. 2004. Assessing secondary students’ disposition toward critical thinking: Development of the California Measure of Mental Motivation. Educational and Psychological Measurement 64: 347–64. [ Google Scholar ] [ CrossRef ]
• Halpern, Diane. 1998. Teaching critical thinking for transfer across domains: Dispositions, skills, structure training, and metacognitive monitoring. American Psychologist 53: 449–55. [ Google Scholar ] [ CrossRef ]
• Halpern, Diane. 2012. Halpern Critical Thinking Assessment . Mödling: Schuhfried (Vienna Test System). Available online: http://www.schuhfried.com/vienna-test-system-vts/all-tests-from-a-z/test/hcta-halpern-critical-thinking-assessment-1/ (accessed on 13 January 2013).
• Halpern, Diane. 2014. Thought and Knowledge: An Introduction to Critical Thinking , 5th ed. New York: Routledge Publishers. [ Google Scholar ]
• Halpern, Diane, Keith Millis, Arthur Graesser, Heather Butler, Carol Forsyth, and Zhiqiang Cai. 2012. Operation ARIES!: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity 7: 93–100. [ Google Scholar ] [ CrossRef ]
• Huber, Christopher, and Nathan Kuncel. 2015. Does college teach critical thinking? A meta-analysis. Review of Educational Research 86: 431–68. [ Google Scholar ] [ CrossRef ]
• Insight Assessment, Inc. n.d. Critical Thinking Attribute Tests: Manuals and Assessment Information . Hermosa Beach: Insight Assessment. Available online: http://www.insightassessment.com (accessed on 22 October 2023).
• Jain, Anjali, Jaclyn Marshall, Ami Buikema, Tim Bancroft, Jonathan Kelly, and Craig Newschaffer. 2015. Autism occurrence by MMR vaccine status among US children with older siblings with and without autism. Journal of the American Medical Association 313: 1534–40. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Klee, Miles, and Nikki McCann Ramirez. 2023. AI Has Made the Israel-Hamas Misinformation Epidemic Much, Much Worse . New York: Rollingstone. Available online: https://www.rollingstone.com/politics/politics-features/israel-hamas-misinformation-fueled-ai-images-1234863586/amp/?fbclid=PAAabKD4u1FRqCp-y9z3VRA4PZZdX52DTQEn8ruvHeGsBrNguD_F2EiMrs3A4_aem_AaxFU9ovwsrXAo39I00d-8NmcpRTVBCsUd_erAUwlAjw16x1shqeC6s22OCpSSx2H-w (accessed on 27 October 2023).
• Klepper, David. 2022. Poll: Most in US Say Misinformation Spurs Extremism, Hate . New York: Associated Press-NORC Center for Public Affairs Research. Available online: https://apnorc.org/poll-most-in-us-say-misinformation-spurs-extremism-hate/ (accessed on 27 October 2023).
• Landis, Richard, and William Michael. 1981. The factorial validity of three measures of critical thinking within the context of Guilford’s Structure-of-Intellect Model for a sample of ninth grade students. Educational and Psychological Measurement 41: 1147–66. [ Google Scholar ] [ CrossRef ]
• Liedke, Jacob, and Luxuan Wang. 2023. Social Media and News Fact Sheet . Washington, DC: Pew Research Center. Available online: https://www.pewresearch.org/journalism/fact-sheet/social-media-and-news-fact-sheet/ (accessed on 15 November 2023).
• Lilienfeld, Scott, Rachel Ammirati, and Kristin Landfield. 2009. Giving debiasing away: Can psychological research on correcting cognitive errors promote human welfare? Perspective on Psychological Science 4: 390–98. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Michael, Joan, Roberta Devaney, and William Michael. 1980. The factorial validity of the Cornell Critical Thinking Test for a junior high school sample. Educational and Psychological Measurement 40: 437–50. [ Google Scholar ] [ CrossRef ]
• National Center for Health Statistics. 2015. Health, United States, 2015, with Special Feature on Racial and Ethnic Health Disparities ; Washington, DC: U.S. Government Printing Office.
• NCS Pearson, Inc. 2009. Watson-Glaser II Critical Thinking Appraisal: Technical Manual and User’s Guide . London: Pearson. Available online: http://www.talentlens.com/en/downloads/supportmaterials/WGII_Technical_Manual.pdf (accessed on 22 October 2023).
• Stanovich, Keith, and Richard West. 2008. On the failure of cognitive ability to predict myside and one-sided thinking biases. Thinking & Reasoning 14: 129–67. [ Google Scholar ] [ CrossRef ]
• The Critical Thinking Company. n.d. Critical Thinking Company. Available online: www.criticalthinking.com (accessed on 13 October 2023).
• Tsipursky, Gleb. 2018. (Dis)trust in Science: Can We Cure the Scourge of Misinformation? New York: Scientific American. Available online: https://blogs.scientificamerican.com/observations/dis-trust-in-science/ (accessed on 11 April 2022).
• Walsh, Catherina, Lisa Seldomridge, and Karen Badros. 2007. California Critical Thinking Disposition Inventory: Further factor analytic examination. Perceptual and Motor Skills 104: 141–51. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• World Health Organization. 2018. Europe Observes a 4-Fold Increase in Measles Cases in 2017 Compared to Previous Year . Geneva: World Health Organization. Available online: http://www.euro.who.int/en/media-centre/sections/press-releases/2018/europe-observes-a-4-fold-increase-in-measles-cases-in-2017-compared-to-previous-year (accessed on 22 October 2023).

Share and Cite

Butler, H.A. Predicting Everyday Critical Thinking: A Review of Critical Thinking Assessments. J. Intell. 2024 , 12 , 16. https://doi.org/10.3390/jintelligence12020016

Butler HA. Predicting Everyday Critical Thinking: A Review of Critical Thinking Assessments. Journal of Intelligence . 2024; 12(2):16. https://doi.org/10.3390/jintelligence12020016

Butler, Heather A. 2024. "Predicting Everyday Critical Thinking: A Review of Critical Thinking Assessments" Journal of Intelligence 12, no. 2: 16. https://doi.org/10.3390/jintelligence12020016

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

OPINION article

Redefining critical thinking: teaching students to think like scientists.

• Department of Psychology, MacEwan University, Edmonton, AB, Canada

From primary to post-secondary school, critical thinking (CT) is an oft cited focus or key competency (e.g., DeAngelo et al., 2009 ; California Department of Education, 2014 ; Alberta Education, 2015 ; Australian Curriculum Assessment and Reporting Authority, n.d. ). Unfortunately, the definition of CT has become so broad that it can encompass nearly anything and everything (e.g., Hatcher, 2000 ; Johnson and Hamby, 2015 ). From discussion of Foucault, critique and the self ( Foucault, 1984 ) to Lawson's (1999) definition of CT as the ability to evaluate claims using psychological science, the term critical thinking has come to refer to an ever-widening range of skills and abilities. We propose that educators need to clearly define CT, and that in addition to teaching CT, a strong focus should be placed on teaching students how to think like scientists. Scientific thinking is the ability to generate, test, and evaluate claims, data, and theories (e.g., Bullock et al., 2009 ; Koerber et al., 2015 ). Simply stated, the basic tenets of scientific thinking provide students with the tools to distinguish good information from bad. Students have access to nearly limitless information, and the skills to understand what is misinformation or a questionable scientific claim is crucially important ( Smith, 2011 ), and these skills may not necessarily be included in the general teaching of critical thinking ( Wright, 2001 ).

This is an issue of more than semantics. While some definitions of CT include key elements of the scientific method (e.g., Lawson, 1999 ; Lawson et al., 2015 ), this emphasis is not consistent across all interpretations of CT ( Huber and Kuncel, 2016 ). In an attempt to provide a comprehensive, detailed definition of CT, the American Philosophical Association (APA), outlined six CT skills, 16 subskills, and 19 dispositions ( Facione, 1990 ). Skills include interpretation, analysis, and inference; dispositions include inquisitiveness and open-mindedness. 1 From our perspective, definitions of CT such as those provided by the APA or operationally defined by researchers in the context of a scholarly article (e.g., Forawi, 2016 ) are not problematic—the authors clearly define what they are referring to as CT. Potential problems arise when educators are using different definitions of CT, or when the banner of CT is applied to nearly any topic or pedagogical activity. Definitions such as those provided by the APA provide a comprehensive framework for understanding the multi-faceted nature of CT, however the definition is complex and may be difficult to work with at a policy level for educators, especially those who work primarily with younger students.

The need to develop scientific thinking skills is evident in studies showing that 55% of undergraduate students believe that a full moon causes people to behave oddly, and an estimated 67% of students believe creatures such as Bigfoot and Chupacabra exist, despite the lack of scientific evidence supporting these claims ( Lobato et al., 2014 ). Additionally, despite overwhelming evidence supporting the existence of anthropogenic climate change, and the dire need to mitigate its effects, many people still remain skeptical of climate change and its impact ( Feygina et al., 2010 ; Lewandowsky et al., 2013 ). One of the goals of education is to help students foster the skills necessary to be informed consumers of information ( DeAngelo et al., 2009 ), and providing students with the tools to think scientifically is a crucial component of reaching this goal. By focusing on scientific thinking in conjunction with CT, educators may be better able design specific policies that aim to facilitate the necessary skills students should have when they enter post-secondary training or the workforce. In other words, students should leave secondary school with the ability to rule out rival hypotheses, understand that correlation does not equal causation, the importance of falsifiability and replicability, the ability to recognize extraordinary claims, and use the principle of parsimony (e.g., Lett, 1990 ; Bartz, 2002 ).

Teaching scientific thinking is challenging, as people are vulnerable to trusting their intuitions and subjective observations and tend to prioritize them over objective scientific findings (e.g., Lilienfeld et al., 2012 ). Students and the public at large are prone to naïve realism, or the tendency to believe that our experiences and observations constitute objective reality ( Ross and Ward, 1996 ), when in fact our experiences and observations are subjective and prone to error (e.g., Kahneman, 2011 ). Educators at the post-secondary level tend to prioritize scientific thinking ( Lilienfeld, 2010 ), however many students do not continue on to a post-secondary program after they have completed high school. Further, students who are told they are learning critical thinking may believe they possess the skills to accurately assess the world around them. However, if they are not taught the specific skills needed to be scientifically literate, they may still fall prey to logical fallacies and biases. People tend to underestimate or not understand fallacies that can prevent them from making sound decisions ( Lilienfeld et al., 2001 ; Pronin et al., 2004 ; Lilienfeld, 2010 ). Thus, it is reasonable to think that a person who has not been adequately trained in scientific thinking would nonetheless consider themselves a strong critical thinker, and therefore would be even less likely consider his or her own personal biases. Another concern is that when teaching scientific thinking there is always the risk that students become overly critical or cynical (e.g., Mercier et al., 2017 ). By this, a student may be skeptical of nearly all findings, regardless of the supporting evidence. By incorporating and focusing on cognitive biases, instructors can help students understand their own biases, and demonstrate how the rigor of the scientific method can, at least partially, control for these biases.

Teaching CT remains controversial and confusing for many instructors ( Bensley and Murtagh, 2012 ). This is partly due to the lack of clarity in the definition of CT and the wide range of methods proposed to best teach CT ( Abrami et al., 2008 ; Bensley and Murtagh, 2012 ). For instance, Bensley and Spero (2014) found evidence for the effectiveness of direct approaches to teaching CT, a claim echoed in earlier research ( Abrami et al., 2008 ; Marin and Halpern, 2011 ). Despite their positive findings, some studies have failed to find support for measures of CT ( Burke et al., 2014 ) and others have found variable, yet positive, support for instructional methods ( Dochy et al., 2003 ). Unfortunately, there is a lack of research demonstrating the best pedagogical approaches to teaching scientific thinking at different grade levels. More research is needed to provide an empirically grounded approach to teach scientific thinking, and there is also a need to develop evidence based measures of scientific thinking that are grade and age appropriate. One approach to teaching scientific thinking may be to frame the topic in its simplest terms—the ability to “detect baloney” ( Sagan, 1995 ).

Sagan (1995) has promoted the tools necessary to recognize poor arguments, fallacies to avoid, and how to approach claims using the scientific method. The basic tenets of Sagan's argument apply to most claims, and have the potential to be an effective teaching tool across a range of abilities and ages. Sagan discusses the idea of a baloney detection kit, which contains the “tools” for skeptical thinking. The development of “baloney detection kits” which include age-appropriate scientific thinking skills may be an effective approach to teaching scientific thinking. These kits could include the style of exercises that are typically found under the banner of CT training (e.g., group discussions, evaluations of arguments) with a focus on teaching scientific thinking. An empirically validated kit does not yet exist, though there is much to draw from in the literature on pedagogical approaches to correcting cognitive biases, combatting pseudoscience, and teaching methodology (e.g., Smith, 2011 ). Further research is needed in this area to ensure that the correct, and age-appropriate, tools are part of any baloney detection kit.

Teaching Sagan's idea of baloney detection in conjunction with CT provides educators with a clear focus—to employ a pedagogical approach that helps students create sound and cogent arguments while avoiding falling prey to “baloney”. This is not to say that all of the information taught under the current banner of “critical thinking” is without value. In fact, many of the topics taught under the current approach of CT are important, even though they would not fit within the framework of some definitions of critical thinking. If educators want to ensure that students have the ability to be accurate consumers of information, a focus should be placed on including scientific thinking as a component of the science curriculum, as well as part of the broader teaching of CT.

Educators need to be provided with evidence-based approaches to teach the principles of scientific thinking. These principles should be taught in conjunction with evidence-based methods that mitigate the potential for fallacious reasoning and false beliefs. At a minimum, when students first learn about science, there should also be an introduction to the basics tenets of scientific thinking. Courses dedicated to promoting scientific thinking may also be effective. A course focused on cognitive biases, logical fallacies, and the hallmarks of scientific thinking adapted for each grade level may provide students with the foundation of solid scientific thinking skills to produce and evaluate arguments, and allow expansion of scientific thinking into other scholastic areas and classes. Evaluations of the efficacy of these courses would be essential, along with research to determine the best approach to incorporate scientific thinking into the curriculum.

If instructors know that students have at least some familiarity with the fundamental tenets of scientific thinking, the ability to expand and build upon these ideas in a variety of subject specific areas would further foster and promote these skills. For example, when discussing climate change, an instructor could add a brief discussion of why some people reject the science of climate change by relating this back to the information students will be familiar with from their scientific thinking courses. In terms of an issue like climate change, many students may have heard in political debates or popular culture that global warming trends are not real, or a “hoax” ( Lewandowsky et al., 2013 ). In this case, only teaching the data and facts may not be sufficient to change a student's mind about the reality of climate change ( Lewandowsky et al., 2012 ). Instructors would have more success by presenting students with the data on global warming trends as well as information on the biases that could lead some people reject the data ( Kowalski and Taylor, 2009 ; Lewandowsky et al., 2012 ). This type of instruction helps educators create informed citizens who are better able to guide future decision making and ensure that students enter the job market with the skills needed to be valuable members of the workforce and society as a whole.

By promoting scientific thinking, educators can ensure that students are at least exposed to the basic tenets of what makes a good argument, how to create their own arguments, recognize their own biases and those of others, and how to think like a scientist. There is still work to be done, as there is a need to put in place educational programs built on empirical evidence, as well as research investigating specific techniques to promote scientific thinking for children in earlier grade levels and develop measures to test if students have acquired the necessary scientific thinking skills. By using an evidence based approach to implement strategies to promote scientific thinking, and encouraging researchers to further explore the ideal methods for doing so, educators can better serve their students. When students are provided with the core ideas of how to detect baloney, and provided with examples of how baloney detection relates to the real world (e.g., Schmaltz and Lilienfeld, 2014 ), we are confident that they will be better able to navigate through the oceans of information available and choose the right path when deciding if information is valid.

Author Contribution

RS was the lead author and this paper, and both EJ and NW contributed equally.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

1. ^ There is some debate about the role of dispositional factors in the ability for a person to engage in critical thinking, specifically that dispositional factors may mitigate any attempt to learn CT. The general consensus is that while dispositional traits may play a role in the ability to think critically, the general skills to be a critical thinker can be taught ( Niu et al., 2013 ; Abrami et al., 2015 ).

Abrami, P. C., Bernard, R. M., Borokhovski, E., Waddington, D. I., Wade, C. A., and Persson, T. (2015). Strategies for teaching students to think critically a meta-analysis. Rev. Educ. Res. 85, 275–314. doi: 10.3102/0034654308326084

CrossRef Full Text | Google Scholar

Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., et al. (2008). Instructional interventions affecting critical thinking skills and dispositions: a stage 1 meta-analysis. Rev. Educ. Res. 78, 1102–1134. doi: 10.3102/0034654308326084

Alberta Education (2015). Ministerial Order on Student Learning . Available online at: https://education.alberta.ca/policies-and-standards/student-learning/everyone/ministerial-order-on-student-learning-pdf/

Australian Curriculum Assessment and Reporting Authority (n.d.). Available online at: http://www.australiancurriculum.edu.au

Bartz, W. R. (2002). Teaching skepticism via the CRITIC acronym and the skeptical inquirer. Skeptical Inquirer 17, 42–44.

Google Scholar

Bensley, D. A., and Murtagh, M. P. (2012). Guidelines for a scientific approach to critical thinking assessment. Teach. Psychol. 39, 5–16. doi: 10.1177/0098628311430642

Bensley, D. A., and Spero, R. A. (2014). Improving critical thinking skills and metacognitive monitoring through direct infusion. Think. Skills Creativ. 12, 55–68. doi: 10.1016/j.tsc.2014.02.001

Bullock, M., Sodian, B., and Koerber, S. (2009). “Doing experiments and understanding science: development of scientific reasoning from childhood to adulthood,” in Human Development from Early Childhood to Early Adulthood: Findings from a 20 Year Longitudinal Study , eds W. Schneider and M. Bullock (New York, NY: Psychology Press), 173–197.

Burke, B. L., Sears, S. R., Kraus, S., and Roberts-Cady, S. (2014). Critical analysis: a comparison of critical thinking changes in psychology and philosophy classes. Teach. Psychol. 41, 28–36. doi: 10.1177/0098628313514175

California Department of Education (2014). Standard for Career Ready Practice . Available online at: http://www.cde.ca.gov/nr/ne/yr14/yr14rel22.asp

DeAngelo, L., Hurtado, S., Pryor, J. H., Kelly, K. R., Santos, J. L., and Korn, W. S. (2009). The American College Teacher: National Norms for the 2007-2008 HERI Faculty Survey . Los Angeles, CA: Higher Education Research Institute.

Dochy, F., Segers, M., Van den Bossche, P., and Gijbels, D. (2003). Effects of problem-based learning: a meta-analysis. Learn. Instruct. 13, 533–568. doi: 10.1016/S0959-4752(02)00025-7

Facione, P. A. (1990). Critical thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. Research Findings and Recommendations. Newark, DE: American Philosophical Association.

Feygina, I., Jost, J. T., and Goldsmith, R. E. (2010). System justification, the denial of global warming, and the possibility of ‘system-sanctioned change’. Pers. Soc. Psychol. Bull. 36, 326–338. doi: 10.1177/0146167209351435

PubMed Abstract | CrossRef Full Text | Google Scholar

Forawi, S. A. (2016). Standard-based science education and critical thinking. Think. Skills Creativ. 20, 52–62. doi: 10.1016/j.tsc.2016.02.005

Foucault, M. (1984). The Foucault Reader . New York, NY: Pantheon.

Hatcher, D. L. (2000). Arguments for another definition of critical thinking. Inquiry 20, 3–8. doi: 10.5840/inquiryctnews20002016

Huber, C. R., and Kuncel, N. R. (2016). Does college teach critical thinking? A meta-analysis. Rev. Educ. Res. 86, 431–468. doi: 10.3102/0034654315605917

Johnson, R. H., and Hamby, B. (2015). A meta-level approach to the problem of defining “Critical Thinking”. Argumentation 29, 417–430. doi: 10.1007/s10503-015-9356-4

Kahneman, D. (2011). Thinking, Fast and Slow . New York, NY: Farrar, Straus and Giroux.

Koerber, S., Mayer, D., Osterhaus, C., Schwippert, K., and Sodian, B. (2015). The development of scientific thinking in elementary school: a comprehensive inventory. Child Dev. 86, 327–336. doi: 10.1111/cdev.12298

Kowalski, P., and Taylor, A. K. (2009). The effect of refuting misconceptions in the introductory psychology class. Teach. Psychol. 36, 153–159. doi: 10.1080/00986280902959986

Lawson, T. J. (1999). Assessing psychological critical thinking as a learning outcome for psychology majors. Teach. Psychol. 26, 207–209. doi: 10.1207/S15328023TOP260311

CrossRef Full Text

Lawson, T. J., Jordan-Fleming, M. K., and Bodle, J. H. (2015). Measuring psychological critical thinking: an update. Teach. Psychol. 42, 248–253. doi: 10.1177/0098628315587624

Lett, J. (1990). A field guide to critical thinking. Skeptical Inquirer , 14, 153–160.

Lewandowsky, S., Ecker, U. H., Seifert, C. M., Schwarz, N., and Cook, J. (2012). Misinformation and its correction: continued influence and successful debiasing. Psychol. Sci. Public Interest 13, 106–131. doi: 10.1177/1529100612451018

Lewandowsky, S., Oberauer, K., and Gignac, G. E. (2013). NASA faked the moon landing—therefore, (climate) science is a hoax: an anatomy of the motivated rejection of science. Psychol. Sci. 24, 622–633. doi: 10.1177/0956797612457686

Lilienfeld, S. O. (2010). Can psychology become a science? Pers. Individ. Dif. 49, 281–288. doi: 10.1016/j.paid.2010.01.024

Lilienfeld, S. O., Ammirati, R., and David, M. (2012). Distinguishing science from pseudoscience in school psychology: science and scientific thinking as safeguards against human error. J. Sch. Psychol. 50, 7–36. doi: 10.1016/j.jsp.2011.09.006

Lilienfeld, S. O., Lohr, J. M., and Morier, D. (2001). The teaching of courses in the science and pseudoscience of psychology: useful resources. Teach. Psychol. 28, 182–191. doi: 10.1207/S15328023TOP2803_03

Lobato, E., Mendoza, J., Sims, V., and Chin, M. (2014). Examining the relationship between conspiracy theories, paranormal beliefs, and pseudoscience acceptance among a university population. Appl. Cogn. Psychol. 28, 617–625. doi: 10.1002/acp.3042

Marin, L. M., and Halpern, D. F. (2011). Pedagogy for developing critical thinking in adolescents: explicit instruction produces greatest gains. Think. Skills Creativ. 6, 1–13. doi: 10.1016/j.tsc.2010.08.002

Mercier, H., Boudry, M., Paglieri, F., and Trouche, E. (2017). Natural-born arguers: teaching how to make the best of our reasoning abilities. Educ. Psychol. 52, 1–16. doi: 10.1080/00461520.2016.1207537

Niu, L., Behar-Horenstein, L. S., and Garvan, C. W. (2013). Do instructional interventions influence college students' critical thinking skills? A meta-analysis. Educ. Res. Rev. 9, 114–128. doi: 10.1016/j.edurev.2012.12.002

Pronin, E., Gilovich, T., and Ross, L. (2004). Objectivity in the eye of the beholder: divergent perceptions of bias in self versus others. Psychol. Rev. 111, 781–799. doi: 10.1037/0033-295X.111.3.781

Ross, L., and Ward, A. (1996). “Naive realism in everyday life: implications for social conflict and misunderstanding,” in Values and Knowledge , eds E. S. Reed, E. Turiel, T. Brown, E. S. Reed, E. Turiel and T. Brown (Hillsdale, NJ: Lawrence Erlbaum Associates Inc.), 103–135.

Sagan, C. (1995). Demon-Haunted World: Science as a Candle in the Dark . New York, NY: Random House.

Schmaltz, R., and Lilienfeld, S. O. (2014). Hauntings, homeopathy, and the Hopkinsville Goblins: using pseudoscience to teach scientific thinking. Front. Psychol. 5:336. doi: 10.3389/fpsyg.2014.00336

Smith, J. C. (2011). Pseudoscience and Extraordinary Claims of the Paranormal: A Critical Thinker's Toolkit . New York, NY: John Wiley and Sons.

Wright, I. (2001). Critical thinking in the schools: why doesn't much happen? Inform. Logic 22, 137–154. doi: 10.22329/il.v22i2.2579

Keywords: scientific thinking, critical thinking, teaching resources, skepticism, education policy

Citation: Schmaltz RM, Jansen E and Wenckowski N (2017) Redefining Critical Thinking: Teaching Students to Think like Scientists. Front. Psychol . 8:459. doi: 10.3389/fpsyg.2017.00459

Received: 13 December 2016; Accepted: 13 March 2017; Published: 29 March 2017.

Reviewed by:

Copyright © 2017 Schmaltz, Jansen and Wenckowski. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Rodney M. Schmaltz, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Previous Article
• Next Article

Critical Thinking

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Guest Access
• Get Permissions
• Cite Icon Cite
• Search Site

Thomas Kelley-Kemple; Critical Thinking. Harvard Educational Review 1 March 2021; 91 (1): 133–135. doi: https://doi.org/10.17763/1943-5045-91.1.133a

Download citation file:

• Ris (Zotero)
• Reference Manager

In the preface to Critical Thinking, Jonathan Haber notes that the term critical thinking has become a hallmark of almost any set of educational goals set out in the past thirty years. Yet, the myriad politicians, policy makers, indus-try leaders, and educators who cite the importance of critical thinking as an essential twenty-first-century skill rarely offer a concrete definition or set of criteria for what mastering this all-important skill means. Individual state standards or standards that do detail what is meant by critical thinking are rarely read by the general public. Even teachers may only view the standards amid a litany of other standards and skills that must be imparted to students. Haber sets out to fill this gap for a generalist audience. As such, Critical Thinking fits well among other volumes in MIT Press’s Essential Knowledge Series, a collection of volumes that cover specialized topics from a range of disciplines in a nuanced manner suitable for a nonspecialized audience. This orientation is crucial for understanding its strengths and weaknesses as a book that seeks to provide an overview of the history of, the skills required for, and current pedagogical efforts toward developing critical thinking.

Haber begins with the Greek philosophers and moves through the Enlightenment and up to today. Of note for educators is the fact that even in ancient Greece, the skill of critical thinking was not designated as a specialty of philosophers but as a core component of the curriculum of the time. The common thread tying together the various people and movements Haber highlights is their focus on eschewing adherence to traditional understandings or accepted explanations for physical, social, and psychological phenomena and, instead, their willingness to examine and question underlying assumptions that support contemporary dogma. In many ways, this history lesson provides the clearest picture of what Haber means by “critical thinking” throughout the book: a willingness to continually reexamine one’s own beliefs and to engage in that same generous skepticism with others.

The second chapter is perhaps the most technical in the book, as it focuses primarily on introducing the building blocks of formal logic. Although not written directly as a resource for lesson planning, this chapter outlines a number of the concepts that are easily adaptable to units focused on building skills for critical thinking. Haber shows how diagramming an argument in terms of premises and conclusions can help students of all ages understand its structure. Similarly, drawing the distinction between the validity of an argument (Does the form of the argument align with valid logical principles?) and its soundness (Do the premises correspond to reality? or How likely are they to be disproven?) can provide insight into where an argument is weak or how to anticipate attacks against an argument. Beyond this crash course in introductory logic, Haber also highlights other skills that are important for critical thinking, namely language skills, information literacy, and creativity, as well as dispositions that are essential for critical thinkers, such as intellectual humility, intellectual courage, and fair-mindedness.

In the third and final substantive chapter, Haber outlines current approaches to teaching and measuring critical thinking in the US education system. Here Haber notes that the modern movement around explicitly teaching critical thinking can be traced to a 1983 requirement that all students graduating from California state colleges and universities complete a course on critical thinking. Following this, policy makers and curriculum developers have striven to include critical thinking in many of the standards that have been put forth, most notably the Common Core State Standards. In all, Haber highlights how the messaging and expectations around critical thinking instruction have been growing at almost every level of education and argues that this is a welcome trend. At the same time, he points out how the actual implementation of instruction in this area has been uneven and often lags behind the aspirational rhetoric of educational leaders. He makes a convincing case that the process of becoming a critical thinker and honing critical thinking skills is one that should be lifelong and that, in many ways, the best instruction for developing critical thinkers is consistent practice in applying those skills, both as explicit portions of curricula and implicitly throughout other content areas.

In Critical Thinking , Haber illustrates a portrait of a critical thinker that is rather comprehensive. In his telling, critical thinking is not a skill to be applied to specific situations or circumstances but a stance that one has toward almost all subjects and encounters. In many ways, this portrait is compelling. Haber argues that if students and adults approach political, social, and scientific questions with the tools, skills, and virtues outlined in the book, it is likely that US society will focus on how to deal with difficult problems like climate change or systemic racism rather than debating whether these problems exist. He also grapples with alternatives to the paradigm of critical thinking, most notably proponents of group or team thinking who assert that thinking must be conceived of as a social act, compared to the more individualistic endeavor of critical thinking. Haber further points out that there is room for emotion in the critical thinking framework, noting that “by balancing our emotional, intuitive, and reasoning selves, we avoid cutting ourselves off from valuable data required to apply reasoning effectively in a world made up of people rather than machines” (p. 147).

Yet, the book would have benefited from a slightly more nuanced look into the limitations of critical thinking as Haber describes it. Specifically, what questions or types of questions will this approach find difficult or impossible to resolve? While Haber makes a convincing case that applying critical thinking principles to scientific questions should lead us to continually refine our understanding of the world, it is not clear how these tools can be applied to more nebulous political or moral questions where there may be no objective truth. Certainly, thinking critically about one’s own or an opponent’s political and moral arguments can lead to greater understanding, but it is not clear that such understanding will lead to agreement or resolution.

While the rules for building arguments are clearly laid out in this book, the criteria for selecting their foundations are less obvious. While not everyone will work from the same set of basic premises or values, these starting points are critical for understanding the conclusions that people reach. Nor do the tenets of formal logic offer a clear way to engage with these most fundamental beliefs. Critical thinking can help two people analyze each other’s points of view, but it does not help adjudicate between different starting premises, such as the existence of God or even the appropriate role of government in people’s lives. While it is not necessary for Haber to resolve these tensions, an exploration into or an acknowledgment of these limitations would have been helpful for readers of all levels of expertise as they consider ways to incorporate critical thinking skills into their instruction, policy, or daily lives.

Critical Thinking does succeed in providing a thorough yet high-level introduction to historical and modern thought on the topic. Given the current diversity of curricula, standards, and mandates that feature critical thinking as a central component or outcome, it is helpful to have a resource that not only engages with but synthesizes these sources into a coherent whole. Haber welcomes this diversity as a strength and notes that while each of these sources has a slightly different take on critical thinking, there is wide overlap and consensus on many of the core points. He explores the tension between those who say that critical thinking should be taught explicitly as an independent discipline and those who advocate for incorporating critical thinking tasks into other content areas like science, reading, and writing, ultimately concluding that both approaches are necessary and beneficial. Similarly, although he does review and speak positively about a number of assessments that purport to measure critical thinking skills, he reminds the reader that a single assessment can only capture a limited view of a student’s critical thinking skills. If practitioners and policy makers are serious about making critical thinking an educational priority, approaches to its assessment and instruction need to be dealt with comprehensively rather than simply as another standard to be tacked on to an existing framework.

Recipient(s) will receive an email with a link to 'Critical Thinking' and will not need an account to access the content.

Subject: Critical Thinking

(Optional message may have a maximum of 1000 characters.)

Citing articles via

Get email alerts.

• ©2024 President and Fellows of Harvard College. All Rights Reserved
• Harvard Education Publishing Group
• 8 Story Street, Cambridge, Massachusetts 02138 | (617) 495-3432
• Editorial Board
• Technical Support
• Trademark Notice
• Digital Accessibility Policy

Affiliations

• eISSN 1943-5045
• ISSN 0017-8055
• Privacy Policy
• Get Adobe Acrobat Reader

This Feature Is Available To Subscribers Only

Sign In or Create an Account

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Fostering Critical Thinking, Reasoning, and Argumentation Skills through Bioethics Education

* E-mail: [email protected]

Affiliation Northwest Association for Biomedical Research, Seattle, Washington, United States of America

Affiliation Center for Research and Learning, Snohomish, Washington, United States of America

• Jeanne Ting Chowning, 
• Joan Carlton Griswold, 
• Dina N. Kovarik, 
• Laura J. Collins

• Published: May 11, 2012
• https://doi.org/10.1371/journal.pone.0036791
• Reader Comments

Developing a position on a socio-scientific issue and defending it using a well-reasoned justification involves complex cognitive skills that are challenging to both teach and assess. Our work centers on instructional strategies for fostering critical thinking skills in high school students using bioethical case studies, decision-making frameworks, and structured analysis tools to scaffold student argumentation. In this study, we examined the effects of our teacher professional development and curricular materials on the ability of high school students to analyze a bioethical case study and develop a strong position. We focused on student ability to identify an ethical question, consider stakeholders and their values, incorporate relevant scientific facts and content, address ethical principles, and consider the strengths and weaknesses of alternate solutions. 431 students and 12 teachers participated in a research study using teacher cohorts for comparison purposes. The first cohort received professional development and used the curriculum with their students; the second did not receive professional development until after their participation in the study and did not use the curriculum. In order to assess the acquisition of higher-order justification skills, students were asked to analyze a case study and develop a well-reasoned written position. We evaluated statements using a scoring rubric and found highly significant differences (p<0.001) between students exposed to the curriculum strategies and those who were not. Students also showed highly significant gains (p<0.001) in self-reported interest in science content, ability to analyze socio-scientific issues, awareness of ethical issues, ability to listen to and discuss viewpoints different from their own, and understanding of the relationship between science and society. Our results demonstrate that incorporating ethical dilemmas into the classroom is one strategy for increasing student motivation and engagement with science content, while promoting reasoning and justification skills that help prepare an informed citizenry.

Citation: Chowning JT, Griswold JC, Kovarik DN, Collins LJ (2012) Fostering Critical Thinking, Reasoning, and Argumentation Skills through Bioethics Education. PLoS ONE 7(5): e36791. https://doi.org/10.1371/journal.pone.0036791

Editor: Julio Francisco Turrens, University of South Alabama, United States of America

Received: February 7, 2012; Accepted: April 13, 2012; Published: May 11, 2012

Copyright: © 2012 Chowning et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The “Collaborations to Understand Research and Ethics” (CURE) program was supported by a Science Education Partnership Award grant ( http://ncrrsepa.org ) from the National Center for Research Resources and the Division of Program Coordination, Planning, and Strategic Initiatives of the National Institutes of Health through Grant Number R25OD011138. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

While the practice of argumentation is a cornerstone of the scientific process, students at the secondary level have few opportunities to engage in it [1] . Recent research suggests that collaborative discourse and critical dialogue focused on student claims and justifications can increase student reasoning abilities and conceptual understanding, and that strategies are needed to promote such practices in secondary science classrooms [2] . In particular, students need structured opportunities to develop arguments and discuss them with their peers. In scientific argument, the data, claims and warrants (that relate claims to data) are strictly concerned with scientific data; in a socio-scientific argument, students must consider stakeholder perspectives and ethical principles and ideas, in addition to relevant scientific background. Regardless of whether the arguments that students employ point towards scientific or socio-scientific issues, the overall processes students use in order to develop justifications rely on a model that conceptualizes arguments as claims to knowledge [3] .

Prior research in informal student reasoning and socio-scientific issues also indicates that most learners are not able to formulate high-quality arguments (as defined by the ability to articulate justifications for claims and to rebut contrary positions), and highlights the challenges related to promoting argumentation skills. Research suggests that students need experience and practice justifying their claims, recognizing and addressing counter-arguments, and learning about elements that contribute to a strong justification [4] , [5] .

Proponents of Socio-scientific Issues (SSI) education stress that the intellectual development of students in ethical reasoning is necessary to promote understanding of the relationship between science and society [4] , [6] . The SSI approach emphasizes three important principles: (a) because science literacy should be a goal for all students, science education should be broad-based and geared beyond imparting relevant content knowledge to future scientists; (b) science learning should involve students in thinking about the kinds of real-world experiences that they might encounter in their lives; and (c) when teaching about real-world issues, science teachers should aim to include contextual elements that are beyond traditional science content. Sadler and Zeidler, who advocate a SSI perspective, note that “people do not live their lives according to disciplinary boundaries, and students approach socio-scientific issues with diverse perspectives that integrate science and other considerations” [7] .

Standards for science literacy emphasize not only the importance of scientific content and processes, but also the need for students to learn about science that is contextualized in real-world situations that involve personal and community decision-making [7] – [10] . The National Board for Professional Teaching Standards stresses that students need “regular exposure to the human contexts of science [and] examples of ethical dilemmas, both current and past, that surround particular scientific activities, discoveries, and technologies” [11] . Teachers are mandated by national science standards and professional teaching standards to address the social dimensions of science, and are encouraged to provide students with the tools necessary to engage in analyzing bioethical issues; yet they rarely receive training in methods to foster such discussions with students.

The Northwest Association for Biomedical Research (NWABR), a non-profit organization that advances the understanding and support of biomedical research, has been engaging students and teachers in bringing the discussion of ethical issues in science into the classroom since 2000 [12] . The mission of NWABR is to promote an understanding of biomedical research and its ethical conduct through dialogue and education. The sixty research institutions that constitute our members include academia, industry, non-profit research organizations, research hospitals, professional societies, and volunteer health organizations. NWABR connects the scientific and education communities across the Northwestern United States and helps the public understand the vital role of research in promoting better health outcomes. We have focused on providing teachers with both resources to foster student reasoning skills (such as activities in which students practice evaluating arguments using criteria for strong justifications), as well as pedagogical strategies for fostering collaborative discussion [13] – [15] . Our work draws upon socio-scientific elements of functional scientific literacy identified by Zeidler et al. [6] . We include support for teachers in discourse issues, nature of science issues, case-based issues, and cultural issues – which all contribute to cognitive and moral development and promote functional scientific literacy. Our Collaborations to Understand Research and Ethics (CURE) program, funded by a Science Education Partnership Award from the National Institutes of Health (NIH), promotes understanding of translational biomedical research as well as the ethical considerations such research raises.

Many teachers find a principles-based approach most manageable for introducing ethical considerations. The principles include respect for persons (respecting the inherent worth of an individual and his or her autonomy), beneficence/nonmaleficence (maximizing benefits/minimizing harms), and justice (distributing benefits/burdens equitably across a group of individuals). These principles, which are articulated in the Belmont Report [16] in relation to research with human participants (and which are clarified and defended by Beauchamp and Childress [17] ), represent familiar concepts and are widely used. In our professional development workshops and in our support resources, we also introduce teachers to care, feminist, virtue, deontological and consequentialist ethics. Once teachers become familiar with principles, they often augment their teaching by incorporating these additional ethical approaches.

The Bioethics 101 materials that were the focus of our study were developed in conjunction with teachers, ethicists, and scientists. The curriculum contains a series of five classroom lessons and a culminating assessment [18] and is described in more detail in the Program Description below. For many years, teachers have shared with us the dramatic impacts that the teaching of bioethics can have on their students; this research study was designed to investigate the relationship between explicit instruction in bioethical reasoning and resulting student outcomes. In this study, teacher cohorts and student pre/post tests were used to investigate whether CURE professional development and the Bioethics 101 curriculum materials made a significant difference in high school students’ abilities to analyze a case study and justify their positions. Our research strongly indicates that such reasoning approaches can be taught to high school students and can significantly improve their ability to develop well-reasoned justifications to bioethical dilemmas. In addition, student self-reports provide additional evidence of the extent to which bioethics instruction impacted their attitudes and perceptions and increased student motivation and engagement with science content.

Program Description

Our professional development program, Ethics in the Science Classroom, spanned two weeks. The first week, a residential program at the University of Washington (UW) Pack Forest Conference Center, focused on our Bioethics 101 curriculum, which is summarized in Table S1 and is freely available at http://www.nwabr.org . The curriculum, a series of five classroom lessons and a culminating assessment, was implemented by all teachers who were part of our CURE treatment group. The lessons explore the following topics: (a) characteristics of an ethical question; (b) bioethical principles; (c) the relationship between science and ethics and the roles of objectivity/subjectivity and evidence in each; (d) analysis of a case study (including identifying an ethical question, determining relevant facts, identifying stakeholders and their concerns and values, and evaluating options); and (e) development of a well-reasoned justification for a position.

Additionally, the first week focused on effective teaching methods for incorporating ethical issues into science classrooms. We shared specific pedagogical strategies for helping teachers manage classroom discussion, such as asking students to consider the concerns and values of individuals involved in the case while in small single and mixed stakeholder groups. We also provided participants with background knowledge in biomedical research and ethics. Presentations from colleagues affiliated with the NIH Clinical and Translational Science Award program, from the Department of Bioethics and Humanities at the UW, and from NWABR member institutions helped participants develop a broad appreciation for the process of biomedical research and the ethical issues that arise as a consequence of that research. Topics included clinical trials, animal models of disease, regulation of research, and ethical foundations of research. Participants also developed materials directly relevant and applicable to their own classrooms, and shared them with other educators. Teachers wrote case studies and then used ethical frameworks to analyze the main arguments surrounding the case, thereby gaining experience in bioethical analysis. Teachers also developed Action Plans to outline their plans for implementation.

The second week provided teachers with first-hand experiences in NWABR research institutions. Teachers visited research centers such as the Tumor Vaccine Group and Clinical Research Center at the UW. They also had the opportunity to visit several of the following institutions: Amgen, Benaroya Research Institute, Fred Hutchinson Cancer Research Center, Infectious Disease Research Institute, Institute for Stem Cells and Regenerative Medicine at the UW, Pacific Northwest Diabetes Research Institute, Puget Sound Blood Center, HIV Vaccine Trials Network, and Washington National Primate Research Center. Teachers found these experiences in research facilities extremely valuable in helping make concrete the concepts and processes detailed in the first week of the program.

We held two follow-up sessions during the school year to deepen our relationship with the teachers, promote a vibrant ethics in science education community, provide additional resources and support, and reflect on challenges in implementation of our materials. We also provided the opportunity for teachers to share their experiences with one another and to report on the most meaningful longer-term impacts from the program. Another feature of our CURE program was the school-year Institutional Review Board (IRB) and Institutional Animal Care and Use Committee (IACUC) follow-up sessions. Teachers chose to attend one of NWABR’s IRB or IACUC conferences, attend a meeting of a review board, or complete NIH online ethics training. Some teachers also visited the UW Embryonic Stem Cell Research Oversight Committee. CURE funding provided substitutes in order for teachers to be released during the workday. These opportunities further engaged teachers in understanding and appreciating the actual process of oversight for federally funded research.

Participants

Most of the educators who have been through our intensive summer workshops teach secondary level science, but we have welcomed teachers at the college, community college, and even elementary levels. Our participants are primarily biology teachers; however, chemistry and physical science educators, health and career specialists, and social studies teachers have also used our strategies and materials with success.

The research design used teacher cohorts for comparison purposes and recruited teachers who expressed interest in participating in a CURE workshop in either the summer of 2009 or the summer of 2010. We assumed that all teachers who applied to the CURE workshop for either year would be similarly interested in ethics topics. Thus, Cohort 1 included teachers participating in CURE during the summer of 2009 (the treatment group). Their students received CURE instruction during the following 2009–2010 academic year. Cohort 2 (the comparison group) included teachers who were selected to participate in CURE during the summer of 2010. Their students received a semester of traditional classroom instruction in science during the 2009–2010 academic year. In order to track participation of different demographic groups, questions pertaining to race, ethnicity, and gender were also included in the post-tests.

Using an online sample size calculator http://www.surveysystem.com/sscalc.htm , a 95% Confidence Level, and a Confidence Interval of 5, it was calculated that a sample size of 278 students would be needed for the research study. For that reason, six Cohort 1 teachers were impartially chosen to be in the study. For the comparison group, the study design also required six teachers from Cohort 2. The external evaluator contacted all Cohort 2 teachers to explain the research study and obtain their consent, and successfully recruited six to participate.

Ethics Statement

This study was conducted according to the principles expressed in the Declaration of Helsinki. Prior to the study, research processes and materials were reviewed and approved by the Western Institutional Review Board (WIRB Study #1103180). CURE staff and evaluators received written permission from parents to have their minor children participate in the Bioethics 101 curriculum, for the collection and subsequent analysis of students’ written responses to the assessment, and for permission to collect and analyze student interview responses. Teachers also provided written informed consent prior to study participation. All study participants and/or their legal guardians provided written informed consent for the collection and subsequent analysis of verbal and written responses.

Research Study

Analyzing a case study: cure and comparison students..

Teacher cohorts and pre/post tests were used to investigate whether CURE professional development and curriculum materials made a significant difference in high school students’ abilities to analyze a case study and justify their positions. Cohort 1 teachers (N = 6) received CURE professional development and used the Bioethics 101 curriculum with their students (N = 323); Cohort 2 teachers (N = 6) did not receive professional development until after their participation in the study and did not use the curriculum with their students (N = 108). Cohort 2 students were given the test case study and questions, but with only traditional science instruction during the semester. Each Cohort was further divided into two groups (A and B). Students in Group A were asked to complete a pre-test prior to the case study, while students in Group B did not. All four student groups completed a post-test after analysis of the case study. This four-group model ( Table 1 ) allowed us to assess: 1) the effect of CURE treatment relative to conventional education practices, 2) the effect of the pre-test relative to no pre-test, and 3) the interaction between the pre-test and CURE treatment condition. Random assignment of students to treatment and comparison groups was not possible; consequently we used existing intact classes. In all, 431 students and 12 teachers participated in the research study ( Table 2 ).

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0036791.t001

https://doi.org/10.1371/journal.pone.0036791.t002

In order to assess the acquisition of higher-order justification skills, students used the summative assessment provided in our curriculum as the pre- and post-test. We designed the curriculum to scaffold students’ ability to write a persuasive bioethical position; by the time they participated in the assessment, Cohort 1 students had opportunities to discuss the elements of a strong justification as well as practice in analyzing case studies. For our research, both Cohort 1 and 2 students were asked to analyze the case study of “Ashley X” ( Table S2 ), a young girl with a severe neurological impairment whose parents wished to limit her growth through a combination of interventions so that they could better care for her. Students were asked to respond to the ethical question: “Should one or more medical interventions be used to limit Ashley’s growth and physical maturation? If so, which interventions should be used and why?” In their answer, students were encouraged to develop a well-reasoned written position by responding to five questions that reflected elements of a strong justification. One difficulty in evaluating a multifaceted science-related learning task (analyzing a bioethical case study and justifying a position) is that a traditional multiple-choice assessment may not adequately reflect the subtlety and depth of student understanding. We used a rubric to assess student responses to each of the following questions (Q) on a scale of 1 to 4; these questions represent key elements of a strong justification for a bioethical argument:

• Q1: Student Position: What is your decision?
• Q2: Factual Support: What facts support your decision? Is there missing information that could be used to make a better decision?
• Q3: Interests and Views of Others: Who will be impacted by the decision and how will they be impacted?
• Q4: Ethical Considerations: What are the main ethical considerations?
• Q5: Evaluating Alternative Options: What are some strengths and weaknesses of alternate solutions?

In keeping with our focus on the process of reasoning rather than on having students draw any particular conclusion, we did not assess students on which position they took, but on how well they stated and justified the position they chose.

We used a rubric scoring guide to assess student learning, which aligned with the complex cognitive challenges posed by the task ( Table S3 ). Assessing complex aspects of student learning is often difficult, especially evaluating how students represent their knowledge and competence in the domain of bioethical reasoning. Using a scoring rubric helped us more authentically score dimensions of students’ learning and their depth of thinking. An outside scorer who had previously participated in CURE workshops, has secondary science teaching experience, and who has a Masters degree in Bioethics blindly scored all student pre- and post-tests. Development of the rubric was an iterative process, refined after analyzing a subset of surveys. Once finalized, we confirmed the consistency and reliability of the rubric and grading process by re-testing a subset of student surveys randomly selected from all participating classes. The Cronbach alpha reliability result was 0.80 [19] .

The rubric closely followed the framework introduced through the curricular materials and reinforced through other case study analyses. For example, under Q2, Factual Support , a student rated 4 out of 4 if their response demonstrated the following:

• The justification uses the relevant scientific reasons to support student’s answer to the ethical question.
• The student demonstrates a solid understanding of the context in which the case occurs, including a thoughtful description of important missing information.
• The student shows logical, organized thinking. Both facts supporting the decision and missing information are presented at levels exceeding standard (as described above).

An example of a student response that received the highest rating for Q2 asking for factual support is: “Her family has a history of breast cancer and fibrocystic breast disease. She is bed-bound and completely dependent on her parents. Since she is bed-bound, she has a higher risk of blood clots. She has the mentality of an infant. Her parents’ requests offer minimal side effects. With this disease, how long is she expected to live? If not very long then her parents don’t have to worry about growth. Are there alternative measures?”

In contrast, a student rated a 1 for responses that had the following characteristics:

• Factual information relevant to the case is incompletely described or is missing.
• Irrelevant information may be included and the student demonstrates some confusion.

An example of a student response that rated a 1 for Q2 is: “She is unconscious and doesn’t care what happens.”

All data were entered into SPSS (Statistical Package for the Social Sciences) and analyzed for means, standard deviations, and statistically significant differences. An Analysis of Variance (ANOVA) was used to test for significant overall differences between the two cohort groups. Pre-test and post-test composite scores were calculated for each student by adding individual scores for each item on the pre- and post-tests. The composite score on the post-test was identical in form and scoring to the composite score on the pre-test. The effect of the CURE treatment on post-test composite scores is referred to as the Main Effect, and was determined by comparing the post-test composite scores of the Cohort 1 (CURE) and Cohort 2 (Comparison) groups. In addition, Cohort 1 and Cohort 2 means scores for each test question (Questions 1–5) were compared within and between cohorts using t-tests.

CURE student perceptions of curriculum effect.

During prior program evaluations, we asked teachers to identify what they believed to be the main impacts of bioethics instruction on students. From this earlier work, we identified several themes. These themes, listed below, were further tested in our current study by asking students in the treatment group to assess themselves in these five areas after participation in the lesson, using a retrospective pre-test design to measure self-reported changes in perceptions and abilities [20] .

• Interest in the science content of class (before/after) participating in the Ethics unit.
• Ability to analyze issues related to science and society and make well-justified decisions (before/after) participating in the Ethics unit.
• Awareness of ethics and ethical issues (before/after) participating in the Ethics unit.
• Understanding of the connection between science and society (before/after) participating in the Ethics unit.
• Ability to listen to and discuss different viewpoints (before/after) participating in the Ethics unit.

After Cohort 1 (CURE) students participated in the Bioethics 101 curriculum, we asked them to indicate the extent to which they had changed in each of the theme areas we had identified using Likert-scale items on a retrospective pre-test design [21] , with 1 =  None and 5 =  A lot!. We used paired t-tests to examine self-reported changes in their perceptions and abilities. The retrospective design avoids response-shift bias that results from overestimation or underestimation of change since both before and after information is collected at the same time [20] .

Student Demographics

Demographic information is provided in Table 3 . Of those students who reported their gender, a larger number were female (N = 258) than male (N = 169), 60% and 40%, respectively, though female students represented a larger proportion of Cohort 1 than Cohort 2. Students ranged in age from 14 to 18 years old; the average age of the students in both cohorts was 15. Students were enrolled in a variety of science classes (mostly Biology or Honors Biology). Because NIH recognizes a difference between race and ethnicity, students were asked to respond to both demographic questions. Students in both cohorts were from a variety of ethnic and racial backgrounds.

https://doi.org/10.1371/journal.pone.0036791.t003

Pre- and Post-Test Results for CURE and Comparison Students

Post-test composite means for each cohort (1 and 2) and group (A and B) are shown in Table 4 . Students receiving CURE instruction earned significantly higher (p<0.001) composite mean scores than students in comparison classrooms. Cohort 1 (CURE) students (N = 323) post-test composite means were 10.73, while Cohort 2 (Comparison) students (N = 108) had post-test composite means of 9.16. The ANOVA results ( Table 5 ) showed significant differences in the ability to craft strong justifications between Cohort 1 (CURE) and Cohort 2 (Comparison) students F (1, 429) = 26.64, p<0.001.

https://doi.org/10.1371/journal.pone.0036791.t004

https://doi.org/10.1371/journal.pone.0036791.t005

We also examined if the pre-test had a priming effect on the students’ scores because it provides an opportunity to practice or think about the content. The pre-test would not have this effect on the comparison group because they were not exposed to CURE teaching or materials. If the pre-test provides a practice or priming effect, this would result in higher post-test performance by CURE students receiving the pre-test than by CURE students not receiving the pre-test. For this comparison, the F (1, 321) = 0.10, p = 0.92. This result suggests that the differences between the CURE and comparison groups are attributable to the treatment condition and not a priming effect of the pre-test.

After differences in main effects were investigated, we analyzed differences between and within cohorts on individual items (Questions 1–5) using t-tests. The Mean scores of individual questions for each cohort are shown in Figure 1 . There were no significant differences between Cohort 1 (CURE) and Cohort 2 (Comparison) on pre-test scores. In fact, for Q5, the mean pre-test scores for the Cohort 2 (Comparison) group were slightly higher (1.8) than the Cohort 1 (CURE) group (1.6). On the post-test, the Cohort 1 (CURE) students significantly outscored the Cohort 2 (Comparison) students on all questions; Q1, Q3, and Q4 were significant at p<0.001, Q2 was significant at p<0.01, and Q5 was significant at p<0.05. The largest post-test difference between Cohort 1 (CURE) students and Cohort 2 (Comparison) students was for Q3, with an increase of 0.6; all the other questions showed changes of 0.3 or less. Comparing Cohort 1 (CURE) post-test performance on individual questions yields the following results: scores were highest for Q1 (mean = 2.8), followed by Q3 (mean = 2.2), Q2 (mean = 2.1), and Q5 (mean = 1.9). Lowest Cohort 1 (CURE) post-test scores were associated with Q4 (mean = 1.8).

Mean scores for individual items of the pre-test for each cohort revealed no differences between groups for any of the items (Cohort 1, CURE, N = 323; Cohort 2, Comparison, N = 108). Post-test gains of Cohort 1 (CURE) relative to Cohort 2 (Comparison) were statistically significant for all questions. (Question (Q) 1) What is your decision? (Q2) What facts support your decision? Is there missing information that could be used to make a better decision? (Q3) Who will be impacted by the decision and how will they be impacted? (Q4) What are the main ethical considerations? and (Q5)What are some strengths and weaknesses of alternate solutions? Specifically: (Q1), (Q3), (Q4) were significant at p<0.001 (***); (Q2) was significant at p<0.01 (**); and (Q5) was significant at p<0.05 (*). Lines represent standard deviations.

https://doi.org/10.1371/journal.pone.0036791.g001

Overall, across all four groups, mean scores for Q1 were highest (2.6), while scores for Q4 were lowest (1.6). When comparing within-Cohort scores on the pre-test versus post-test, Cohort 2 (Comparison Group) showed little to no change, while CURE students improved on all test questions.

CURE Student Perceptions of Curriculum Effect

After using our resources, Cohort 1 (CURE) students showed highly significant gains (p<0.001) in all areas examined: interest in science content, ability to analyze socio-scientific issues and make well-justified decisions, awareness of ethical issues, understanding of the connection between science and society, and the ability to listen to and discuss viewpoints different from their own ( Figure 2 ). Overall, students gave the highest score to their ability to listen to and discuss viewpoints different than their own after participating in the CURE unit (mean = 4.2). Also highly rated were the changes in understanding of the connection between science and society (mean = 4.1) and the awareness of ethical issues (mean = 4.1); these two perceptions also showed the largest change pre-post (from 2.8 to 4.1 and 2.7 to 4.1, respectively).

Mean scores for individual items of the retrospective items on the post-test for Cohort 1 students revealed significant gains (p<0.001) in all self-reported items: Interest in science (N = 308), ability to Analyze issues related to science and society and make well-justified decisions (N = 306), Awareness of ethics and ethical issues (N = 309), Understanding of the connection between science and society (N = 308), and the ability to Listen and discuss different viewpoints (N = 308). Lines represent standard deviations.

https://doi.org/10.1371/journal.pone.0036791.g002

NWABR’s teaching materials provide support both for general ethics and bioethics education, as well as for specific topics such as embryonic stem cell research. These resources were developed to provide teachers with classroom strategies, ethics background, and decision-making frameworks. Teachers are then prepared to share their understanding with their students, and to support their students in using analysis tools and participating in effective classroom discussions. Our current research grew out of a desire to measure the effectiveness of our professional development and teaching resources in fostering student ability to analyze a complex bioethical case study and to justify their positions.

Consistent with the findings of SSI researchers and our own prior anecdotal observations of teacher classrooms and student work, we found that students improve in their analytical skill when provided with reasoning frameworks and background in concepts such as beneficence, respect, and justice. Our research demonstrates that structured reasoning approaches can be effectively taught at the secondary level and that they can improve student thinking skills. After teachers participated in a two-week professional development workshop and utilized our Bioethics 101 curriculum, within a relatively short time period (five lessons spanning approximately one to two weeks), students grew significantly in their ability to analyze a complex case and justify their position compared to students not exposed to the program. Often, biology texts present a controversial issue and ask students to “justify their position,” but teachers have shared with us that students frequently do not understand what makes a position or argument well-justified. By providing students with opportunities to evaluate sample justifications, and by explicitly introducing a set of elements that students should include in their justifications, we have facilitated the development of this important cognitive skill.

The first part of our research examined the impact of CURE instruction on students’ ability to analyze a case study. Although students grew significantly in all areas, the highest scores for the Cohort 1 (CURE) students were found in response to Q1 of the case analysis, which asked them to clearly state their own position, and represented a relatively easy cognitive task. This question also received the highest score in the comparison group. Not surprisingly, students struggled most with Q4 and Q5, which asked for the ethical considerations and the strengths and weaknesses of different solutions, respectively, and which tested specialized knowledge and sophisticated analytical skills. The area in which we saw the most growth in Cohort 1 (CURE) (both in comparison to the pre-test and in relation to the comparison group) was in students’ ability to identify stakeholders in a case and state how they might be impacted by a decision (Q3). Teachers have shared with us that secondary students are often focused on their own needs and perspectives; stepping into the perspectives of others helps enlarge their understanding of the many views that can be brought to bear upon a socio-scientific issue.

Many of our teachers go far beyond these introductory lessons, revisiting key concepts throughout the year as new topics are presented in the media or as new curricular connections arise. Although we have observed this phenomenon for many years, it has been difficult to evaluate these types of interventions, as so many teachers implement the concepts and ideas differently in response to their unique needs. Some teachers have used the Bioethics 101 curriculum as a means for setting the tone and norms for the entire year in their classes and fostering an atmosphere of respectful discussion. These teachers note that the “opportunity cost” of investing time in teaching basic bioethical concepts, decision-making strategies, and justification frameworks pays off over the long run. Students’ understanding of many different science topics is enhanced by their ability to analyze issues related to science and society and make well-justified decisions. Throughout their courses, teachers are able to refer back to the core ideas introduced in Bioethics 101, reinforcing the wide utility of the curriculum.

The second part of our research focused on changes in students’ self-reported attitudes and perceptions as a result of CURE instruction. Obtaining accurate and meaningful data to assess student self-reported perceptions can be difficult, especially when a program is distributed across multiple schools. The traditional use of the pretest-posttest design assumes that students are using the same internal standard to judge attitudes or perceptions. Considerable empirical evidence suggests that program effects based on pre-posttest self-reports are masked because people either overestimate or underestimate their pre-program perceptions [20] , [22] – [26] . Moore and Tananis [27] report that response shift can occur in educational programs, especially when they are designed to increase students’ awareness of a specific construct that is being measured. The retrospective pre-test design (RPT), which was used in this study, has gained increasing prominence as a convenient and valid method for measuring self-reported change. RPT has been shown to reduce response shift bias, providing more accurate assessment of actual effect. The retrospective design avoids response-shift bias that results from overestimation or underestimation of change since both before and after information is collected at the same time [20] . It is also convenient to implement, provides comparison data, and may be more appropriate in some situations [26] . Using student self-reported measures concerning perceptions and attitudes is also a meta-cognitive strategy that allows students to think about their learning and justify where they believe they are at the end of a project or curriculum compared to where they were at the beginning.

Our approach resulted in a significant increase in students’ own perceived growth in several areas related to awareness, understanding, and interest in science. Our finding that student interest in science can be significantly increased through a case-study based bioethics curriculum has implications for instruction. Incorporating ethical dilemmas into the classroom is one strategy for increasing student motivation and engagement with science content. Students noted the greatest changes in their own awareness of ethical issues and in understanding the connection between science and society. Students gave the highest overall rating to their ability to listen to and discuss viewpoints different from their own after participation in the bioethics unit. This finding also has implications for our future citizenry; in an increasingly diverse and globalized society, students need to be able to engage in civil and rational dialogue with others who may not share their views.

Conducting research studies about ethical learning in secondary schools is challenging; recruiting teachers for Cohort 2 and obtaining consent from students, parents, and teachers for participation was particularly difficult, and many teachers faced restraints from district regulations about curriculum content. Additional studies are needed to clarify the extent to which our curricular materials alone, without accompanying teacher professional development, can improve student reasoning skills.

Teacher pre-service training programs rarely incorporate discussion of how to address ethical issues in science with prospective educators. Likewise, with some noticeable exceptions, such as the work of the University of Pennsylvania High School Bioethics Project, the Genetic Science Learning Center at the University of Utah, and the Kennedy Institute of Ethics at Georgetown University, relatively few resources exist for high school curricular materials in this area. Teachers have shared with us that they know that such issues are important and engaging for students, but they do not have the experience in either ethical theory or in managing classroom discussion to feel comfortable teaching bioethics topics. After participating in our workshops or using our teaching materials, teachers shared that they are better prepared to address such issues with their students, and that students are more engaged in science topics and are better able to see the real-world context of what they are learning.

Preparing students for a future in which they have access to personalized genetic information, or need to vote on proposals for stem cell research funding, necessitates providing them with the tools required to reason through a complex decision containing both scientific and ethical components. Students begin to realize that, although there may not be an absolute “right” or “wrong” decision to be made on an ethical issue, neither is ethics purely relative (“my opinion versus yours”). They come to realize that all arguments are not equal; there are stronger and weaker justifications for positions. Strong justifications are built upon accurate scientific information and solid analysis of ethical and contextual considerations. An informed citizenry that can engage in reasoned dialogue about the role science should play in society is critical to ensure the continued vitality of the scientific enterprise.

“I now bring up ethical issues regularly with my students, and use them to help students see how the concepts they are learning apply to their lives…I am seeing positive results from my students, who are more clearly able to see how abstract science concepts apply to them.” – CURE Teacher “In ethics, I’ve learned to start thinking about the bigger picture. Before, I based my decisions on how they would affect me. Also, I made decisions depending on my personal opinions, sometimes ignoring the facts and just going with what I thought was best. Now, I know that to make an important choice, you have to consider the other people involved, not just yourself, and take all information and facts into account.” – CURE Student

Supporting Information

Bioethics 101 Lesson Overview.

https://doi.org/10.1371/journal.pone.0036791.s001

Case Study for Assessment.

https://doi.org/10.1371/journal.pone.0036791.s002

Grading Rubric for Pre- and Post-Test: Ashley’s Case.

https://doi.org/10.1371/journal.pone.0036791.s003

Acknowledgments

We thank Susan Adler, Jennifer M. Pang, Ph.D., Leena Pranikay, and Reitha Weeks, Ph.D., for their review of the manuscript, and Nichole Beddes for her assistance scoring student work. We also thank Carolyn Cohen of Cohen Research and Evaluation, former CURE Evaluation Consultant, who laid some of the groundwork for this study through her prior work with us. We also wish to thank the reviewers of our manuscript for their thoughtful feedback and suggestions.

Author Contributions

Conceived and designed the experiments: JTC LJC. Performed the experiments: LJC. Analyzed the data: LJC JTC DNK. Contributed reagents/materials/analysis tools: JCG. Wrote the paper: JTC LJC DNK JCG. Served as Principal Investigator on the CURE project: JTC. Provided overall program leadership: JTC. Led the curriculum and professional development efforts: JTC JCG. Raised funds for the CURE program: JTC.

• 1. Bell P (2004) Promoting students’ argument construction and collaborative debate in the science classroom. Mahwah, NJ: Erlbaum.
• View Article
• Google Scholar
• 3. Toulmin S (1958) The Uses of Argument. Cambridge: Cambridge University Press.
• 6. Zeidler DL, Sadler TD, Simmons ML, Howes , EV (2005) Beyond STS: A research-based framework for socioscientific issues education. Wiley InterScience. pp. 357–377.
• 8. AAAS (1990) Science for All Americans. New York: Oxford University Press.
• 9. National Research Council (1996) National Science Education Standards. Washington, DC: National Academies Press.
• 10. National Research Council (2011) A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. Washington, DC: National Academies Press.
• 11. National Board for Professional Teaching Standards (2007) Adolescence and Young Adulthood Science Standards. Arlington, VA.
• 17. Beauchamp T, Childress JF (2001) Principles of biomedical ethics. New York: Oxford University Press.
• 18. Chowning JT, Griswold JC (2010) Bioethics 101. Seattle, WA: NWABR.
• 26. Klatt J, Taylor-Powell E (2005) Synthesis of literature relative to the retrospective pretest design. Presentation to the 2005 Joint CES/AEA Conference, Toronto.

Exploring Students' Critical Thinking Skills Using the Engineering Design Process in a Physics Classroom

• Regular Article
• Published: 30 November 2021
• Volume 32 , pages 141–149, ( 2023 )

Cite this article

• Pramudya Dwi Aristya Putra   ORCID: orcid.org/0000-0002-1166-8220 1 ,
• Nurul Fitriyah Sulaeman 2 ,
• Supeno 1 &
• Sri Wahyuni 1  

5725 Accesses

9 Citations

1 Altmetric

Explore all metrics

Critical thinking skills (CTS) have been applied in the learning environment to address students' challenges in the twenty-first century. Therefore, specific approaches need to be implemented in the learning environment to support students' CTS. This research explores students' CTS during the learning process through the engineering design process (EDP) in a physics classroom. The methodology relied on a case study where students were situated for the first time in an EDP classroom. Data were analyzed for each of the EDP stages based on CTS criteria codes. The accuracy of the data was tested through a peer review process to demonstrate the validity of the analysis. The results showed that students exhibited specific CTS criteria in each EDP stage. Therefore, EDP could be an alternative method to engage CTS. This result contributes empirical evidence that the research on CTS also needs the students' performance while engaged in EDP.

Similar content being viewed by others

T/E design based learning: assessing student critical thinking and problem solving abilities

Toward a Systematic and Model-Based Approach to Design Learning Environments for Critical Thinking

Systematic design of a learning environment for domain-specific and domain-general critical thinking skills.

Avoid common mistakes on your manuscript.

In the twenty-first century, critical thinking skills (CTS) are becoming one of the most crucial learning activities (Fuad et al., 2017 ; Kavenuke et al., 2020 ). CTS support students in making decisions in a specific way during the learning process. When students face a given problem, CTS drive a person to analyze a problem and evaluate possible solutions. In this approach, CTS also offer an opportunity for students to use a reasonable rationale for their thinking, reflecting on the problem, and the potential solution (Ennis, 1993 ).

Critical thinking (CT) has been defined as a cognitive process involving reasonable reflective thinking to develop a decision based on the problem faced by a person; CTS include a person's ability for higher-order thinking, problem-solving, and metacognition (Ennis, 1989 ). Furthermore, CT is reasonable reflective thinking focused on a decision that the students believe in or do, which is in the cognitive domain (Ennis, 1993 ). In contrast, Facione ( 1990 ) conceptualized that CTS relate to cognitive ability and affective ability to be a good thinker. CT is a reflective thinking skill involving analyzing, evaluating, or synthesizing relevant information to form an argument to make a decision (Ennis, 1993 ; Ghanizadeh, 2017 ). Wechsler et al. ( 2018 ) proposed that CTS be required to display tests conducted after the CTS process in the classroom and review student behavior during the learning process.

However, research on CTS has been conducted using single tests to determine students' cognitive ability for CT. For example., in a study by Mutakinati et al. ( 2018 ), the students in a junior high school were given a CTS post-test following a science lesson. The results showed that the students had sufficient thinking skills to critique their plan for systematic practice, including constructing a realistic critique on their power of thought. Additionally, a study by Fuad et al. ( 2017 ) trained students to study more by using exploratory questions and information about how to develop a hypothesis, assisting students in creating learning based on the students' needs. He gave a post-test to the students to evaluate their CTS.

It is crucial to utilize a learning approach that supports a student's thinking in the learning process (Shaw et al., 2020 ). CTS can be developed during the learning process using teaching approaches that prompt students to face real-world problems. The teacher can select a teaching approach that pushes students to explore their CT through argumentation to make decisions (Ghanizadeh, 2017 ). One of the teaching approaches to facilitate CTS is the engineering design process (EDP). In the EDP, reflective thinking is needed to produce a better decision in solving a problem given by the teacher. Yu et al. ( 2020 ) investigated the relationship between CTS and the EDP when students design a product. The results indicated that the EDP stages played an essential role in students' understanding of their own CTS. Using EDP student experienced to define a problem, develop the argumentation, and finally make a decision that matches the EDP step (Spector & Ma, 2019 ; Sulaeman et al., 2021 ). The EDP implementation with several stages of learning, allows students to define the problem before they make a decision (Arık & Topçu, 2020 ; Tank et al., 2018 ).

Research exploring how EDP stages engage CTS is scarce. Additionally, most of the research on CTS has been conducted quantitatively using statistical analyses (e.g., Kavenuke et al., 2020 ; Mutakinati et al., 2018 ; Yu et al., 2020 ). The performance of CTS is also essential to highlight the students' behavior during the utilization of their CT abilities (Ennis, 1993 ). This study explores the students' CTS in the EDP project. One challenge in this study is the possibility of excessive subjectivity when analyzing CT performance; the authors used the peer review process to reduce this potential concern (Merriam & Tisdell, 2016 ). Thus, the research questions to guide this study are as follows:

To what extent could the EDP support CTS?

How does the EDP support a student's CTS in the physics classroom by defining the problem, using argumentation, and developing a solution?

Theoretical Framework

Critical thinking skills (cts).

The urgency of CTS could be traced from the educational theory by Dewey. Experiential learning theory explains the practical learning in enquiry practices (Dewey, 1993 ). Dewey suggested that the essence of an enquiry is formulated in the experiential learning cycle, which is initiated with the perception of solving a problem and exploration of relevant knowledge to construct a meaningful explanation of the solution in solving the problem (Garrison et al., 2001 ). This experiential learning cycle is a form of reflective thinking to produce better solutions (Garrison & Arbaugh, 2007 ). Reflection demands to think critically to identify solutions from a problem (Antonieta et al., 2005 ). CT is reasonable thinking to develop a decision based on a problem caused (Ennis, 1989 ). The CT be included a person skill in reflecting a solution of a problem given (Ennis, 1993 ). CTS need to be identified in both cognitive and affective concepts (Facione, 1990 ; Shaw et al., 2020 ). Kavenuke et al. ( 2020 ) explained that CTS involves synthesizing, analyzing, and evaluating information to make a cognitive decision and transform it into affective domain performance.

Besides being related to the cognitive domain, CTS is also related to the affective domain. This domain engages students in communication to support their decision through argumentation (Antonieta et al., 2005 ). Students have an opportunity to criticize using scientific statements in a scientific environment when they are communicating their idea (Farmer & Wilkinson, 2018 ). CTS begin with a simple experience, such as observing a difference, encountering a problem, or questioning someone's statement, and then leads to an enquiry; then, more complex experiences are encountered, such as interactions through communication in the application of higher-order thinking skills (Spector & Ma, 2019 ).

Measurement tools have been developed using criteria to describe a person's ability in CT. Ernst and Monroe ( 2004 ) analyzed criteria while measuring CTS, such as interpretation, analysis, evaluation, inference, explanation, and self-regulation. Also, CT ability is developed in detail through enquiry, argumentation, and self-regulation (Kabir, 2002 ; Spector & Ma, 2019 ). Using argumentation in CT, students can select evidence that supports their decision (Giri & Paily, 2020 ). Moreover, the measurement of CT was developed based on some of the research and models available. CT could be assessed using an open-ended assessment model, multiple-choice with written justification model, essay testing of critical thinking model, and performance assessment model (Ennis, 1993 ). In general, the measurement in CTS is given through an experimental study (e.g., Farmer & Wilkinson, 2018 ; Fuad et al., 2017 ; Yu et al., 2020 ). The CTS then can be described in the median data collected statistically to different students' levels, such as low and high (Kim et al., 2013 ). However, studies to demonstrate the performance of CT in the learning process are lacking, requiring further investigation.

Research in measuring CT through performance tests can be done by applying a learning approach that describes reflective thinking (SEN et al., 2021 ; Yu et al., 2020 ). The learning approach shows the cycle learning that includes defining a problem, developing a design solution through scientific argumentation, and deciding. One of the learning approaches that facilitate the cycle model in the classroom is the implementation of the EDP.

Engineering Design Process (EDP)

Engineering is a discipline that solves problems by obeying constraints, using a body of knowledge implemented through science, math, and technological tools (NGSS, 2013 ; NRC, 2012 ). Continuing design is a critical aspect in engineering that aims to solve a problem by iterative thinking, being open to the idea of having many possible solutions, along with a meaningful understanding of the integration of science, math, and technological concepts (Guzey et al., 2019 ; Moore et al., 2014 ). The design process in engineering is needed to develop collaboration and social communication (Sulaeman et al., 2021 ; Yazici et al., 2020 ). Thus, the EDP's goal is to solve a real-world problem with an engineer-designed activity.

The engineering practice is recognized by students during the cycling process, solving a problem in several stages. These stages engage students in identifying a problem, understanding the engineering need, and the opportunity to offer multiple possible solutions (Lottero-perdue et al., 2015 ; Whitworth & Wheeler, 2017 ). Students develop a critical understanding of the potentially relevant issues within the problem statement, allowing them to generate the best solution in the engineering classroom (Arık & Topçu, 2020 ).

The EDP addresses students' abilities to make decisions by defining a problem, developing argumentation, and identifying a solution for the problem (Guzey et al., 2016 ; Mathis et al., 2017 ). Therefore, in this study, the EDP stage is a bridge to facilitate CTS. More specifically, the implementation of EDP involves cycling, which starts with defining a problem, learning a scientific concept, planning a solution, trying a solution, and deciding (Tank et al., 2018 ). Furthermore, each EDP step can facilitate the CTS to develop a solution in the engineering classroom using the investigation through defining a problem, developing argumentation, and making a decision (Ahern et al., 2012 ).

Methodology

Research design.

A single case study was utilized to explore a student's experience in the EDP classroom in relation to CTS (Yin, 2018 ). The single case study was selected because of the desire to explore an in-depth EDP based on the criteria of general CTS based on Ernst and Monroe ( 2004 ). This study was conducted during the pandemic era (when covid-19 hit in the selected area), so the study was conducted using two different approaches: both online and off-line learning (i.e., blended learning). The authors developed an EDP worksheet that guided individual activities and group activities. Due to the regulations affecting education during the pandemic, the classroom only allowed a maximum of 15 students.

Context of Study

The study was conducted in a physics classroom in one of the high schools, located in one district, in Indonesia. Students never followed the same program during the EDP project, particularly in physics. Through the EDP, the students had to define a problem, learn the physics concepts and the related subjects, develop a solution plan, and make a decision about their solution (Tank et al., 2018 ). The authors developed a EDP worksheet. This EDP worksheet is an instructional sheet for students to understand the EDP stages in this project (Sulaeman et al., 2021 ).

The team project addressed one challenge in which students could build a solution based on the given problem. The problem given by the worksheet involved asking students to solve a problem regarding the location of rice fields. The situation addressed the lack of water during the dry season and the amount of water during the rainy season. Figure  1 shows the EDP activities in the classroom, which total 315 min of activities. The project was solved by students individually and in groups to assess the consistency of improving the CTS of the students.

The EDP steps during implementation in the physics classroom

Participants

There were 12 students in this study in the tenth grade at the time of data collection. They volunteered their participation, joining the EDP project in both the online and off-line classroom. They also agreed to follow government regulations regarding health protocols and received permission from their parents to participate. The full-time physics teacher identified students' levels in various physics achievements. The demographics of the students are described in Table 1 . The level of achievement is divided into three categories: high (student's achievement was more than 75); medium (student's achievement was more than 65 but less than 75); and low (student's achievement was less than 65). The score of 75 was a standard value to grade students' mastery in physics concepts in this school.

Data Collection

There were three data sources: text based on the EDP worksheet, the recording of students' group discussion, and the recording of the students' interviews. First, Text was collected based on the students answer from EDP worksheet. EDP worksheet presented students with activities to work on individually, enabling the collection of data regarding students' problem definition and learning. Students then developed an opinion based on the question asked: for example, who has a problem? What is the problem? And who is the user? In the individual work, the students' writing was collected. Second, when student worked in their group, the discussion process was recorded. All the students' communication was transcribed to analyze. The data collection was focused on stages of plan, try, test, and decide. Third, Students were also interviewed to acquire the necessary supporting data on the changes in students' CT abilities (see Online Appendix A).

Data Analysis

The three types of data were analyzed to triangulate strategies for confirming the accuracy of the data (Creswell & Poth, 2016 ). The authors then identified the stage of the EDP (see Fig.  1 ) and matched the code of CT. All the data collected were transcribed into text and coded as shown in Table 2 . The code of CT was developed using the criteria developed by Ernst and Monroe ( 2004 ). All students’ statements on the each EDP stage was read carefully and given a justification based on the CTS criteria. The number of the CTS criteria were calculated in each EDP and presented in Table 3 . Those criteria were divided into two levels to express the students' CT abilities based on the development of inductive code in the site (Saldana, 2016 ). Thus, each author coded the data (see example in Online Appendix B) as a peer examination (Merriam & Tisdell, 2016 ). When the coding presentation differed between authors, the authors met to negotiate a consensus students’ statement.

This research aimed to explore students' CTS through approaches of the EDP in a physics classroom. Our findings discuss the matrix of suitability of CTS aspects and each EDP stage. In addition, the results were organized based on the students' ability to define a problem, provide scientific argumentation, and generate a solution.

Matrix of Suitability of CTS Criteria and EDP Stages

From our analysis of students' worksheets and students' discussions in the physics classroom, the results indicated the justification of CTS criteria in each EDP stages. Table 3 provides code frequencies counts for the students’ statement in the EDP project.

Defining a Problem

In the EDP, defining a solution is the first step for students in the problem-solving process. Students experienced solving a problem with a focus on the necessary part of the situation and the constraints given by the teacher. In the "Define" step, students generally started by interpreting and identifying a problem given by the teacher, highlighting the problem statement and the need for problem-solving processes. Some examples were given that were expressed in the worksheet about the problem given: difficulties of watering in the farm field.

[A1]: Farmers in the western rice field of the village made small wells near their fields with the help of diesel pumps to supply water to their fields. The way for farmers to stop pumping water is by building a dam. [A1]: The client wants the dam to last a long time with an estimated cost of $2000. It is useful for storing water during the rainy season and supplies water for the dry season so that farmers do not have to pump water anymore. [A2]: The water supply is low in the irrigation area of the river. The river flow in the area is very small during the dry season, while other factors also influence the depth and width of the river. As a result, the branches of the river sometimes do not reach the rice fields, so the rice plants often lack water. [A2]: The head of the village wants to make a dam that has a width in total 3 metres, 2.5 metres for storing the water, and ½ metre for anticipating when the water overflows

Students [A1] and [A2] categorized the two statements based on the engineering problem given. They stated the problem, and they clarified a constraint to solve the problem. They did not only state the problem about the lack of watering in the rice field, but also mentioned constraints to solve the problem. Those students were categorized in the high level of interpretation criteria of the CTS. Students provide the problem information and the constrain to solve the problem. On the other hand, the two examples below show students low in CTS.

[U1]: During the dry season, the river flow in the area was very small, so that the water in branches of the river did not reach the rice fields [for watering]. [U2]: There was very little water supply in the irrigation area of the river and the [water] flow rate was very small during the dry season, so [the] availability of water in the river is little during the dry season.

Students [U1] and [U2] were less skilled in interpreting the problem given by the teacher. Even though the students could state the problem in the village, they could not adequately to explain of the constraint when asked to develop a solution.

Students' Argumentation

Students in the EDP classroom also provided an argument when they planned, tried, and tested possible effective solutions. In those steps, students worked as a group to discuss their solution for solving a problem, totalling four students per group. The data shown in this section uses vignettes to show each group's manner of discussion. The discussion in this example shows when students were trying to develop a solution. Students [A1], [S1], and [U1] discussed the possible implementation of one of the physics concepts to build a dam. The student [U1] showed an increase in the level of CTS when joining the discussion section with the group.

[A1]: I think the materials used must be strong and durable to build the foundation of the dam. We also need the concept of physics using hydrostatic pressure. The lowest foundation is built wide and thicker, while the higher one is like the shape of a cone like this [while demonstrating by hand the shape of cone]. [S1]: Then Pascal's Law also explain the water capacity. During the rainy season, the water does not flow out, or in other words, water can accommodate both in raining and dry season. The water can irrigate the rice fields continuously. [U1]: Yes, I agree, so we also implement Pascal's Law, and we pay attention to the capacity during the rainy season so that the water doesn't overflow [water spill from the dam].

In Vignette 1, the presented discussion is between the students to plan the creation of the dam. They collected physics-related evidence to emphasize the possible implementation of this real-world problem. They learned the concepts of hydrostatic pressure, serving as a base to solve the problem. Student [A1] demonstrated high CTS because he examined ideas by analyzing the concept of physics. His statement is a clear expression of the CTS of analysis, and student [S1] examined the alternative of concept work in emphasizing the situation by collecting evidence of the amount of water during dry or rainy seasons. She demonstrated high CTS of evaluation criteria. Student [U1] also described and emphasized the solution by drawing a conclusion of the dam being built. He also demonstrated high CTS in terms of the inference criteria.

In the "Test" step, students concluded their problem design and coordinated it with their understanding of the criteria needed. In Vignette 2, the discussion shows students offering further clarification about their design. Students evaluated the criteria of the design based on the problem given.

[U1]: The dam has a width of 3 metres; it's almost the same as the constraint. [S1]: Yes, 2.5 metres is added during the rainy season and ½ metres during the dry season. [A1]: The dam must last a long time with a budget of $2000. [S1]: Okay, this means, yes, the criteria requested by the client are a dam that has a width of 3 metres and a depth of 2.5 metres during the rainy season, and ½ metre during the dry season, and the dam must last a long time with a budget of $2000." (Writes down the criteria requested by the client)

Students [U1] and [S1] emphasized the dam size based on the client's request, and they also analyzed the total approved budget. The budget was used to guide the criteria of the dam, so here they are rethinking the criteria request by the client (chief of the village). [S1] agreed with the situation offered by his peer, and she interpreted it by clarifying the size and the budget.

Develop Decision

The "Decision" step is the final step in the EDP. The students worked in groups, comparing their design to the other group's design. In this step, the majority of CTS were self-regulated (see Table 4 ). Students presented the results of their final design, at the front of the classroom, to show that their design could effectively solve the problem. Furthermore, the students compared their designs, drawing a conclusion to redesign when their design failed. Table 4 shows Group 1's design compared to that of Group 2.

Students used this data to develop an improved design. Moreover, in this step, students also implemented a redesign to solve the problem given. This situation expressed high CTS in self-regulation criteria; student [A1] stated the advantages of their group's dam design.

[A1]: The dam made by our group uses the concepts of Pascal's Law and hydrostatic pressure, so that the construction can be durable and sturdy. It also has another advantage, namely the budget is not more than $2000. However, it still has a drawback, which is to look at the situation and condition of the cost of the dam.

This study showed that EDP is beneficial in supporting students' CTS. Each stage of the EDP could be investigated, focusing on the majority of CTS exhibited. This result is in line with the results uncovered by Yu et al. ( 2020 ): the EDP plays an important role in developing CTS. When students work through all the stages of the EDP, they also develop and meet the criteria of CTS. Especially, when students did individual work, they described the cognitive domain based on the CTS; students identified the problem in the situation, highlighted the goal of the human need, and paid attention in the constraint to solve a problem (Ernst & Monroe, 2004 ). When students worked in the groups to discuss their ideas, they tried to exchange their ideas in the stage of planning a solution, tried their design, tested, and decided on their design (Giri & Paily, 2020 ; Kabir, 2002 ). This situation described the affective domain because students communicated based on their argumentation to reinforce their ideas showed an affective domain in CTS (Antonieta et al., 2005 ).

The goal in the EDP classroom was for students to make decisions regarding effective solutions to solve the lack of water in rice fields. Following the steps of the EDP, going from "Plan" to "Test," students clearly utilized the process of argumentation to decide. This implementation of the EDP followed reflective thinking because students also conducted self-regulation to redesign their solution. Self-regulation involves the necessity of re-establishing the performance that students think is recursive to setting the goal (Ghanizadeh, 2017 ). Furthermore, students thought back and forth between the problem given and the solution produced, following the learning cycle (Dewey, 1993 ; Garrison et al., 2001 ). In addition, during these activities, students designed a solution based on their understanding of the physics concepts learned.

This process highlighted that student can improve their CTS. Student [U1] showed that in the individual activities he had low CTS, but after joining the group discussion he had a high level of CTS. This phenomenon shows that group interaction can improve a student's level of CTS through communication (Farmer & Wilkinson, 2018 ). Students' communication regarding the design of the solution showed the concept of argumentation because students provided evidence to support their claims (Mathis et al., 2017 ). Therefore, students argued using a well-founded reason from various sources, including discussion activities, which in turn require CTS (Yazici et al., 2020 ).

This study emphasizes that in the EDP stages could express the specific criteria of student's CTS. Additionally, the contribution of this study is that it seems essential to show that measuring of CTS need student’s performance to conform achieving in CTS. Mainly, the cyclical thinking and the use of self-regulation could improve solutions based on the given problems. Using the EDP also gives students an opportunity to communicate with each other to build a better solution based on physics concepts. Furthermore, this research also asked students to rethink the management strategies for solving problems via communication in the group that described CTS based on the affective domain. This research differed from previous research that has investigated the link between the EDP and students' CTS by giving students post-tests of CTS (Mutakinati et al., 2018 ; Yu et al., 2020 ). Students could be investigated in more detail using cyclical thinking to generate the best solution based on the problem given and the constraints (Arık & Topçu, 2020 ; Lottero-perdue et al., 2015 ; Tank et al., 2018 ).

This research provides, through a qualitative study, empirical evidence that the EDP supports a student's CTS. In the case study, students utilize their CTS based on the dominant criteria that appeared in each step of the EDP. The EDP facilitated students' collaboration, working in groups, where students could share and explore their ideas. Additionally, students engaged in argumentation while they began the phases of planning, trying, and testing. After students decided on their design to solve the problem, they conducted self-examination, looking over their design, seeing if the results were comparable to other groups. This situation demonstrated iterative thinking, which is one of the goals of effective CTS.

This research described the infusion of engineering as central to integrating science, technology, engineering, and mathematics (STEM) approaches. This study implies that CTS should be measured during the learning process. For the teacher, the EDP approach might be formulated to teach integrated STEM in the future. Through EDP stages, the integration subjects could be involved in the classroom to support the students' 21st-century skills. The policymaker needs to support the implementation of EDP in the school curricula because students showed positive behavior in the CTS. Additionally, providing professional development (PD) in implementing engineering education is also essential to running the EDP in the classroom. Through this PD, the application of EDP in STEM learning will improve both in quality and quantity.

Furthermore, this study was limited in the sampling of participants, but the exploration of CTS in EDP stages could be described in detail. Analysis in the greater participants needs to be assessed to show the results consistently. Moreover, the EDP is to infuse the engineering in STEM education, so that in the future research, the analysis of student’ CTS through STEM learning can be investigated to gather student’ understanding comprehensively in science, technology, engineering, and mathematics subjects than in silo subject.

Ahern, A., O’Connor, T., McRuairc, G., McNamara, M., & O’Donnell, D. (2012). Critical thinking in the university curriculum—The impact on engineering education. European Journal of Engineering Education, 37 (2), 125–132. https://doi.org/10.1080/03043797.2012.666516

Article   Google Scholar  

Antonieta, M., Celani, A., & Collins, H. (2005). Critical thinking in reflective sessions and in online interactions. AILA Review, 18 , 41.

Arık, M., & Topçu, M. S. (2020). Implementation of engineering design process in the K-12 science classrooms: Trends and issues. Research in Science Education . https://doi.org/10.1007/s11165-019-09912-x

Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five approaches . Sage publications.

Google Scholar  

Dewey, J. (1993). How we think. A restatement of the relation of reflective thinking to the educative process . Berlin: D. C. Heath.

Ennis, R. H. (1989). Critical thinking and subject specificity: Clarification and needed research. Educational Researcher, 18 (3), 4.

Ennis, R. H. (1993). Critical thinking assessment. Theory into Practice, 32 (3), 179–186. https://doi.org/10.1080/00405849309543594

Ernst, J., & Monroe, M. (2004). The effects of environment-based education on students’ critical thinking skills and disposition toward critical thinking. Environmental Education Research, 10 (4), 507–522. https://doi.org/10.1080/1350462042000291038

Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction . The California Academic Press.

Farmer, J. L., & Wilkinson, L. (2018). Engineering success: Using problem-based learning to develop critical thinking and communication skills in a Chemical Engineering classroom. Proceedings of the Canadian Engineering Education Association (CEEA) . https://doi.org/10.24908/pceea.v0i0.13057

Fuad, N. M., Zubaidah, S., Mahanal, S., & Suarsini, E. (2017). Improving junior high schools’ critical thinking skills based on test three different models of learning. International Journal of Instruction, 10 (1), 101–116. https://doi.org/10.12973/iji.2017.1017a

Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future directions. Internet and Higher Education, 10 (3), 157–172. https://doi.org/10.1016/j.iheduc.2007.04.001

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. International Journal of Phytoremediation, 21 (1), 7–23. https://doi.org/10.1080/08923640109527071

Ghanizadeh, A. (2017). The interplay between reflective thinking, critical thinking, self-monitoring, and academic achievement in higher education. Higher Education, 74 (1), 101–114. https://doi.org/10.1007/s10734-016-0031-y

Giri, V., & Paily, M. U. (2020). Effect of scientific argumentation on the development of critical thinking. Science and Education, 29 (3), 673–690. https://doi.org/10.1007/s11191-020-00120-y

Guzey, S. S., Moore, T. J., & Harwell, M. (2016). Building up stem: An analysis of teacher-developed engineering design-based stem integration curricular materials. Journal of Pre-College Engineering Education Research, 6 (1), 10–29. https://doi.org/10.7771/2157-9288.1129

Guzey, S. S., Ring-Whalen, E. A., Harwell, M., & Peralta, Y. (2019). Life STEM: A case study of life science learning through engineering design. International Journal of Science and Mathematics Education, 17 (1), 23–42. https://doi.org/10.1007/s10763-017-9860-0

Kabir, S. (2002). Literature review Literature review. Basic Guidelines for Research, 6 (July), 33–37.

Kavenuke, P. S., Kinyota, M., & Kayombo, J. J. (2020). The critical thinking skills of prospective teachers: Investigating their systematicity, self-confidence and scepticism. Thinking Skills and Creativity, 37 (March 2019), 100677. https://doi.org/10.1016/j.tsc.2020.100677

Kim, K., Sharma, P., Land, S. M., & Furlong, K. P. (2013). Effects of active learning on enhancing student critical thinking in an undergraduate general science course. Innovative Higher Education, 38 (3), 223–235. https://doi.org/10.1007/s10755-012-9236-x

Lottero-perdue, B. P., Bolotin, S., Benyameen, R., Brock, E., & Metzger, E. (2015). The engineering design process-5E. Science and Children, 3 (1), 60–66.

Mathis, C. A., Siverling, E. A., Glancy, A. W., & Moore, T. J. (2017). Teachers’ incorporation of argumentation to support engineering learning in STEM integration curricula. Journal of Pre-College Engineering Education Research, 7 (1), 76–89. https://doi.org/10.7771/2157-9288.1163

Merriam, S. B., & Tisdell, E. J. (2016). Qualitative Research A guide to design and implementation (4th ed.). Jossey-Bass A Wiley Brand.

Moore, T. J., Stohlmann, M. S., Wang, H. H., Tank, K. M., Glancy, A. W., & Roehrig, G. H. (2014). Implementation and integration of engineering in K-12 STEM education. In Engineering in pre-college settings: Synthesizing research, policy, and practices. Purdue University Press

Mutakinati, L., Anwari, I., & Yoshisuke, K. (2018). Analysis of students’ critical thinking skill of middle school through stem education project-based learning. Jurnal Pendidikan IPA Indonesia, 7 (1), 54–65. https://doi.org/10.15294/jpii.v7i1.10495

NGSS. (2013). Next generation science standards: For states, by states . National Academies Press.

NRC. (2012). STEM integration in K-12: Status, prospects, and an agenda for research engineering . The National Academic Press.

Saldana, J. (2016). Qualitative coding: The manual researchers for qualitative researchers . SAGE Publications Ltd.

Sen, C., Sonay, Z., & Ahmet, K. S. (2021). Computational thinking skills of gifted and talented students in integrated STEM activities based on the engineering design process: The case of robotics and 3D robot modeling. Thinking Skills and Creativity . https://doi.org/10.1016/j.tsc.2021.100931

Shaw, A., Liu, O. L., Gu, L., Kardonova, E., Chirikov, I., Li, G., Hu, S., Yu, N., Ma, L., Guo, F., Su, Q., Shi, J., Shi, H., & Loyalka, P. (2020). Thinking critically about critical thinking: Validating the Russian HEIghten® critical thinking assessment. Studies in Higher Education, 45 (9), 1933–1948. https://doi.org/10.1080/03075079.2019.1672640

Spector, J. M., & Ma, S. (2019). Inquiry and critical thinking skills for the next generation: from artificial intelligence back to human intelligence. Smart Learning Environments . https://doi.org/10.1186/s40561-019-0088-z

Sulaeman, N. F., Putra, P. D. A., Mineta, I., Hakamada, H., Takahashi, M., Ide, Y., & Kumano, Y. (2021). Exploring Student Engagement in STEM Education through the Engineering Design Process. Jurnal Penelitian Dan Pembelajaran IPA, 7 (1), 1. https://doi.org/10.30870/jppi.v7i1.10455

Tank, K. M., Rynearson, A. M., & Moore, T. J. (2018). Examining student and teacher talk within engineering design in kindergarten. European Journal of STEM Education . https://doi.org/10.20897/ejsteme/3870

Wechsler, S. M., Saiz, C., Rivas, S. F., Vendramini, C. M. M., Almeida, L. S., Mundim, M. C., & Franco, A. (2018). Creative and critical thinking: Independent or overlapping components? Thinking Skills and Creativity, 27 , 114–122. https://doi.org/10.1016/j.tsc.2017.12.003

Whitworth, B., & Wheeler, L. (2017). Is It engineering or not? The Science Teacher, 084 (05), 25–29. https://doi.org/10.2505/4/tst17_084_05_25

Yazici, H. J., Zidek, L. A., & St. Hill, H. (2020). A study of critical thinking and cross-disciplinary teamwork in engineering education. In A. E. Smith (Ed.), Women in industrial and systems engineering: Key advances and perspectives on emerging topics (pp. 185–196). Berlin: Springer. https://doi.org/10.1007/978-3-030-11866-2_8

Chapter   Google Scholar  

Yin, R. K. (2018). Case study research and applications: Design and methods. Journal of Hospitality & Tourism Research . https://doi.org/10.1177/109634809702100108

Yu, K. C., Wu, P. H., & Fan, S. C. (2020). Structural relationships among high school students’ scientific knowledge, critical thinking, engineering design process, and design product. International Journal of Science and Mathematics Education, 18 (6), 1001–1022. https://doi.org/10.1007/s10763-019-10007-2

Download references

Author information

Authors and affiliations.

Department of Science Education, Faculty of Teacher Training and Education, University of Jember, Jember, Indonesia

Pramudya Dwi Aristya Putra,  Supeno & Sri Wahyuni

Department of Physics Education, Faculty of Teacher Training and Education, Mulawarman University, Samarinda, Indonesia

Nurul Fitriyah Sulaeman

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Pramudya Dwi Aristya Putra .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 17 kb)

Rights and permissions.

Reprints and permissions

About this article

Putra, P.D.A., Sulaeman, N.F., Supeno et al. Exploring Students' Critical Thinking Skills Using the Engineering Design Process in a Physics Classroom. Asia-Pacific Edu Res 32 , 141–149 (2023). https://doi.org/10.1007/s40299-021-00640-3

Download citation

Accepted : 19 November 2021

Published : 30 November 2021

Issue Date : February 2023

DOI : https://doi.org/10.1007/s40299-021-00640-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Critical thinking skills
• Engineering design process
• Physics classroom
• Find a journal
• Publish with us
• Track your research

Double Helix: A Journal of Critical Thinking and Writing

• faculty development, online writing instruction, scientific writing, postsecondary education, writing studies, WAC, writing in the disciplines, Teaching strategies, Pedagogy

Advisory Board

Editorial Team

Past Volumes

Submissions

About Double Helix

Current Volume:

Critical Thinking, Writing, and Artificial Intelligence

Ask AI. Double Helix: A Journal of Critical Thinking and Writing , 2024. Large Language Model.

Table of Contents

Research articles, reports from the field, book reviews.

Double Helix is a publication of the College of Arts and Sciences at Quinnipiac University. ISSN 2372-7497. Works published in Double Helix are released under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 international License.

• Open access
• Published: 01 March 2024

Assessment of the effectiveness of the BOPPPS model combined with case-based learning on nursing residency education for newly recruited nurses in China: a mixed methods study

• Yongli Wang 1   na1 ,
• Yiqian Chen 2   na1 ,
• Ling Wang 1 ,
• Wen Wang 1 ,
• Xiangyan Kong 1 &
• Xiaodan Li 1  

BMC Medical Education volume  24 , Article number:  215 ( 2024 ) Cite this article

801 Accesses

Metrics details

Expanding new nurse training and education is a priority for nursing educators as well as a critical initiative to stabilize the nursing workforce. Given that there is currently no standardized program for the training of new nurses in China, we investigated the effectiveness of the bridge-in, objective, pre-assessment, participatory learning, post-assessment, and summary model combined with case-based learning ((BOPPPS-CBL) for the standardized training of new nurses.

The mixed method approach with explanatory sequential (quantitative-qualitative) method was used. A questionnaire was used to compare the impact of the BOPPPS-CBL model and the Traditional Learning Model (TLM) on the core competencies of 185 new nurses for two years of standardized training. Quantitative data were analyzed using SPSS 22.0. Focus group interviews were used with four groups of new nurses and perceptions of BOPPPS-CBL training were recorded. Qualitative data were analyzed thematically.

According to the quantitative data, more new nurses agreed that the BOPPPS-CBL model stimulated their learning and improved their core nursing competencies than the TLM. The BOPPPS-CBL group outperformed the TLM group on theoretical knowledge tests. Qualitative data revealed that 87.5% of new nurses agreed on the value of BOPPPS-CBL training, and three themes were extracted: (1) role promotion; (2) formation of new thinking to solve clinical problems; and (3) suggestions for improvement.

BOPPPS-CBL training had a significant impact on improving new nurses’ core competencies and promoting the transition of new nurses to clinical practice nurses in China. The study recommends BOPPPS-CBL training as an effective teaching model for the standardized training and education of new nurses.

Peer Review reports

At the end of 2020, there were nearly 4.45 million registered nurses in China [ 1 ], and some studies predict that by 2035 [ 1 , 2 ], the demand for nurses in China will be 6.75 per 1,000 people. Because of this, many nurses will enter the clinic in the future. Unfortunately, however, new nurses are one of the groups with a high incidence of adverse nursing events [ 3 ]. Research has shown that strengthening nursing education and training to improve core competencies at all stages of care can ensure patient safety and improve global health [ 4 , 5 ]. New nurse training is a vital aspect of hospital nurse training and can help new nurses solve problems during the transition from the nursing student stage to the clinical nurse role.

New nurses encounter various obstacles [ 6 ], such as, in the nursing student stage, they focus on acquiring theoretical knowledge and lack nursing practical ability which together with the lack of clinical work experience, leads to a weak link between their theoretical knowledge and clinical practice. Moreover, due to the poor core competence of new nurses, when facing clinical work, they become mentally stressed with negative emotions such as fear, anxiety, and even burnout [ 7 ]. These factors often make it difficult for new nurses to feel professionally fulfilled, which can severely affect their clinical performance and career planning. As a result, some new nurses may decide to leave this field [ 8 ]. According to research, new nurses can successfully transition into the role of clinical practice nurses by strengthening their core competencies [ 9 ].

There are no uniform standards for nursing core competencies globally. The International Council of Nurses (ICN) states that nursing core competencies are the application of a nurse’s knowledge, skills, judgment, and personal attributes in the performance of nursing duties [ 10 ]. Nursing core competencies have been defined differently depending on the state of nursing in each country. The Australian Nurses and Midwives Association [ 11 ] considers nursing core competencies as the foundation of nursing practice and the criteria and basis for assessing nurses’ competence in the workplace, which encompasses skills, knowledge, attitudes, values, and competencies in the professional domain. In China, core competencies are defined as “knowledge, skills, attitudes, judgments, and clinical problem-solving abilities within the prescribed practical roles and environments“ [ 12 ]. In the UK, new nurses receive no less than four months of training with a focus on mentoring support [ 13 ]. Australia launched a one-year general practice training program for new graduate nurses to emphasize the importance of primary health care [ 14 ].

The National Health Commission of China issued the “Training Outline for Newly Recruited Nurses” (hereafter referred to as the “Outline”), which is a guideline for on-the-job training and continuing education. All new registered nurses (first-time job holders, regardless of education) must receive 2 years of clinical training in their nursing specialization upon entry, shortening the transition of new nurses to clinical practice nurses [ 15 , 16 ]. However, the Outline is only a guiding document for in-service training and continuing education, and there is not yet a unified, specific, detailed, and standardized training system for new nurses nationwide. Most hospitals design their training system under the guidance of the Outline, whose effectiveness is yet to be considered.

Most hospital’s training content includes theoretical and skill training, the traditional learning model education method is that the teacher mainly explains, and nurses only passively accept this knowledge, not including some active teaching methods and techniques [ 17 ]. Furthermore, overall training takes a long time, and the training format is relatively simple, which does not encourage participation and enthusiasm, resulting in an unsatisfactory training effect [ 18 ]. The new-nurse education includes many knowledge points that focus on the ability to combine theory and practice [ 8 ]. Improving nurses’ core competencies during their 2-year standardized training and education is an urgent issue that must be addressed.

The bridge-in, objective, pre-assessment, participatory learning, post-assessment, and summary (BOPPPS) model is a closed-loop teaching process that emphasizes student participation and feedback and is internationally recognized for its effectiveness [ 19 ]. It is based on constructivist and humanistic learning theories. Based on the “student-centered” approach, the BOPPPS model divides the teaching design process into six links: bridge-in, objective, pre-assessment, participatory learning, post-assessment, and summary to ensure that the teaching objectives are met. Research confirms that the BOPPPS model has significant advantages over traditional teaching modes and places a greater emphasis on student participation [ 20 ]. The BOPPPS teaching model has been promoted and used in many nations worldwide. Shih [ 21 ] implemented the concept in a flipped classroom for a business etiquette course using quasi-experimental research, which resulted in increased teacher-student interaction, a more dynamic and fascinating class, and enhanced student learning outcomes ( t  = 3.10, P  < 0.01). Zhen employed a mixed research design to investigate the usage of design ideas based on the BOPPPS model in his teaching practice, which enhanced teaching methods, raised student interest in learning from 65 to 90%, and improved students’ higher-order thinking [ 22 ]. Other studies have demonstrated how the BOPPPS model can enhance ophthalmology teaching [ 23 ] and dental materials education [ 24 ] by encouraging clinical thinking abilities. As a result, it is important to acknowledge the BOPPPS model’s usefulness in medical education.

Moreover, a recent systematic evaluation [ 25 ] revealed that case-based learning (CBL), constructed upon authentic contexts within a constructivist framework, proved to be a advantageous teaching strategy for improving the performance and case-analysis abilities of medical students. In bridging the knowledge gap between theoretical understanding and clinical practice, CBL disseminates knowledge through clinical cases, with students taking a central role and cases providing guidance [ 26 , 27 ]. Additionally, it has been shown to enhance students’ problem-solving skills, critical thinking, and motivation to study [ 28 ].

In recent years, the teaching mode of BOPPPS combined with CBL has emerged, and it has been widely used in medical education such as continuing medical education and ophthalmology, among others, with satisfactory results [ 23 , 29 ]. Our study used an explanatory mixed methods research design which employed a quantitative quasi-experimental comparative design and a qualitative descriptive design from nurse focus group interview analysis. The study’s objectives were as follows:

Investigate the effectiveness of the BOPPPS-CBL model in the training of new nurses in China.

Explore nurses’ experience and suggestions regarding BOPPPS-CBL training to improve the program.

An interpretive sequential approach (quantitative-qualitative) mixed methods was used in this study [ 30 ]. Phase 1 was a quantitative experimental study, and Phase 2 was a descriptive qualitative study.

Participants

Participants included new nurses with standardized training at Peking University People’s Hospital from September 2017 to August 2021. We used a convenience sampling to enroll 115 nurses in the traditional learning model group (TLM group) who were almost all new in 2017 (due to resignation, 2 of the 117 nurses were excluded) and 70 in the BOPPPS-CBL group who were about all new in 2019 (3 nurses out of 73 withdrew from the study midway due to sick leave; Table  1 ). Inclusion criteria were: registered nurses, nurses just recruited and having participated in standardized training who agreed to participate in the research, and signed informed consent. The exclusion criteria were those who could not complete all the research contents for resignation, leave, and other reasons.

Training design

The training program was developed under the requirements of the Chinese Nursing Regulations and the “Outline”. Basic theoretical knowledge training, clinical nursing operation technique training, and professional theory and practical ability training in rotating clinical departments comprised the training content. The training period was set at 2 years. The assessment methods chosen were theoretical examination and clinical practice ability assessment.

BOPPPS-CBL group

This group was built on the foundation of the TLM group using the combined BOPPPS model and CBL.

First, we set up a research team. Team members comprised a deputy director of the nursing department ( n  = 1), senior specialist nurses ( n  = 3), as well as clinical teaching management nurses ( n  = 4). The team leader was assigned as the nursing department deputy director in charge of the design and quality control of CBL programs. Senior specialist nurses were in charge of retrieving, sorting, writing, and implementing CBL programs, while clinical teaching management nurses handled program implementation and evaluation.

Second, according to the bridge-in of the BOPPPS model, the principles of CBL writing were formulated: (1) typicality: inclusion of typical cases within the specialty; (2) authenticity:taken from real cases; (3) relevance: ensuring a significant correlation between the cases and teaching objectives; (4) guidance: embedding questions in the cases to steer nurses in their case discussions, analysis, and stimulate their thought processes; and (5) difficulty: introducing cases with a certain level of complexity to foster the cultivation of nurses’ independent thinking and judgment.

Third, we established CBL training objectives based on the BOPPPS model objectives. The first draft of the training plan was created using a literature review and teaching materials, following the compilation principles.

Fourth, the implementation and evaluation of the CBL training plan were formulated using the BOPPPS model’s pre-assessment, post-assessment, participatory learning, and summary, i.e., an eight-session training program that was conducted every 3 months for 3  ∼  4 h for 24 months. Before the case presentation, new nurses were given questions and quizzes to help them understand their existing knowledge structure and help them adjust their teaching content and methods. Subsequently, nurses participated in a full group discussion of the case. Following that, they were asked to list the nursing problems in the case while proposing the group’s solution strategies through problem discussion, as well as the group members’ contributions and sharing of insights to test the new nurses’ learning and assess their overall ability to participate in discussions, communicate, ask questions, and solve problems. Finally, the teacher provided feedback, summarized the case study’s key points, and guided the new nurses in their reflection.

Fifth, expert group meeting. According to Hasson’s opinion [ 31 ] the ideal number of experts selected is 4  ∼  16, in this study 9 experts were selected for the panel meeting discussion, and the experts were chosen using the following criteria: (1) research areas covering nursing education ( n  = 4), medical education ( n  = 3), and psychology ( n  = 2); (2) familiarity with the field of nursing education research; (3) intermediate and higher professional technical titles plus a master’s degree or above; and (4) voluntary participation on this paper and provision of informed consent. The experts’ familiarity was Cs = 0.833; the experts’ judgment basis Ca = 0.957; and the experts’ authority coefficient Cr = 0.895: the training program was revised through the expert group meeting, and the final plan was developed. The general information about the experts, their level of familiarity, and the revisions proposed by the experts were shown in Appendix 1 . The specific training mode was shown in Table  1 . The BOPPPS-CBL and TLM model flowchart was summarized in Fig.  1 .

Flowchart of teaching design of the BOPPPS-CBL and TLM groups

Data collection

The quantitative phase of the questionnaire survey.

The Competency Inventory for Registered Nurses (CIRN) was a Chinese self-assessment instrument created by Liu [ 32 ] to assess nurses’ core competencies using the ICN’s Core Competency Framework for Nurses. In China, this scale is widely used by registered nurses. The CIRN includes 55 items organized into seven categories: critical thinking and research, clinical care, leadership, interpersonal relationships, ethical and legal practice, professional development, and educational consultation. The response options on a 5-point Likert scale ranged from 0 (not at all competent) to 4, with total scores ranging from 0 to 220. A higher score indicates greater core competency. It takes about 10 to 15 min to complete the inventory. The total Cronbach’s α coefficient was 0.89; each dimension was 0.718 to 0.908 and the Content Validity Index was 0.852. The total Cronbach’s α coefficient was verified in other studies as 0.92 − 0.76 [ 33 ]; indicating good reliability and validity.

The Theoretical Knowledge Examination was created by the research team to assess nurses’ mastery of theoretical knowledge after training. After content validity evaluation by 10 nursing teaching experts, the Scale-level Content Validity Index, S-CVI/AVe of the paper was 0.949. The Cronbach’s α was 0.822, and the difficulty level of the exam paper was medium. The questions were scored out of 100 points and included multiple-choice and short text-based questions.

Focus group interviews in the qualitative phase

To avoid causing stress to the interviewees, the hospital education service staff (with a certificate in qualitative research) conducted focus group interviews with the nurses in the intervention group. The interviews were conducted to gain an understanding of their experiences and suggestions for training, as well as to improve the BOPPPS-CBL training program. Purposive sampling was used to recruit intervention group participants, and before the interview, all participants were informed as to the aims of the study and volunteered to participate. At the end of the BOPPPS-CBL training, 24 nurses were divided into four focus groups (5–7 nurses per group) to participate in the interviews, of which 22 were women and 2 were men, aged between 20 and 26 years. The first draft of the interview outline was finalized through literature reading, research team, and expert group meeting discussions, and the final outline was discussed again after pre-interviews with four nurses. The following was included in the interview outline: (a) How do you feel about taking part in this BOPPPS-CBL training course? (b) How has your clinical work changed after this training? (c) In what areas of this training program do you believe improvements should be made, and why? Share your ideas and opinions with us. The location of our interviews was chosen to take place in a quiet classroom. Interviews lasted 45–55 min. The whole process was conducted according to the qualitative research method.

The researcher’s team directed the TLM and BOPPPS-CBL training. Before the implementation of the training program, 10 nursing teaching teachers and 3 nurses with master’s degrees in nursing were uniformly and systematically trained. We explained the training’s aim, the procedure, what to expect during implementation, and how to gather data to assure quality control.

Data analysis

The quantitative data were coded and entered into IBM SPSS version 22 software for statistical analysis. Quantitative data for nurses were analyzed by t-test and expressed as mean ± standard deviation. Categorical data were analyzed by the chi-square test. The significance level for all tests was set at p  = 0.05.

For the qualitative part, we transcribed the focus group data within 24 h after the interviews were completed, and the data were analyzed by YW and YC researchers (both of whom were certified in qualitative research), respectively, according to the Colaizzi Seven-Step Method [ 34 ], which includes: (1) transcribing the audio recordings into text promptly and reading them over and over again; (2) excerpting the statements that are closely related to the theme of the study; (3) coding the recurring and meaningful ideas; (4) pooling the coded ideas; (5) writing detailed and missing descriptions; (6) identifying similar ideas and sublimating the thematic concepts; and (7) returning the results to the interviewees for verification and validation. In cases of disagreement, a third researcher (XL) was involved. Considering the cultural nature of the language, our analyses were conducted after transcribing into Chinese and discussing it repeatedly, organizing, reading, summarizing, coding, and re-reading the data, and finally conducting a summary of the themes in Chinese, after which we translated the Chinese themes into English.

Ethical considerations

The study protocol was approved by the Ethics Committee of Peking University People’s Hospital (2018THB145). Our study adhered to the principles of the Helsinki Declaration, and all participants provided their informed consent by signing a consent form. In the qualitative portion of the study, the research team members did not directly interact with the interviewees. Instead, we engaged a researcher from the hospital’s education office who informed the participants about the voluntary nature of their participation and assured them that there would be no negative consequences for their work. Additionally, the participants were informed that no personal information would be disclosed and that all data would be collected anonymously. The audio recordings of the interviews were transcribed within 24 h and subsequently destroyed within 3 weeks after obtaining confirmation from the interviewees.

Demographic result

185 new nurses were enrolled in the trial, with the intervention group’s mean age at 21.97 ± 1.58 years and the control group’s mean age at 22.13 ± 1.63 years. A bachelor’s degree or higher had been earned by 47.1% of the intervention group and by 47.8% of the control group. The results indicated that there was no statistically significant difference ( P  > 0.05) between the intervention group and the control group when comparing general variables such as gender, educational level, and department. The baseline data of the two groups were comparable, as shown in Table  2 .

Quantitative results

There was no discernible difference between the two groups’ comparable CIRN results at the beginning of the training. Following 2 years of training, the BOPPPS-CBL group’s overall CIRN scores were compared with those of the TLM group, and the differences between the two groups’ CIRN scores were statistically significant ( t  = 8.240, P  < 0.05). Additionally, the analysis of the two groups’ CIRN scores before and after the training revealed that both groups’ CIRN scores increased (TLM group: t  = 5.661, P  < 0.01; BOPPPS-CBL group: t  = 7.148, P  < 0.01) after the standardized training (Table  3 ).

The theoretical knowledge examination scores of the BOPPPS-CBL group were significantly higher than those of the TLM group (79.36 ± 10.27 vs. 70.25 ± 9.31, t  = 6.201, P  < 0.01), and the difference was statistically significant ( P  < 0.05) (Table  4 ).

Comparing Table  3 with Table  5 , the results indicate that the CIRN questionnaire comprised seven dimensions. Significant differences were observed between the two groups in the dimensions of critical thinking and research, clinical care, professional development, and educational consultation ( P  < 0.05). However, no differences were found in the dimensions of leadership, interpersonal relationships, ethical, or legal practice.

Qualitative results

Twenty-four nurses participated in focus group interviews to investigate their experiences with BOPPPS-CBL training. The resultant data was analyzed, and three major themes and six subthemes were identified. The first theme, role facilitation, was characterized by the stimulation of interest in learning, affirmation of the benefits of the training, and clarification of the orientation of clinical nurses. The second theme, forming new thinking about solving clinical problems, encompassed learning to analyze clinical issues from diverse perspectives, improving communication skills, expanding cognitive abilities, and fostering teamwork. The third theme focused on suggestions for improvement (Table  6 ).

Of the nurses interviewed, 87.6% (21/24) expressed that the BOPPPS-CBL training had numerous benefits, asserting that the training enhanced their understanding of the role and orientation of clinical nurses. Each participant played a fundamental role in constructing knowledge during case study processing, which focused on self-directed learning and developed problem-solving skills in nursing practice through multidimensional analysis and discussion of a range of cases. Furthermore, participants reported that group peer discussions facilitated learning, enhanced their enjoyment of inquiry-based self-directed learning, and increased engagement.

Respondents also conveyed enhanced confidence and proficiency in adopting proactive communication practices, examining clinical issues from diverse viewpoints, and thinking innovatively. These acquired skills were subsequently applied in fostering effective communication between healthcare professionals and patients, influencing patient clinical decision-making, and optimizing care delivery. Participants affirmed that the BOPPPS-CBL training played a pivotal role in amalgamating theoretical medical knowledge with practical clinical care, thereby bridging the gap between the two domains. It facilitated the cultivation of a novel perspective on clinical challenges and equipped them with the ability to utilize mind maps for problem analysis and clinical decision-making. This integrative approach not only instilled a sense of accomplishment but also underscored the value they contributed to their clinical practice.

However, other nurses suggested ideas to improve the training, such as providing more teaching resources and adopting deeper teaching approaches. They also expressed hope that visualization technologies could be utilized in the future to create immersive settings, while online learning resources could be made available for trainees to watch and learn from repeatedly.

In this study, an explanatory mixed-methods approach was employed to analyze quantitative and qualitative data to validate the effectiveness of BOPPPS-CBL training. The quantitative and qualitative results were complementary to each other. In the first part of the study, the quantitative research supported the research hypothesis that BOPPPS-CBL training was effective in improving the core competencies and theoretical knowledge scores of newly recruited nurses. In the second part of the study, the qualitative study affirmed the benefits of BOPPPS-CBL on the clinical roles and thinking skills of newly recruited nurses, which increased their enthusiasm and enjoyment of independent learning and affirmed the effectiveness of the training method.

In the first part of the study, we examined alterations in the core competencies of nurses before and after their participation in the TLM and BOPPPS-CBL groups. Both groups exhibited an elevation in CIRN scores post-training, indicating that both training programs enhanced nurses’ competencies, which was consistent with Burgess’ findings [ 35 ]. However, the quantitative study revealed that the training in the BOPPPS-CBL group was particularly effective, leading to a greater enhancement in nurses’ core competencies and theoretical knowledge compared to the TLM group. This difference can be attributed to several factors. First, BOPPPS was grounded in constructivist and humanistic learning theories [ 23 ], while CBL constituted an application of social cognitive theory [ 36 ]. BOPPPS-CBL, as an innovative medical teaching approach, emphasized the autonomy of nurses’ learning abilities, prioritized nurse participation and feedback, and facilitated the transformation of nurses from passive recipients of external stimuli and recipients of indoctrination to active information processors and meaning constructors. This transformative process encouraged the mastery, internalization, and absorption of knowledge. These outcomes align with the findings of Xue et al.‘s research [ 37 ]. In contrast, the TLM model, being teacher-centered and heavily reliant on the curriculum, led to passive knowledge reception by students with limited autonomous learning capabilities [ 38 ].

Moreover, in the BOPPPS-CBL training, the learning process was compartmentalized into organic modules through the application of BOPPPS, with each module thoroughly engaging and motivating the nurse [ 23 ]. Simultaneously incorporating the benefits of CBL, the training proceeded by building on real-life cases. In Bridge-in, to attract the nurses’ attention and stimulate their interest, the objective was to make it clear the goal of this learning and the direction of teaching. In the BOPPPS-CBL training, participatory learning, repeated exchanges and collisions among peers, grounded in the case and their individual knowledge, enhanced nurses’ initiative, knowledge acquisition, and core competencies. These outcomes align with the findings of previous studies [ 39 ].

In the second phase of the study, we analyzed nurses’ experiences with participating in BOPPPS-CBL training. Most nurses affirmed the effectiveness of BOPPPS-CBL training. Nurses indicated that various forms of CBL methods heightened their enjoyment of learning, fostered a positive collaborative learning atmosphere, and enhanced their comprehension abilities. Consistent with previous research, diverse learning formats increase student engagement and intrinsic motivation [ 40 ]. In the current study, nurses provided feedback on encountering real-life cases from clinical practice, which conveyed responsibility and pressure, compelling them to clarify their roles as competent clinical nurses. Previous research has indicated that feedback from teachers contributes to student growth [ 41 ]. In our study, we discovered that, in addition to teacher feedback helping nurses identify knowledge construction issues when dealing with clinical problems, discussions among peers also expanded nurses’ thinking, allowed them to draw on their peers’ strengths, and aided in their personal development. This may be related to the philosophical concept of “self-cultivation” advocated in Chinese Confucian culture, promoting a learning attitude that emphasizes humility, continuous learning from others, and avoiding arrogance and impatience [ 42 ]. Interestingly, nurses also shared changes in their clinical thinking as BOPPPS-CBL training challenged their inherent learning and thinking patterns, such as rote learning, passive knowledge acceptance, and memorization. This stimulation prompted them to actively explore and analyze clinical issues from multiple perspectives, enhancing their ability to collaborate and communicate with healthcare professionals, patients, and peers in problem-solving. This may be linked to the Confucian cultural background, where the “middle ground” principle in education emphasizes the cultivation of positive interpersonal relationships and the importance of collaborative cooperation [ 43 ]. In addition, 87.65% of the nurses in the focus groups indicated that they strongly preferred and supported the promotion of the BOPPPS-CBL training model, which was consistent with previous studies [ 39 ]. However, some nurses also indicated that if future BOPPPS-CBL training could leverage intelligent visual aids, immersive scenario simulations, and additional online learning resources, facilitating repeated viewing after training, it would further enhance their ability to apply the learning to clinical practice.

Our research findings indicate that quantitative research confirmed the efficiency of BOPPPS-CBL training, while qualitative research investigated the fundamental variables and underlying motivations that contribute to its effectiveness. The qualitative findings supplemented and confirmed the benefits of BOPPPS-CBL training revealed in the quantitative analysis. The training, which was based on the BOPPPS closed-loop instructional process model with nurses at its core, grounded in real-life experiences, and guided through group discussions, proved effective in immersing nurses in clinical environments. This approach facilitated the cultivation of clinical practice skills and the learning of core competencies, ultimately benefiting newly recruited nurses.

Limitations and implications

This was the first time that BOPPPS-CBL was used in China in the education and training of new nurses. This study design combines the capabilities of quantitative and qualitative research by first evaluating the effectiveness of comparing TLM and BOPPPS-CBL through a quantitative research design and then refining it through interviews with nurses in the BOPPPS-CBL group. The findings of this study demonstrate that it was successful and beneficial. Our study, however, had several limitations. Firstly, because this was a classroom experiment, it was impossible to control all of the confounding factors of the training effect, especially when both the intervention and control groups were training. Given the 2-year training interval and the intervention group training during the COVID-19 epidemic, one must consider the influence of external environmental factors arising from the epidemic on the results. Secondly, the predominant inclusion of female subjects raises uncertainty about the generalizability of the results to the male nurse population in China. Thirdly, we did not compare the differences between the BOPPPS model and CBL, a consideration that could be addressed in a future study comparing BOPPPS, CBL, and TLM. Moreover, our investigation into the effect on nurses’ core competencies relied on self-assessment questionnaires, which, to some extent, might not truly and objectively reflect nurses’ core competencies. In the future, utilizing an objective assessment tool could provide a more accurate evaluation of the efficacy of BOPPPS-CBL training in enhancing nurses’ core competencies. Fourthly, this study only examined the effects at the conclusion of the standardized training for new nurses, without assessing the long-term effects. Longitudinal studies based on BOPPPS-CBL training programs could be conducted in the future to explore how the core competencies developed by nurses manifest in long-term clinical practice. Finally, this study exclusively focused on the effect within one hospital; future large-scale multicenter validation studies could be undertaken in different regions and hospital levels.

Conclusions

This study showed that the BOPPPS-CBL model was more effective than TLM, and the core competencies and theoretical knowledge of nurses in the BOPPPS-CBL group increased significantly. Focus group nurses also confirmed the benefits of BOPPPS-CBL training in terms of role enhancement and clinical decision-making thinking. The BOPPPS-CBL model is an effective pedagogical model for the standardized training and education of new nurses. More research in multicenter studies incorporating smart teaching tools is needed to validate the effectiveness of the model in other contexts. In addition, the model may provide new ideas for researchers or clinical education administrators in other countries when developing continuing education training programs for nurses.

Data availability

Our research data is related to the personal identity information of nurses. If the data sets analyzed during the study are used publicly, there is a risk of the personal privacy disclosure of nurses. Therefore, we declare that the data will not be disclosed. If there is a strong demand, please send a request to the corresponding author (XD L, [email protected]).

Cai. Transition of newly graduated nurses in China: an evaluation study. Nurse Educ Pract. 2021;29:12–6. https://doi.org/10.1016/j.nepr.2020.102951

Article   Google Scholar  

Aimei W, Xiaohong W, Hongxia DU, Xiaoli YU, Xiaorong LUAN. Path analysis of safety behavior status and related influencing factors of newly recruited nurses. Chin J Nurs. 2022;57(03):318–25. https://doi.org/10.3761/j.issn.0254-1769

Xianfeng D, Yilan L, Jiaohua Y, et al. Analysis of 99 nursing errors made by novice nurses: the countermeasures. J Nurs Sci. 2013;28(03):54–5. https://doi.org/10.3870/hlxzz

Bradbury-Jones C, Clark M. Globalisation and global health: issues for nursing. Nurs Stand. 2017;31(39):54–63. https://doi.org/10.7748/ns.2017.e10797

Article   PubMed   Google Scholar  

Salvage J, White J. Our future is global: nursing leadership and global health. Rev Latino-Am Enfermagem. 2020;28:e3339. https://doi.org/10.1590/1518-8345.4542.3339

Fernet C, Trepanier SG, Demers M, Austin S. Motivational pathways of occupational and organizational turnover intention among newly registered nurses in Canada. Nurs Outlook. 2017;65:444–54. https://doi.org/10.1016/j.outlook.2017.05.008

Austin S, Fernet C, Trepanier SG, Lavoie-Tremblay M. Fatigue in new registered nurses: a 12-month cross-lagged analysis of its association with work motivation, engagement, sickness absence and turnover intention. J Nurs Manag. 2020;28:606–14. https://doi.org/10.1111/jonm.12962

Sun B, Fu L, Yan C, Wang Y, Fan L. Quality of work life and work engagement among nurses with standardized training: the mediating role of burnout and career identity. Nurse Educ Pract. 2022;58:103276. https://doi.org/10.1016/j.nepr.2021.103276

Ashton KS. The orientation period: essential for new registered nurses’ adaptation. Nurs Sci Q. 2015;28:142–50. https://doi.org/10.1177/0894318415571600

Alexander M, Runciman P. ICN Framework of Competencies for the generalist nurse: report of the development process and consultation: International Council of Nurse. 2003:1–51.

American Association of College of Nurses: National Competency Standards for the Registered Nurse.[2005.5.13]. (2023.06.18). https://www.nursingworld.org/organizational-programs/

Wang Z, Deng B, Wei Y. Conceptual analysis of core nursing competencies. Chin J Nurs. 2017;47:562–4.

Google Scholar  

Whitehead B, Owen P, Holmes D, Beddingham E, Simmons M, Henshaw L, Barton M, Walker C. Supporting newly qualified nurses in the UK: a systematic literature review. Nurse Educ Today. 2013;33:370–7. https://doi.org/10.1016/j.nedt.2013.01.009

Aggar C, Bloomfield J, Thomas TH, Gordon CJ. Australia’s first transition to professional practice in primary care program for graduate registered nurses: a pilot study. BMC Nurs. 2017;16:14. https://doi.org/10.1186/s12912-017-0207-5

Article   PubMed   PubMed Central   Google Scholar  

Jiang X, Liu D. Clinical nurses’ standardization training status and needs. Chin Nurs Managment. 2012;12:50–2.

Jiao J, Xu K, Ying Q, Li Z, Wang W, Wu Y. Construction of the evaluation system for the effects of standardized training of new nurses. Chin J Nurs. 2019;54:1285–90. https://doi.org/10.3761/j.issn.0254-1769.2019.09.001

Wu X. Development and thinking of new nurses’ standardized training in china. Chinese Journal of Modern Nursing. 2017;23:593–597. https://doi.org/10.3760/cma.j.issn.1674-2907.2017.05.001

Shang X, Lin Z, Xie X, Fang S, Chen L, Gu Z, Wang R. Establishment of standardized training system for new nurses based on post competency. Chin Nurs Res. 2019;33:1334–7. https://doi.org/10.12102/j.issn.1009-6493.2019.08.012

Ma X, Ma X, Li L, Luo X, Zhang H, Liu Y. Effect of blended learning with BOPPPS model on Chinese student outcomes and perceptions in an introduction course of health services management. Adv Physiol Educ. 2021;45:409–17. https://doi.org/10.1152/advan.00180.2020

Wang S, Xu X, Li F, Fan H, Zhao E, Bai J. Effects of modified BOPPPS-based SPOC and flipped class on 5th-year undergraduate oral histopathology learning in China during COVID-19. BMC Med Educ. 2021;21:540. https://doi.org/10.1186/s12909-021-02980-6

Shih W, Tsai C. Effect of flipped classroom with BOPPPS model on learners’ learning outcomes and perceptions in a business etiquette course. Asia-Pacific Educ Researcher. 2020;29:257–68. https://doi.org/10.1007/s40299-019-00475-z

Yang Z. Psychological health course teaching mode based on students’ high-order thinking ability development. Int J Emerg Technol Learn. 2019;14:101–12. https://doi.org/10.3991/ijet.v14i04.10111

Article   ADS   Google Scholar  

Chen L, Tang XJ, Chen XK, Ke N, Liu Q. Effect of the BOPPPS model combined with case-based learning versus lecture-based learning on ophthalmology education for five-year paediatric undergraduates in Southwest China. BMC Med Educ. 2022;22:437. https://doi.org/10.1186/s12909-022-03514-4

Yang Y, You J, Wu J, Hu C, Shao L. The effect of microteaching combined with the BOPPPS model on dental materials education for predoctoral dental students. J Dent Educ. 2019;83:567–74. https://doi.org/10.21815/JDE.019.068

Cen XY, Hua Y, Niu S, Yu T. Application of case-based learning in medical student education: a meta-analysis. Eur Rev Med Pharmacol Sci. 2021;25:3173–81. https://doi.org/10.26355/eurrev_202104_25726

Hasamnis AA, Arya A, Patil SS. Case-based learning: our experience in clinical pharmacology teaching. J Pharm Bioallied Sci. 2019;11:187–9. https://doi.org/10.4103/jpbs.JPBS_135_18

Thistlethwaite JE, Davies D, Ekeocha S, Kidd JM, MacDougall C, Matthews P, Purkis J, Clay D. The effectiveness of case-based learning in health professional education. A BEME systematic review: BEME Guide 23. Med Teach. 2012;34:e421–44. https://doi.org/10.3109/0142159X.2012.680939

McLean SF. Case-based learning and its application in medical and health-care fields_ a review of worldwide literature. J Med Educ Curric Dev. 2016;3:39–49.

Wang H, Xuan J, Liu L, Shen X, Xiong Y. Problem-based learning and case-based learning in dental education. Ann Transl Med. 2021;9:1137. https://doi.org/10.21037/atm-21-165

Greswell J. A concise introduction to mixed methods research. SAGE; 2014.

Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38:195–200. https://doi.org/10.1016/s0020-7489(00)00044-4

Article   CAS   PubMed   Google Scholar  

Liu M, Yin L, Ma M, Luo S. Confirmatory factor analysis of competency inventory for registered nurse. Chin J Nurs. 2008;43:204–6.

Zhang W, Gao W. Application status and prospect of core competence scale for registered nurses in China. Hainan Med J. 2021;32:1751–4. https://doi.org/10.3969/j.issn.1003-6350.2021.13.034

Morley J. Phenomenology in nursing studies: new perspectives - commentary. Int J Nurs Stud. 2019;93:163–7. https://doi.org/10.1016/j.ijnurstu.2019.02.002

Burgess A, van Diggele C, Roberts C, Mellis C. Key tips for teaching in the clinical setting. Bmc Med Educ. 2020;20:463. https://doi.org/10.1186/s12909-020-02283-2

Yan X, Zhu Y, Fang L, Ding P, Fang S, Zhou J, Wang J. Enhancing medical education in respiratory diseases: efficacy of a 3d printing, problem-based, and case-based learning approach. Bmc Med Educ. 2023;23:512. https://doi.org/10.1186/s12909-023-04508-6

Xue P, Wang X, Zhang C, Guo J, Zhang Y, Liu L. Construction of core competency index system for newly recruited nurses. Chin Nurs Res. 2018;32:2062–5.

Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE guide 131. Med Teach. 2020;42:846–54. https://doi.org/10.1080/0142159X.2020.1755030

Li Z, Cai X, Zhou K, Qin J, Zhang J, Yang Q, Yan F. Effects of BOPPPS combined with TBL in surgical nursing for nursing undergraduates: a mixed-method study. BMC Nurs. 2023;22:133. https://doi.org/10.1186/s12912-023-01281-1

Kao M, Yuan Y, Wang Y. The study on designed gamified mobile learning model to assess students’ learning outcome of accounting education. Heliyon. 2023;9:e13409. https://doi.org/10.1016/j.heliyon.2023.e13409

Hu K, Ma RJ, Ma C, Zheng QK, Sun ZG. Comparison of the BOPPPS model and traditional instructional approaches in thoracic surgery education. BMC Med Educ. 2022;22:447. https://doi.org/10.1186/s12909-022-03526-0

Qiao CX. Exploring the feasibility and path of integrating confucian culture into civic and political education. Educ Sci Forum. 2023:11–5.

Meng R. Research on the application of confucian culture in ideological and political education for college students. Educ Teach Forum. 2023:172–5.

Download references

Acknowledgements

This research received financial support from the Research and Development Fund of Peking University People’s Hospital (RDE 2018-03) and the Education and Teaching Research Fund of Peking University Medical School (2021YB19).

Author information

Yongli Wang and Yiqian Chen contributed equally to this manuscript and should be regarded as co-first authors.

Authors and Affiliations

Peking University People’s Hospital, No. 11 Xizhimen South Street, Xicheng Dist, 100044, Beijing, China

Yongli Wang, Ling Wang, Wen Wang, Xiangyan Kong & Xiaodan Li

Nursing School of Peking University, 100191, Beijing, China

Yiqian Chen

You can also search for this author in PubMed   Google Scholar

Contributions

Study design: YLW, YQC, XDL; acquisition of data: LW, XYK, WW; analysis: YLW, YQC, XDL, LW, XYK; drafting of the article: YLW, YQC, WW, XDL.

Corresponding author

Correspondence to Xiaodan Li .

Ethics declarations

Ethics approval and consent to participate.

This study arose from an education project (Peking University People’s Hospital Research and Development Fund [RDE 2018-03]), which received ethical clearance from the Ethics Committee of Peking University People’s Hospital(2018THB145). Our study was compliant with the Helsinki Declaration, and all participants signed informed consent.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wang, Y., Chen, Y., Wang, L. et al. Assessment of the effectiveness of the BOPPPS model combined with case-based learning on nursing residency education for newly recruited nurses in China: a mixed methods study. BMC Med Educ 24 , 215 (2024). https://doi.org/10.1186/s12909-024-05202-x

Download citation

Received : 24 September 2022

Accepted : 20 February 2024

Published : 01 March 2024

DOI : https://doi.org/10.1186/s12909-024-05202-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• BOPPPS model
• Case‑based learning
• Core competence
• Mixed methods

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]