why use comparative case study

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

We're Hiring!
Help Center

Comparative Case Studies: An Innovative Approach

What is a case study and what is it good for? In this article, we argue for a new approach—the comparative case study approach—that attends simultaneously to macro, meso, and micro dimensions of case-based research. The approach engages two logics of comparison: first, the more common compare and contrast; and second, a 'tracing across' sites or scales. As we explicate our approach, we also contrast it to traditional case study research. We contend that new approaches are necessitated by conceptual shifts in the social sciences, specifically in relation to culture, context, space, place, and comparison itself. We propose that comparative case studies should attend to three axes: horizontal, vertical, and transversal comparison. We conclude by arguing that this revision has the potential to strengthen and enhance case study research in Comparative and International Education, clarifying the unique contributions of qualitative research.

Related Papers

Lesley Bartlett

Higher Education Quarterly

Anna Kosmützky , Terhi Nokkala

Abstract Finding the balance between adequately describing the uniqueness of the context of studied phenomena and maintaining sufficient common ground for comparability and analytical generalization has widely been recognized as a key challenge in international comparative research. Methodological reflections on how to adequately cover context and comparability have extensively been discussed for quantitative survey or secondary data research. In addition, most recently, promising methodological considerations for qualitative comparative research have been suggested in comparative fields related to higher education. The article's aim is to connect this discussion to comparative higher education research. Thus, the article discusses recent advancements in the methodology of qualitative international comparative research, connects them to older analytical methods that have been used within the field in the 1960s and 1970s, and demonstrates their analytical value based on their application to a qualitative small-N case study on research groups in diverse organizational contexts in three country contexts.

John C Weidman

This is the inaugural volume in the PSCIE (Pittsburgh Studies in Comparative and International Education) Series which expands on the life work of University of Pittsburgh professor Rolland G. Paulston (1929-2006). Recognized as a stalwart in the field of comparative and international education, Paulston's most widely recognized contribution is social cartography. He demonstrated that mapping comparative, international, and development education is no easy task and, depending on the perspective of the mapper, there may be multiple cartographies to chart. This collection of nineteen essays and research studies is a festschrift celebrating and developing Robert Paulston's scholarship in comparative, international, and development education (CIDE). Considering key international education issues, national education systems, and social and educational theories, essays in this volume explore and go beyond Paulston's seminal works in social cartography. Organized into three sec...

Ben Hawbaker , Candace Jones , Brooke Boren , Reut Livne-Tarandach

Qualitative researchers utilize comparative and case-based methods to develop theory through elaboration or abduction. They pursue research in intermediate fields where some but not all relevant constructs are known (Edmonson & McManus, 2007). When cases and comparisons move beyond a few, it threatens researchers with information overload. Qualitative Comparative Analysis (QCA) is a novel method of analysis that is appropriate for larger case or comparative studies and provides a flexible tool for theory elaboration and abduction. Building on recently published exemplars from organizational research, we illuminate three key benefits of QCA: (1) allows researchers to examine cases as wholes, effectively addressing the complexity of action embedded in organizational phenomena; (2) provides indicators of whether results are reliable and valid so qualitative researchers, and others, can assess their findings within a study and across studies; and (3) explores potentially overlooked connections between qualitative and quantitative research.

Eleanor Knott

This course focuses on how to design and conduct small-n case study and comparative research. Thinking outside of students' areas of interest and specialisms and topics, students will be encouraged to develop the concepts and comparative frameworks that underpin these phenomena. In other words, students will begin to develop their research topics as cases of something. The course covers questions of design and methods of case study research, from single-n to small-n case studies including discussions of process tracing and Mill's methods. The course addresses both the theoretical and methodological discussions that underpin research design as well as the practical questions of how to conduct case study research, including gathering, assessing and using evidence. Examples from the fields of comparative politics, IR, development studies, sociology and European studies will be used throughout the lectures and seminars.

Reut Livne-Tarandach , Candace Jones

Qualitative researchers utilize comparative and case-based methods to develop theory through elaboration or abduction. They pursue research in intermediate fields where some but not all relevant constructs are known (Edmonson & McManus, 2007). When cases and comparisons move beyond a few, it threatens researchers with information overload. Qualitative Comparative Analysis (QCA) is a novel method of analysis that is appropriate for larger case and comparative studies and provides a flexible tool for theory elaboration and abduction. Building on recently published exemplars from organizational research, we illuminate three key benefits of QCA: (1) allows researchers to examine cases as wholes, effectively addressing the complexity of action embedded in organizational phenomena; (2) provides indicators of whether results are reliable and valid so qualitative researchers, and others, can assess their findings within a study and across studies; and (3) explores potentially overlooked connections between qualitative and quantitative research.

Bedrettin Yazan

Case study methodology has long been a contested terrain in social sciences research which is characterized by varying, sometimes opposing, approaches espoused by many research methodologists. Despite being one of the most frequently used qualitative research methodologies in educational research, the methodologists do not have a full consensus on the design and implementation of case study, which hampers its full evolution. Focusing on the landmark works of three prominent methodologists, namely Robert Yin, Sharan Merriam, Robert Stake, I attempt to scrutinize the areas where their perspectives diverge, converge and complement one another in varying dimensions of case study research. I aim to help the emerging researchers in the field of education familiarize themselves with the diverse views regarding case study that lead to a vast array of techniques and strategies, out of which they can come up with a combined perspective which best serves their research purpose.

The SAGE Handbook of Qualitative Data Analysis

Monika Palmberger

KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie

Markus Siewert

This article presents the case study as a type of qualitative research. Its aim is to give a detailed description of a case study-its definition, some classifications, and several advantages and disadvantages-in order to provide a better understanding of this widely used type of qualitative approac h. In comparison to other types of qualitative research, case studies have been little understood both from a methodological point of view, where disagreements exist about whether case studies should be considered a research method or a research type, and from a content point of view, where there are ambiguities regarding what should be considered a case or research subject. A great emphasis is placed on the disadvantages of case studies, where we try to refute some of the criticisms concerning case studies, particularly in comparison to quantitative research approaches.

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

Dr. BABOUCARR NJIE

johnpatrick gonzales

Romina Costa , Stephanie M Hall

Gerald LeTendre

Quality & Quantity

Sofia Pagliarin

Edgar Ojeda

A. Biba Rebolj

Philosophy of the Social Sciences

Attilia Ruzzene

Erasmus Journal for Philosophy and Economics

International Journal of Research & Method in Education

Linda Graham

IOSR Journals publish within 3 days

Anna Kosmützky

Current Issues in Comparative Education

Handbook of Qualitative Research Methods for International Business

Paul Matthyssens

Teachers College Record

Merli Tamtik

Anju Chhetri

Frances Vavrus , Lesley Bartlett

Oxford Research Encyclopedia of Education

Veritas: The Academic Journal of St Clements Education Group

Mohamed A Eno , abderrazak dammak

Qualitative Inquiry, vol. 12, no. 2, pp. 219-245

Bent Flyvbjerg

COMPARE - journal of comparative and international education

Karen Evans

Martin Oller Alonso , Daniel Barredo Ibáñez

Working Papers

Ipsita Sapra

Article contents

Comparative case study research.

Lesley Bartlett Lesley Bartlett University of Wisconsin–Madison
and Frances Vavrus Frances Vavrus University of Minnesota
https://doi.org/10.1093/acrefore/9780190264093.013.343
Published online: 26 March 2019

Case studies in the field of education often eschew comparison. However, when scholars forego comparison, they are missing an important opportunity to bolster case studies’ theoretical generalizability. Scholars must examine how disparate epistemologies lead to distinct kinds of qualitative research and different notions of comparison. Expanded notions of comparison include not only the usual logic of contrast or juxtaposition but also a logic of tracing, in order to embrace approaches to comparison that are coherent with critical, constructivist, and interpretive qualitative traditions. Finally, comparative case study researchers consider three axes of comparison : the vertical, which pays attention across levels or scales, from the local through the regional, state, federal, and global; the horizontal, which examines how similar phenomena or policies unfold in distinct locations that are socially produced; and the transversal, which compares over time.

comparative case studies
case study research
comparative case study approach
epistemology

You do not currently have access to this article

Please login to access the full content.

Access to the full content requires a subscription

Printed from Oxford Research Encyclopedias, Education. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 18 June 2024

Cookie Policy
Privacy Policy
Legal Notice
Accessibility
[81.177.182.174]
81.177.182.174

Character limit 500 /500

Breadcrumbs Section. Click here to navigate to respective pages.

Rethinking Case Study Research

DOI link for Rethinking Case Study Research

Get Citation

Comparative case studies are an effective qualitative tool for researching the impact of policy and practice in various fields of social research, including education. Developed in response to the inadequacy of traditional case study approaches, comparative case studies are highly effective because of their ability to synthesize information across time and space. In Rethinking Case Study Research: A Comparative Approach , the authors describe, explain, and illustrate the horizontal, vertical, and transversal axes of comparative case studies in order to help readers develop their own comparative case study research designs. In six concise chapters, two experts employ geographically distinct case studies—from Tanzania to Guatemala to the U.S.—to show how this innovative approach applies to the operation of policy and practice across multiple social fields. With examples and activities from anthropology, development studies, and policy studies, this volume is written for researchers, especially graduate students, in the fields of education and the interpretive social sciences.

Chapter 1 | 26 pages, follow the inquiry: an introduction, chapter 2 | 24 pages, case studies: an overview, chapter 3 | 22 pages, horizontal comparison, chapter 4 | 19 pages, vertical comparison, chapter 5 | 21 pages, tracing the transversal, chapter 6 | 16 pages, follow the inquiry: reflections on comparative case study research.

Privacy Policy
Terms & Conditions
Cookie Policy
Taylor & Francis Online
Taylor & Francis Group
Students/Researchers
Librarians/Institutions

Connect with us

case selection and the comparative method: introducing the case selector

Published: 14 August 2017
Volume 17 , pages 422–436, ( 2018 )

Cite this article

timothy prescott 1 &
brian r. urlacher 1

678 Accesses

4 Citations

5 Altmetric

Explore all metrics

We introduce a web application, the Case Selector ( http://und.edu/faculty/brian.urlacher ), that facilitates comparative case study research designs by creating an exhaustive comparison of cases from a dataset on the dependent, independent, and control variables specified by the user. This application was created to aid in systematic and transparent case selection so that researchers can better address the charge that cases are ‘cherry picked.’ An examination of case selection in a prominent study of rebel behaviour in civil war is then used to illustrate different applications of the Case Selector.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

What is Qualitative in Qualitative Research

Criteria for Good Qualitative Research: A Comprehensive Review

A new criterion for assessing discriminant validity in variance-based structural equation modeling.

Ahmed, A. and Sil, R. (2012) ‘When Multi-method Research Subverts Methodological Pluralism—or, Why We Still Need Single-Method Research’, Perspectives on Politics 10(4): 935–953.

Article Google Scholar

Asher, H. (2016) Polling and the Public: What every citizen should know , Washington, DC: CQ Press.

Google Scholar

Balcells, L. (2010) ‘Rivalry and Revenge: Violence Against Civilians in Conventional Civil Wars’, International Studies Quarterly 54(2): 291–313.

Brady, H. E. and Collier, D. (eds) (2004) Rethinking Social Inquiry: Diverse Tools, Shared Standards , Lanhan, MD: Rowman and Littlefield.

Dafoe, A. and Kelsey, N. (2014) ‘Observing the capitalist peace: Examining market-mediated signaling and other mechanisms’, Journal of Peace Research 51(5): 619–633.

DeRouen Jr, K., Ferguson, M. J., Norton, S., Park, Y. H., Lea, J., and Streat-Bartlett, A. (2010) ‘Civil War Peace Agreement Implementation and State Capacity’, Journal of Peace Research 47(3): 333–346.

Dogan, M. (2009) ‘Strategies in Comparative Sociology’, New Frontiers in Comparative Sociology , Leiden, Netherlands: Brill, pp. 13–44.

Durkheim, E. (1982) [1895] The Rules of Sociological Method , New York, NY: Free Press.

Book Google Scholar

Eck, K and Hultman, L. (2007) ‘One-Sided Violence Against Civilians in War: Insights from New Fatality Data’, Journal of Peace Research 44(2): 233–246.

Fearon, J. D. and Laitin, D. D. (2008) ‘Integrating Qualitative and Quantitative Methods’, The Oxford Handbook of Political Methodology , New York, NY: Oxford University Press. pp. 756–776.

Freedman, D. A. (2008) ‘Do the Ns Justify the Means?’, Qualitative & Multi - Method Research 6(2): 4–6.

George, A. L. and Bennett, A. (2004) Case Studies and Theory Development in the Social Sciences , Cambridge, MA: The MIT Press.

Gerring, J. and Cojocaru, L. (2016) ‘Selecting Cases for Intensive Analysis: A Diversity of Goals and Methods’, Sociological Methods & Research 45(3): 392–423.

Gerring, J. (2001) Social Science Methodology: A Criterial Framework, Cambridge, UK: University of Cambridge Press.

Gerring, J. (2004) ‘What Is a Case Study and What Is It Good for?’, American Political Science Review 98(2): 341–354.

Gerring, J. (2007) Case Study Research: Principles and Practices , Cambridge, UK: Cambridge University Press.

Glynn, A. N., and Ichino, N. (2016) ‘Increasing Inferential Leverage in the Comparative Method Placebo Tests in Small-n Research’, Sociological Methods & Research 45(3): 598–629.

Iacus, S. M., King, G., Porro, G., and Katz, J. N. (2012) ‘Causal Inference Without Balance Checking: Coarsened Exact Matching’, Political Analysis 20(1): 1–24.

Kohli, A., Evans, P., Katzenstein, P. J., Przeworski, A., Rudolph, S. H., Scott, J. C., and Skocpol, T. (1995) ‘The Role of Theory in Comparative Politics: A Symposium’, World Politics 48(1), 1–49.

Kalyvas, S. N. 2006. The Logic of Violence in Civil War , Cambridge, UK: Cambridge University Press.

King, G., Keohane, R. O., and Verba, S. (1994) Designing Social Inquiry: Scientific Inference in Qualitative Research , Princeton, NJ: Princeton University Press.

Kuperman, A. J. (2004) ‘Is Partition Really the Only Hope? Reconciling Contradictory Findings About Ethnic Civil Wars’, Security Studies 13(4): 314–349.

Lijphart, A. (1971) ‘Comparative Politics and Comparative Method, American Political Science Review 65(3): 682–698.

Maoz, Z. (2002) ‘Case Study Methodology in International Studies: From Storytelling to Hypothesis Testing’, Evaluating Methodology in International Studies , Ann Arbor, MI: University of Michigan Press, pp. 161–186.

Mahoney, J. (2010) ‘After KKV: The New Methodology of Qualitative Research’, World Politics , 62(1): 120–147.

Mill, J. S. (1872) A System of Logic . London, England: Longmans, Green, Reader, and Dyer.

Nielsen, Richard A. 2016. Case Selection via Matching. Sociological Methods & Research 45(3): 569 - 597.

Peters, B. G. (1998) Comparative politics: Theory and methods , Washington Square, NY: New York University Press.

Przeworski, A. and Teune, H. (1970) The Logic of Comparative Social Inquiry , New York, NY: Wiley-Interscience.

Ragin, C. C. (2014) The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies . Berkley, CA: University of California Press.

Ragin, C. C., Berg-Schlosser, D., and de Meur, G. (1996) ‘Political Methodology: Qualitative Methods’, A New Handbook of Political Science , New York, NY: Oxford University Press, pp. 749–769.

Sambanis, N. (2004a) ‘What is Civil War? Conceptual and Empirical Complexities of an Operational Definition’, Journal of Conflict Resolution 48(6): 814–858.

Sambanis, N. (2004b) ‘Using Case Studies to Expand Economic Models of Civil War’, Perspectives on Politics 2(2): 259–279.

Seawright, J. and Gerring, J. (2008) ‘Case Selection Techniques in Case Study Research: A Menu of Qualitative and Quantitative Options’, Political Research Quarterly 61(2): 294–308.

Schrodt, P. A. (2014) ‘Seven Deadly Sins of Contemporary Quantitative Political Analysis’, Journal of Peace Research 51(2): 287–300.

Slater, D. and Ziblatt, D. (2013) ‘The Enduring Indispensability of the Controlled Comparison’, Comparative Political Studies 46(10): 1301–1327.

Snyder, R. (2001) ‘Scaling Down: The Subnational Comparative Method’, Studies in Comparative International Development 36(1): 93–110.

Tarrow, S. (2010) ‘The Strategy of Paired Comparison: Toward a Theory of Practice’, Comparative Political Studies 43(2): 230–259.

Weinstein, J. M. (2007) Inside Rebellion: The Politics of Insurgent Violence , Cambridge, UK: Cambridge University Press.

Wood, R. M. (2010) ‘Rebel Capability and Strategic Violence Against Civilians’, Journal of Peace Research 47(5): 601–614.

Yang, Z., Matsumura, Y. Kuwata, S., Kusuoka, H., and Takeda, H. (2003) ‘Similar Case Retrieval from the Database of Laboratory Test Results’, Journal of Medical Systems 27(3): 271–281.

Yin, R. K. (2003) Case Study Research: Design and Method , Thousand Oaks, CA: Sage Publications, Inc.

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their insightful comments and feedback over the course of the review processes. This project has been significantly improved by their suggestions. The authors have also agreed to provide access to the Case Selector through their faculty webpages at their affiliated institutions.

Author information

Authors and affiliations.

Political Science Department, University of North Dakota, 293 Centennial Drive, Grand Forks, ND, USA

timothy prescott & brian r. urlacher

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to brian r. urlacher .

Rights and permissions

Reprints and permissions

About this article

prescott, t., urlacher, b.r. case selection and the comparative method: introducing the case selector. Eur Polit Sci 17 , 422–436 (2018). https://doi.org/10.1057/s41304-017-0128-5

Download citation

Published : 14 August 2017

Issue Date : September 2018

DOI : https://doi.org/10.1057/s41304-017-0128-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

comparative method
case selection
qualitative methods
Find a journal
Publish with us
Track your research

Evidence & Evaluation
Evaluation guidance
Impact evaluation with small cohorts (evaluation guidance)
Get started with small n evaluation

Comparative Case Study

Share content.

A comparative case study (CCS) is defined as ‘the systematic comparison of two or more data points (“cases”) obtained through use of the case study method’ (Kaarbo and Beasley 1999, p. 372). A case may be a participant, an intervention site, a programme or a policy. Case studies have a long history in the social sciences, yet for a long time, they were treated with scepticism (Harrison et al. 2017). The advent of grounded theory in the 1960s led to a revival in the use of case-based approaches. From the early 1980s, the increase in case study research in the field of political sciences led to the integration of formal, statistical and narrative methods, as well as the use of empirical case selection and causal inference (George and Bennett 2005), which contributed to its methodological advancement. Now, as Harrison and colleagues (2017) note, CCS:

“Has grown in sophistication and is viewed as a valid form of inquiry to explore a broad scope of complex issues, particularly when human behavior and social interactions are central to understanding topics of interest.”

It is claimed that CCS can be applied to detect causal attribution and contribution when the use of a comparison or control group is not feasible (or not preferred). Comparing cases enables evaluators to tackle causal inference through assessing regularity (patterns) and/or by excluding other plausible explanations. In practical terms, CCS involves proposing, analysing and synthesising patterns (similarities and differences) across cases that share common objectives.

What is involved?

Goodrick (2014) outlines the steps to be taken in undertaking CCS.

Key evaluation questions and the purpose of the evaluation: The evaluator should explicitly articulate the adequacy and purpose of using CCS (guided by the evaluation questions) and define the primary interests. Formulating key evaluation questions allows the selection of appropriate cases to be used in the analysis.

Propositions based on the Theory of Change: Theories and hypotheses that are to be explored should be derived from the Theory of Change (or, alternatively, from previous research around the initiative, existing policy or programme documentation).

Case selection: Advocates for CCS approaches claim an important distinction between case-oriented small n studies and (most typically large n) statistical/variable-focused approaches in terms of the process of selecting cases: in case-based methods, selection is iterative and cannot rely on convenience and accessibility. ‘Initial’ cases should be identified in advance, but case selection may continue as evidence is gathered. Various case-selection criteria can be identified depending on the analytic purpose (Vogt et al., 2011). These may include:

Very similar cases
Very different cases
Typical or representative cases
Extreme or unusual cases
Deviant or unexpected cases
Influential or emblematic cases

Identify how evidence will be collected, analysed and synthesised: CCS often applies mixed methods.

Test alternative explanations for outcomes: Following the identification of patterns and relationships, the evaluator may wish to test the established propositions in a follow-up exploratory phase. Approaches applied here may involve triangulation, selecting contradicting cases or using an analytical approach such as Qualitative Comparative Analysis (QCA). Download a Comparative Case Study here Download a longer briefing on Comparative Case Studies here

Useful resources

A webinar shared by Better Evaluation with an overview of using CCS for evaluation.

A short overview describing how to apply CCS for evaluation:

Goodrick, D. (2014). Comparative Case Studies, Methodological Briefs: Impact Evaluation 9 , UNICEF Office of Research, Florence.

An extensively used book that provides a comprehensive critical examination of case-based methods:

Byrne, D. and Ragin, C. C. (2009). The Sage handbook of case-based methods . Sage Publications.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

Use of Comparative Case Study Methodology for US Public Health Policy Analysis: A Review

Affiliation.

1 Department of Nutrition and Food Studies, Montclair State University, Montclair, New Jersey (Dr Dinour); Department of Community Health and Social Sciences, City University of New York School of Public Health and Health Policy, New York, New York (Ms Kwan and Dr Freudenberg).
PMID: 27798523
DOI: 10.1097/PHH.0000000000000406

Context: There is growing recognition that policies influence population health, highlighting the need for evidence to inform future policy development and reform.

Objectives: This review describes how comparative case study methodology has been applied to public health policy research and discusses the methodology's potential to contribute to this evidence.

Methods: English-language, peer-reviewed articles published between 1995 and 2012 were sought from 4 databases. Articles were included if they described comparative case studies addressing US public health policy. Two researchers independently assessed the 20 articles meeting review criteria.

Main outcome measures: Case-related characteristics and research design tactics utilized to minimize threats to reliability and validity, such as the use of multiple sources of evidence and a case study protocol, were extracted from each article.

Results: Although comparative case study methodology has been used to analyze a range of public health policies at all stages and levels, articles reported an average use of only 3.65 (out of 10) research design tactics.

Conclusion: By expanding the use of accepted research design tactics, public health policy researchers can contribute to expanding the evidence needed to advance health-promoting policies.

PubMed Disclaimer

Publication types

Search in MeSH

LinkOut - more resources

Full text sources.

Ingenta plc
Ovid Technologies, Inc.
Wolters Kluwer

Other Literature Sources

scite Smart Citations

Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries

A new study reveals the need for benchmarking and public evaluations of AI tools in law.

Artificial intelligence (AI) tools are rapidly transforming the practice of law. Nearly three quarters of lawyers plan on using generative AI for their work, from sifting through mountains of case law to drafting contracts to reviewing documents to writing legal memoranda. But are these tools reliable enough for real-world use?

Large language models have a documented tendency to “hallucinate,” or make up false information. In one highly-publicized case, a New York lawyer faced sanctions for citing ChatGPT-invented fictional cases in a legal brief; many similar cases have since been reported. And our previous study of general-purpose chatbots found that they hallucinated between 58% and 82% of the time on legal queries, highlighting the risks of incorporating AI into legal practice. In his 2023 annual report on the judiciary , Chief Justice Roberts took note and warned lawyers of hallucinations.

Across all areas of industry, retrieval-augmented generation (RAG) is seen and promoted as the solution for reducing hallucinations in domain-specific contexts. Relying on RAG, leading legal research services have released AI-powered legal research products that they claim “avoid” hallucinations and guarantee “hallucination-free” legal citations. RAG systems promise to deliver more accurate and trustworthy legal information by integrating a language model with a database of legal documents. Yet providers have not provided hard evidence for such claims or even precisely defined “hallucination,” making it difficult to assess their real-world reliability.

AI-Driven Legal Research Tools Still Hallucinate

In a new preprint study by Stanford RegLab and HAI researchers, we put the claims of two providers, LexisNexis (creator of Lexis+ AI) and Thomson Reuters (creator of Westlaw AI-Assisted Research and Ask Practical Law AI)), to the test. We show that their tools do reduce errors compared to general-purpose AI models like GPT-4. That is a substantial improvement and we document instances where these tools provide sound and detailed legal research. But even these bespoke legal AI tools still hallucinate an alarming amount of the time: the Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time, while Westlaw’s AI-Assisted Research hallucinated more than 34% of the time.

Read the full study, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools

To conduct our study, we manually constructed a pre-registered dataset of over 200 open-ended legal queries, which we designed to probe various aspects of these systems’ performance.

Broadly, we investigated (1) general research questions (questions about doctrine, case holdings, or the bar exam); (2) jurisdiction or time-specific questions (questions about circuit splits and recent changes in the law); (3) false premise questions (questions that mimic a user having a mistaken understanding of the law); and (4) factual recall questions (questions about simple, objective facts that require no legal interpretation). These questions are designed to reflect a wide range of query types and to constitute a challenging real-world dataset of exactly the kinds of queries where legal research may be needed the most.

comparison of hallucinated and incomplete responses

Figure 1: Comparison of hallucinated (red) and incomplete (yellow) answers across generative legal research tools.

These systems can hallucinate in one of two ways. First, a response from an AI tool might just be incorrect —it describes the law incorrectly or makes a factual error. Second, a response might be misgrounded —the AI tool describes the law correctly, but cites a source which does not in fact support its claims.

Given the critical importance of authoritative sources in legal research and writing, the second type of hallucination may be even more pernicious than the outright invention of legal cases. A citation might be “hallucination-free” in the narrowest sense that the citation exists , but that is not the only thing that matters. The core promise of legal AI is that it can streamline the time-consuming process of identifying relevant legal sources. If a tool provides sources that seem authoritative but are in reality irrelevant or contradictory, users could be misled. They may place undue trust in the tool's output, potentially leading to erroneous legal judgments and conclusions.

Figure 2: Top left: Example of a hallucinated response by Westlaw's AI-Assisted Research product. The system makes up a statement in the Federal Rules of Bankruptcy Procedure that does not exist (and Kontrick v. Ryan, 540 U.S. 443 (2004) held that a closely related bankruptcy deadline provision was not jurisdictional). Top right: Example of a hallucinated response by LexisNexis's Lexis+ AI. Casey and its undue burden standard were overruled by the Supreme Court in Dobbs v. Jackson Women's Health Organization, 597 U.S. 215 (2022); the correct answer is rational basis review. Bottom left: Example of a hallucinated response by Thomson Reuters's Ask Practical Law AI. The system fails to correct the user’s mistaken premise—in reality, Justice Ginsburg joined the Court's landmark decision legalizing same-sex marriage—and instead provides additional false information about the case. Bottom right: Example of a hallucinated response from GPT-4, which generates a statutory provision that has not been codified.

RAG Is Not a Panacea

a chart showing an overview of the retrieval-augmentation generation (RAG) process.

Figure 3: An overview of the retrieval-augmentation generation (RAG) process. Given a user query (left), the typical process consists of two steps: (1) retrieval (middle), where the query is embedded with natural language processing and a retrieval system takes embeddings and retrieves the relevant documents (e.g., Supreme Court cases); and (2) generation (right), where the retrieved texts are fed to the language model to generate the response to the user query. Any of the subsidiary steps may introduce error and hallucinations into the generated response. (Icons are courtesy of FlatIcon.)

Under the hood, these new legal AI tools use retrieval-augmented generation (RAG) to produce their results, a method that many tout as a potential solution to the hallucination problem. In theory, RAG allows a system to first retrieve the relevant source material and then use it to generate the correct response. In practice, however, we show that even RAG systems are not hallucination-free.

We identify several challenges that are particularly unique to RAG-based legal AI systems, causing hallucinations.

First, legal retrieval is hard. As any lawyer knows, finding the appropriate (or best) authority can be no easy task. Unlike other domains, the law is not entirely composed of verifiable facts —instead, law is built up over time by judges writing opinions . This makes identifying the set of documents that definitively answer a query difficult, and sometimes hallucinations occur for the simple reason that the system’s retrieval mechanism fails.

Second, even when retrieval occurs, the document that is retrieved can be an inapplicable authority. In the American legal system, rules and precedents differ across jurisdictions and time periods; documents that might be relevant on their face due to semantic similarity to a query may actually be inapposite for idiosyncratic reasons that are unique to the law. Thus, we also observe hallucinations occurring when these RAG systems fail to identify the truly binding authority. This is particularly problematic as areas where the law is in flux is precisely where legal research matters the most. One system, for instance, incorrectly recited the “undue burden” standard for abortion restrictions as good law, which was overturned in Dobbs (see Figure 2).

Third, sycophancy—the tendency of AI to agree with the user's incorrect assumptions—also poses unique risks in legal settings. One system, for instance, naively agreed with the question’s premise that Justice Ginsburg dissented in Obergefell , the case establishing a right to same-sex marriage, and answered that she did so based on her views on international copyright. (Justice Ginsburg did not dissent in Obergefell and, no, the case had nothing to do with copyright.) Notwithstanding that answer, here there are optimistic results. Our tests showed that both systems generally navigated queries based on false premises effectively. But when these systems do agree with erroneous user assertions, the implications can be severe—particularly for those hoping to use these tools to increase access to justice among pro se and under-resourced litigants.

Responsible Integration of AI Into Law Requires Transparency

Ultimately, our results highlight the need for rigorous and transparent benchmarking of legal AI tools. Unlike other domains, the use of AI in law remains alarmingly opaque: the tools we study provide no systematic access, publish few details about their models, and report no evaluation results at all.

This opacity makes it exceedingly challenging for lawyers to procure and acquire AI products. The large law firm Paul Weiss spent nearly a year and a half testing a product, and did not develop “hard metrics” because checking the AI system was so involved that it “makes any efficiency gains difficult to measure.” The absence of rigorous evaluation metrics makes responsible adoption difficult, especially for practitioners that are less resourced than Paul Weiss.

The lack of transparency also threatens lawyers’ ability to comply with ethical and professional responsibility requirements. The bar associations of California , New York , and Florida have all recently released guidance on lawyers’ duty of supervision over work products created with AI tools. And as of May 2024, more than 25 federal judges have issued standing orders instructing attorneys to disclose or monitor the use of AI in their courtrooms.

Without access to evaluations of the specific tools and transparency around their design, lawyers may find it impossible to comply with these responsibilities. Alternatively, given the high rate of hallucinations, lawyers may find themselves having to verify each and every proposition and citation provided by these tools, undercutting the stated efficiency gains that legal AI tools are supposed to provide.

Our study is meant in no way to single out LexisNexis and Thomson Reuters. Their products are far from the only legal AI tools that stand in need of transparency—a slew of startups offer similar products and have made similar claims , but they are available on even more restricted bases, making it even more difficult to assess how they function.

Based on what we know, legal hallucinations have not been solved.The legal profession should turn to public benchmarking and rigorous evaluations of AI tools.

This story was updated on Thursday, May 30, 2024, to include analysis of a third AI tool, Westlaw’s AI-Assisted Research.

Paper authors: Varun Magesh is a research fellow at Stanford RegLab. Faiz Surani is a research fellow at Stanford RegLab. Matthew Dahl is a joint JD/PhD student in political science at Yale University and graduate student affiliate of Stanford RegLab. Mirac Suzgun is a joint JD/PhD student in computer science at Stanford University and a graduate student fellow at Stanford RegLab. Christopher D. Manning is Thomas M. Siebel Professor of Machine Learning, Professor of Linguistics and Computer Science, and Senior Fellow at HAI. Daniel E. Ho is the William Benjamin Scott and Luna M. Scott Professor of Law, Professor of Political Science, Professor of Computer Science (by courtesy), Senior Fellow at HAI, Senior Fellow at SIEPR, and Director of the RegLab at Stanford University.