best case study synonyms

• Daily Crossword
• Word Puzzle
• Word Finder
• Word of the Day
• Synonym of the Day
• Word of the Year
• Language stories
• All featured
• Gender and sexuality
• All pop culture
• Writing hub
• Grammar essentials
• Commonly confused
• All writing tips
• Pop culture
• Writing tips

Advertisement

noun as in case history

Strongest match

• medical history

Weak matches

• medical record
• psychiatric history

Discover More

Example sentences.

In a case study from Metric Theory, Target Impression Share bidding, the total cost per click increased with both mobile and desktop devices.

It would also become the subject of a fair number of business school case studies.

Not just blog posts, you can also share other resources like case studies, podcast episodes, and webinars via Instagram Stories.

They become the architecture for a case study of Flint, expressed in a more personal and poetic way than a straightforward investigation could.

The Creek Fire was a case study in the challenge facing today’s fire analysts, who are trying to predict the movements of fires that are far more severe than those seen just a decade ago.

A case study would be your Twilight director Catherine Hardwicke.

A good case study for the minority superhero problem is Luke Cage.

He was asked to review a case study out of Lebanon that had cited his work.

Instead, now we have a political science case-study proving how political fortunes can shift and change at warp speed.

One interesting case study is Sir Arthur Evans, the original excavator and “restorer” of the Minoan palace of Knossos on Crete.

As this is a case study, it should be said that my first mistake was in discrediting my early religious experience.

The author of a recent case study of democracy in a frontier county commented on the need for this kind of investigation.

How could a case study of Virginia during this period illustrate these developments?

Related Words

Words related to case study are not direct synonyms, but are associated with the word case study . Browse related words to learn more about word associations.

noun as in record of what happened

On this page you'll find 6 synonyms, antonyms, and words related to case study, such as: medical history, anamnesis, dossier, medical record, and psychiatric history.

From Roget's 21st Century Thesaurus, Third Edition Copyright © 2013 by the Philip Lief Group.

What Is Another Way to Say “Case Study”?

January 1, 2024

Linda Brown

Looking for synonyms for case study ? We’ve got you covered!

Here’s a list of other ways to say case study .

• Examination
• Investigation
• Research study
• Detailed study
• Exploration
• Observational study
• Case report
• Case analysis

Want to learn how to say case study professionally? Keep reading for examples and use cases.

1. Analysis

“Analysis” is used when a detailed examination of the elements or structure of something is conducted. It’s appropriate in scientific, business, and academic contexts. Example: The team conducted an analysis of market trends for the new product launch.

“Report” refers to a formal account or statement describing the findings of an investigation or research. It’s used in professional, academic, and governmental contexts. Example: She compiled a comprehensive report on the company’s financial health over the past year.

3. Examination

“Examination” is used when referring to a detailed inspection or analysis of a subject or phenomenon. It’s suitable in medical, academic, and technical fields. Example: The doctor conducted a thorough examination of the patient’s case history.

4. Investigation

“Investigation” implies a systematic or formal inquiry to discover facts or information. It’s commonly used in law enforcement, scientific research, and journalism. Example: The environmental agency launched an investigation into the cause of the pollution.

“Profile” refers to an analysis or description of a particular thing or person. It’s often used in journalism, psychology, and marketing. Example: The magazine published a detailed profile of the innovative tech startup.

“Survey” is used for a comprehensive examination or review of a particular area or subject. It’s suitable in research, social sciences, and market analysis. Example: The city conducted a survey to understand the housing needs of its residents.

7. Research Study

“Research study” refers to a detailed and systematic examination of a subject to discover new information or reach new understandings. It’s used in academic and scientific contexts. Example: The research study provided new insights into the effects of climate change on agriculture.

“Inquiry” implies a formal investigation or examination. It’s often used in academic research, legal contexts, and public policy. Example: The committee launched an inquiry into the effectiveness of the new health policy.

9. Assessment

“Assessment” is the evaluation or analysis of the nature, quality, or ability of someone or something. It’s used in educational, professional, and healthcare settings. Example: The consultant did an assessment of the project’s risks and opportunities.

“Review” involves a formal assessment or examination of a subject or situation. It’s appropriate in academic, professional, and critical analysis contexts. Example: The team conducted a review of the existing literature on renewable energy technologies.

11. Detailed Study

“Detailed study” refers to an in-depth examination and analysis of a subject. It’s used in contexts where comprehensive understanding is required. Example: A detailed study of the region’s history revealed unknown cultural influences.

12. Exploration

“Exploration” is used to describe a thorough analysis or discussion of a subject, often in a more open or investigative manner. It’s suitable in scientific and academic research. Example: His book is an exploration into the psychological impacts of social media.

13. Observational Study

“Observational study” refers to research where the investigator observes subjects without manipulation. It’s commonly used in social sciences and medicine. Example: The observational study focused on children’s behavior in different learning environments.

14. Case Report

“Case report” is a detailed report of the symptoms, signs, diagnosis, treatment, and follow-up of an individual patient. It’s used in medical and clinical contexts. Example: The doctor published a case report on the rare genetic disorder.

15. Case Analysis

“Case analysis” involves a detailed examination of a case in order to understand its various aspects. It’s often used in business, law, and academic settings. Example: The business school students performed a case analysis of the company’s strategic turnaround.

Related posts:

• What Is Another Way to Say “Easy to Understand”?
• What Is Another Way to Say “Analogy”?
• What Is Another Way to Say “Better Understanding”?
• What Is Another Way to Say “Carrying Out”?
• What Is Another Way to Say “Excited for What’s to Come”?
• What Is Another Way to Say “As You Can See”?
• What Is Another Way to Say “Bad Mood”?
• What Is Another Way to Say “Look At”?
• What Is Another Way to Say “Calm Down”?
• What Is Another Way to Say “Deep Thinking”?

[email protected]

We help you expand your vocabulary and improve your understanding of the English language.

© SynonymPro

• ABBREVIATIONS
• BIOGRAPHIES
• CALCULATORS
• CONVERSIONS
• DEFINITIONS

  Vocabulary      

What is another word for case study ?

Synonyms for case study case study, this thesaurus page includes all potential synonyms, words with the same meaning and similar terms for the word case study ., princeton's wordnet.

• case study noun

a careful study of some social unit (as a corporation or division within a corporation) that attempts to determine what factors led to its success or failure

a detailed analysis of a person or group from a social or psychological or medical point of view

Matched Categories

• Corporation

Concise Medical Dictionary, by Joseph C Segen, MD Rate these synonyms: 2.2 / 5 votes

Synonyms: Epidemiology Anecdotal report, anecdote, single case report

How to pronounce case study?

How to say case study in sign language, how to use case study in a sentence.

Josh Holmes :

For those asking, this is my response to West Virginia Roy Moore :' This clown is a walking, talking case study for the limitation of a prison's ability to rehabilitate,'.

Sam Goodman :

The Hong Kong BNO scheme is an interesting case study of what can happen if there is political will, there are 12 welcome centers across the country and a really good support package which costs relatively little, including help with English language. And most importantly they just didn’t politicize it. All this has meant that 144,000 Hong Kongers have come here with little to no fuss, integrated quickly and there have been minimal issues.

Houston Astros :

I think I’m kind of a case study on this one.

Tesoro Corp :

We agree on the critical importance of continually learning from incidents and improving the safety of our operations, and inaccuracies in the case study do not detract from our resolve to learn from these incidents.

Alba Pasini :

This case study is really important, since it testifies that a medical approach to maternal morbidity actually existed during the Lombard period, despite the rejection of the scientific progress which denoted all the Early Middle Age, also, it shows two rare findings, since post-mortem fetal extrusion is a quite rare phenomenon( especially in archaeological specimens), while only a few examples of trepanation are known for the European Early Middle Age.

Use the citation below to add these synonyms to your bibliography:

Style: MLA Chicago APA

"case study." Synonyms.com. STANDS4 LLC, 2024. Web. 19 Jun 2024. < https://www.synonyms.com/synonym/case+study >.

Discuss these case study synonyms with the community:

Report Comment

We're doing our best to make sure our content is useful, accurate and safe. If by any chance you spot an inappropriate comment while navigating through our website please use this form to let us know, and we'll take care of it shortly.

You need to be logged in to favorite .

Create a new account.

Your name: * Required

Your email address: * Required

Pick a user name: * Required

Username: * Required

Password: * Required

Forgot your password?    Retrieve it

Are we missing a good synonym for case study ?

Image credit, the web's largest resource for, synonyms & antonyms, a member of the stands4 network, image or illustration of.

Free, no signup required :

Add to chrome, add to firefox, browse synonyms.com, are you a human thesaurus, which of the following words is not a synonym of the others, nearby & related entries:.

• case load noun
• case officer noun
• case reporter
• case sensitive
• case shot noun
• case-by-case adj
• case-fatality
• case-fatality proportion noun

Alternative searches for case study :

• Search for case study on Amazon

Case study synonyms

What is another word for case study .

• dossier record of what happened
• medical history record of what happened
• anamnesis record of what happened
• medical record record of what happened
• psychiatric history record of what happened
• case history
• instance example
• pilot study
• examination
• investigation

• Rhymes with Case-study
• Case-study in a sentence

Study Past Tense

The past tense of Study is studied.

noun. ['ˈstʌdi'] a detailed critical inspection.

• examination
• indiscipline
• Romanticism
• studie (Middle English (1100-1500))
• estudier (Old French (842-ca. 1400))

Rhymes with Case Study

Sentences with case-study.

1. Noun Phrase For these questions, a case study is provided for analysis. 2. Noun Phrase This might be a real-world scenario or a case ," aria-label="Link to study ,"> study , depending on the specific course requirements.

verb. ['ˈstʌdi'] consider in detail and subject to an analysis in order to discover essential features or meaning.

• check up on
• investigate

verb. ['ˈstʌdi'] be a student; follow a course of study; be enrolled at an institute of learning.

noun. ['ˈstʌdi'] applying the mind to learning and understanding a subject (especially by reading).

• acquisition

verb. ['ˈstʌdi'] give careful consideration to.

• contemplate

verb. ['ˈstʌdi'] be a student of a certain subject.

noun. ['ˈkeɪs'] an occurrence of something.

• mortification
• natural event
• humiliation
• postmeridian
• antemeridian
• cas (Middle English (1100-1500))
• cas (Old English (ca. 450-1100))

noun. ['ˈkeɪs'] a special set of circumstances.

noun. ['ˈkeɪs'] a comprehensive term for any proceeding in a court of law whereby an individual seeks a legal remedy.

• class action
• countersuit
• bastardy proceeding
• proceedings
• class-action suit
• criminal suit
• legal proceeding
• paternity suit
• motionlessness
• stand still
• distributed

noun. ['ˈkeɪs'] the actual state of things.

download a flashcard

Look up a word, learn it forever.

/ˌkeɪ(s) ˈstʌdi/, /keɪs ˈstʌdi/.

Other forms: case studies

• noun a detailed analysis of a person or group from a social or psychological or medical point of view see more see less type of: analysis an investigation of the component parts of a whole and their relations in making up the whole
• noun a careful study of some social unit (as a corporation or division within a corporation) that attempts to determine what factors led to its success or failure see more see less type of: report , study , written report a written document describing the findings of some individual or group

Vocabulary lists containing case study

Get your neurons firing with this list of words related to psychology. You'll learn about parts of the brain, cognition and memory, psychiatry, phobias and psychological disorders, and more. This list will blow your mind!

Sign up now (it’s free!)

Whether you’re a teacher or a learner, vocabulary.com can put you or your class on the path to systematic vocabulary improvement..

Search for synonyms and antonyms

case study > synonyms

• 150 Synonyms
• more  
• 207 Related

104 expressions ×

List search

• ??? - shows 3-letter terms
• a??e - 4-letter terms starting with 'a' and ending with 'e'
• a* - terms starting with 'a'
• *ment - terms ending with 'ment'

If nothing is found, then alternative search will try to find the terms that:

• start with searched query
• sound like it
• similar to it

Support us by sharing "synonyms for case study" page!

Share on Facebook | Twitter

• Searched With

How to Use case study in a Sentence

• The company's recent history is a case study in bad management.

Some of these examples are programmatically compiled from various online sources to illustrate current usage of the word 'case study.' Any opinions expressed in the examples do not represent those of Merriam-Webster or its editors. Send us feedback about these examples.

Can you solve 4 words at once?

Word of the day.

See Definitions and Examples »

Get Word of the Day daily email!

• Skip to main content
• Skip to FDA Search
• Skip to in this section menu
• Skip to footer links

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

U.S. Food and Drug Administration

•   Search
•   Menu
• News & Events
• FDA Newsroom
• Press Announcements

FDA Roundup: June 18, 2024

FDA News Release

Today, the U.S. Food and Drug Administration is providing an at-a-glance summary of news from around the agency: 

• Today, the FDA announced a request for input about patient safety associated with certain medical software functions excluded from the medical device definition. This input will help the FDA develop the 2024 report on the risks and benefits to health of non-device software functions. Please submit comments under docket number FDA-2018-N-1910 at www.regulations.gov by July 18, 2024.
• The FDA recently issued a warning letter to Dollar Tree Inc., as an additional step following the investigation of lead and chromium in apple cinnamon fruit puree pouches this past fall that was prompted by findings of elevated blood lead levels in children. The FDA issued the warning letter to Dollar Tree, Inc., because, at the time of the recall, the company failed to adequately remove recalled WanaBana apple cinnamon fruit puree pouches from its store shelves. The FDA has asked the company to respond within 15 days of receipt of the warning letter stating the specific steps it has taken to address any violations and prevent the recurrence of violations or providing its reasoning and supporting information as to why the company believes it is not in violation of the law.
• for the prevention of invasive disease caused by 22 different serotypes of Streptococcus pneumoniae covered by the vaccine for individuals 18 years of age and older.
• for the prevention of pneumonia caused by 21 different serotypes of Streptococcus pneumoniae covered by the vaccine for individuals 18 years of age and older.
• On Monday, the FDA announced the first of a series of five public meetings to provide educational sessions for stakeholders who are interested in the new animal drug approval process. The annual educational conferences the FDA will host over the next four years are described in the “ Animal Drug User Fee Act Reauthorization Performance Goals and Procedures Fiscal Years 2024 Through 2028 .” The first educational conference will be held on July 17, 2024, from 9 a.m. to 5 p.m. ET. Attendees can join in-person or virtually.   
• On Monday, the FDA’s Center for Devices and Radiological Health published a new blog: The Promise of Artificial Intelligence for Improving Health Care . Artificial intelligence (AI) is rapidly changing the health care industry and holds transformative potential. Among the possibilities AI offers are significantly improving patient care and medical professional satisfaction, accelerating and advancing research in medical device development and drug discovery, and driving operational efficiency by enabling personalized treatment and streamlining health care processes. At the FDA, we know that the proper integration of AI across the health care ecosystem will be important to achieving its potential while reducing risks and challenges.
• On Monday, the American Society of Addiction Medicine issued the draft “ Clinical Practice Guideline on Benzodiazepine Tapering ,” which is now available for public comment. The development of this guideline was funded by an FDA-provided grant and in partnership with the American Academy of Family Physicians, the American Academy of Neurology, the American Academy of Physician Associates, the American Association of Medical Toxicology, the American Association of Nurse Practitioners, the American Association of Psychiatric Pharmacists, the American College of Obstetricians and Gynecologists, the American Geriatrics Society, and the American Psychiatric Association.
• On Monday, the FDA approved pembrolizumab (Keytruda, Merck) with carboplatin and paclitaxel, followed by single-agent pembrolizumab, for adult patients with primary advanced or recurrent endometrial carcinoma. Adverse reactions associated with pembrolizumab and chemotherapy were generally similar to those previously reported for pembrolizumab or chemotherapy with the exception of a higher incidence of rash. See the prescribing information for complete adverse reactions. Full prescribing information for Keytruda will be posted on Drugs@FDA . 
• On Friday, the FDA issued a Proposed Administrative Order: Amending Over-the-Counter Monograph M013: Internal Analgesic, Antipyretic, and Antirheumatic Drug Products for Over-the-Counter Human Use. The agency is issuing this FDA-initiated proposed administrative order (proposed order), to address a safety issue related to over-the-counter monograph drug products containing acetaminophen. This proposed order, if finalized, would require drug companies to add a warning to the labeling. Additional information can be found in the CDER Statement about this proposed order. 
• On Friday, the FDA published a summary of the patient listening sessions held in March 2024 on Attention-Deficit/Hyperactivity Disorder (ADHD). The goal of the sessions is to help FDA’s staff better understand patient perspectives about their diagnosis and the risks and benefits associated with stimulant and non-stimulant treatment for ADHD.
• On Friday, the FDA approved durvalumab (Imfinzi, AstraZeneca UK Limited) with carboplatin plus paclitaxel followed by single-agent durvalumab for adult patients with primary advanced or recurrent endometrial cancer that is mismatch repair deficient (dMMR). The most common adverse reactions (>25%) with durvalumab in combination with carboplatin and paclitaxel were peripheral neuropathy, musculoskeletal pain, nausea, alopecia, fatigue, abdominal pain, constipation, rash, diarrhea, vomiting, and cough. Full prescribing information for Imfinzi will be posted on Drugs@ FDA..
• On Friday, the FDA approved blinatumomab (Blincyto, Amgen Inc.) for adult and pediatric patients one month and older with CD19-positive Philadelphia chromosome-negative B-cell precursor acute lymphoblastic leukemia (Ph-negative BCP ALL) in the consolidation phase of multiphase chemotherapy. In Study E1910, the most common adverse reactions (≥20%) in the blinatumomab arm were neutropenia, thrombocytopenia, anemia, leukopenia, headache, infection, nausea, lymphopenia, diarrhea, musculoskeletal pain, and tremor. In Study 20120215, the most common adverse reactions (≥20%) in the blinatumomab arm were pyrexia, nausea, headache, rash, hypogammaglobulinemia, and anemia. Full prescribing information for Blincyto will be posted on Drugs@FDA .

Related Information

The FDA, an agency within the U.S. Department of Health and Human Services, protects the public health by assuring the safety, effectiveness, and security of human and veterinary drugs, vaccines and other biological products for human use, and medical devices. The agency also is responsible for the safety and security of our nation’s food supply, cosmetics, dietary supplements, radiation-emitting electronic products, and for regulating tobacco products.

• Health Tech
• Health Insurance
• Medical Devices
• Gene Therapy
• Neuroscience
• H5N1 Bird Flu
• Health Disparities
• Infectious Disease
• Mental Health
• Cardiovascular Disease
• Chronic Disease
• Alzheimer's
• Coercive Care
• The Obesity Revolution
• The War on Recovery
• Adam Feuerstein
• Matthew Herper
• Jennifer Adaeze Okwerekwu
• Ed Silverman
• CRISPR Tracker
• Breakthrough Device Tracker
• Generative AI Tracker
• Obesity Drug Tracker
• 2024 STAT Summit
• Wunderkinds Nomination
• STAT Madness
• STAT Brand Studio

Don't miss out

Subscribe to STAT+ today, for the best life sciences journalism in the industry

Ambitious federal study failed to curb opioid deaths, NIH announces

By Lev Facher June 16, 2024

I n 2019, amid an ever-worsening drug crisis, the federal government launched a research study with an ambitious goal: to lower opioid overdoses in participating communities by 40% using evidence-based interventions like distributing naloxone and providing access to addiction medications.

But communities that implemented the public health strategies did not see a statistically significant reduction in opioid overdose deaths, according to data published Sunday in the New England Journal of Medicine . 

advertisement

Given the study’s simple premise — that helping communities use proven strategies could help prevent deaths — the results came as a surprise. But its leaders warn against making too much of the disappointing data, citing the fast-changing drug supply and, critically, the backdrop of the Covid-19 pandemic.  

“We started this study in January of 2020, and guess what happened in March of 2020?” Redonna Chandler, the National Institute on Drug Abuse official who directed the research project. “And while our communities continued working in the background, we weren’t able to get into hospitals. We weren’t able to get into jails. We weren’t able to get into a lot of the places and spaces where we wanted to implement our evidence-based practices.” 

Related: How the U.S. is sabotaging its best tools to prevent deaths in the opioid epidemic

In fact, of the hundreds of individual interventions that communities had planned to use as part of the study, just 38% had been implemented by the start of the year that data was analyzed, according to the NEJM analysis.

The National Institutes of Health launched the initiative, known as the HEALing Communities Study — short for Helping End Addiction Long-term — in April 2018. It awarded $344 million to participating communities, using funds that Congress had appropriated for substance use research the previous year. 

“Now is the time to channel the efforts of the scientific community to deliver effective — and sustainable — solutions to this formidable public health challenge,” several top NIH officials, including then-Director Francis Collins and NIDA Director Nora Volkow, wrote in 2018 when the project kicked off. 

The study focused on 67 communities across Massachusetts, New York, Kentucky, and Ohio. Roughly half were picked to implement their overdose-prevention strategies beginning in 2020; the rest put their strategies into place beginning in 2022, following the comparison period between the two groups. 

Interventions included increasing access to medications like methadone and buprenorphine by reducing restrictions, supporting health care providers, and working with jails and prisons. It also included education surrounding opioid prescribing, and bolstering the distribution of naloxone , a medication used to reverse opioid overdoses. 

Sign up for Morning Rounds

Understand how science, health policy, and medicine shape the world every day

During the comparison period, the communities that put their strategies into place sooner experienced 47.2 opioid-related overdose deaths per 100,000 people; the communities that hadn’t begun experienced 51.7. Despite a nearly 10% lower death rate in communities that had implemented their strategies, however, the results fell short of statistical significance. 

In statements, federal health officials cast the study as at least a partial victory. While the interventions did not meaningfully reduce overdose deaths, the officials argued, they set the stage for future action and created a framework to help hard-hit communities choose new policy approaches and begin to implement them, with the hope that with more time and without Covid-19, deaths would fall.  

Volkow, the NIDA director, said that increasing use of stimulants like methamphetamine and cocaine, and the proliferation of fentanyl, mean society must “continue developing new tools and approaches” for preventing overdose deaths. Miriam Delphin-Rittmon, the administrator of the Substance Abuse and Mental Health Services Administration, said the study “recognizes there is no quick fix.” 

And in an interview, Chandler, the director of the study, stressed that the results should not challenge what research has long demonstrated: There is a “mountain of evidence,” she said, supporting the belief that tools like naloxone, medications for opioid use disorder, and safer prescribing techniques, save lives. The challenge, Chandler said, lies in implementation — not the strategies themselves. 

The study released Sunday, she said, “doesn’t negate, in any way, the evidence that suggests the strengths of those interventions.”

STAT’s coverage of chronic health issues is supported by a grant from  Bloomberg Philanthropies . Our financial supporters  are not involved in any decisions about our journalism.

About the Author Reprints

Addiction Reporter

Lev Facher covers the U.S. addiction and overdose crisis.

public health

STAT encourages you to share your voice. We welcome your commentary, criticism, and expertise on our subscriber-only platform, STAT+ Connect

To submit a correction request, please visit our Contact Us page .

Recommended

Recommended Stories

STAT Plus: Humana scores a win in a Medicare Advantage case

STAT Plus: The inside story of how Lykos’ MDMA research went awry

STAT Plus: Google’s Verily to offer GLP-1 drugs through Lightpath, its retooled chronic care app

STAT Plus: Biotech co-founded by entrepreneur Bob Langer is testing a new approach to developing obesity treatments

STAT Plus: Why a big California employer ditched Elevance for some of its health plans

This paper is in the following e-collection/theme issue:

Published on 18.6.2024 in Vol 26 (2024)

Monitoring Adverse Drug Events in Web Forums: Evaluation of a Pipeline and Use Case Study

Authors of this article:

Original Paper

• Pierre Karapetiantz 1 , PhD   ; 
• Bissan Audeh 1 , PhD   ; 
• Akram Redjdal 1 , PhD   ; 
• Théophile Tiffet 2, 3 , MD   ; 
• Cédric Bousquet 1, 2 , PhD, PharmD   ; 
• Marie-Christine Jaulent 1 , PhD  

1 Inserm, Sorbonne Université, université Paris 13, Laboratoire d’informatique médicale et d’ingénierie des connaissances en e-santé, LIMICS, F-75006, Paris, France

2 Service de santé publique et information médicale, CHU de Saint Etienne, 42000 Saint-Etienne, France

3 Institut National de la Santé et de la Recherche Médicale, Université Jean Monnet, SAnté INgéniérie BIOlogie St-Etienne, SAINBIOSE, 42270 Saint-Priest-en-Jarez, France

Corresponding Author:

Marie-Christine Jaulent, PhD

Sorbonne Université

université Paris 13, Laboratoire d’informatique médicale et d’ingénierie des connaissances en e-santé, LIMICS, F-75006

15 rue de l'école de Médecine

Paris, 75006

Phone: 33 144279108

Email: [email protected]

Background: To mitigate safety concerns, regulatory agencies must make informed decisions regarding drug usage and adverse drug events (ADEs). The primary pharmacovigilance data stem from spontaneous reports by health care professionals. However, underreporting poses a notable challenge within the current system. Explorations into alternative sources, including electronic patient records and social media, have been undertaken. Nevertheless, social media’s potential remains largely untapped in real-world scenarios.

Objective: The challenge faced by regulatory agencies in using social media is primarily attributed to the absence of suitable tools to support decision makers. An effective tool should enable access to information via a graphical user interface, presenting data in a user-friendly manner rather than in their raw form. This interface should offer various visualization options, empowering users to choose representations that best convey the data and facilitate informed decision-making. Thus, this study aims to assess the potential of integrating social media into pharmacovigilance and enhancing decision-making with this novel data source. To achieve this, our objective was to develop and assess a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. The goal was to create a user-friendly tool that enables regulatory authorities to make better-informed decisions effectively.

Methods: To enhance pharmacovigilance efforts, we have devised a pipeline comprising 4 distinct modules, each independently editable, aimed at efficiently analyzing health-related French web forums. These modules were (1) web forums’ posts extraction, (2) web forums’ posts annotation, (3) statistics and signal detection algorithm, and (4) a graphical user interface (GUI). We showcase the efficacy of the GUI through an illustrative case study involving the introduction of the new formula of Levothyrox in France. This event led to a surge in reports to the French regulatory authority.

Results: Between January 1, 2017, and February 28, 2021, a total of 2,081,296 posts were extracted from 23 French web forums. These posts contained 437,192 normalized drug-ADE couples, annotated with the Anatomical Therapeutic Chemical (ATC) Classification and Medical Dictionary for Regulatory Activities (MedDRA). The analysis of the Levothyrox new formula revealed a notable pattern. In August 2017, there was a sharp increase in posts related to this medication on social media platforms, which coincided with a substantial uptick in reports submitted by patients to the national regulatory authority during the same period.

Conclusions: We demonstrated that conducting quantitative analysis using the GUI is straightforward and requires no coding. The results aligned with prior research and also offered potential insights into drug-related matters. Our hypothesis received partial confirmation because the final users were not involved in the evaluation process. Further studies, concentrating on ergonomics and the impact on professionals within regulatory agencies, are imperative for future research endeavors. We emphasized the versatility of our approach and the seamless interoperability between different modules over the performance of individual modules. Specifically, the annotation module was integrated early in the development process and could undergo substantial enhancement by leveraging contemporary techniques rooted in the Transformers architecture. Our pipeline holds potential applications in health surveillance by regulatory agencies or pharmaceutical companies, aiding in the identification of safety concerns. Moreover, it could be used by research teams for retrospective analysis of events.

Introduction

Social media as a complementary data source for pharmacovigilance.

One primary mission of regulatory agencies such as the FDA (Food and Drug Administration) or the EMA (European Medicines Agency) is to monitor drug usage and adverse drug events (ADEs) to mitigate the risks associated with drugs within the population. This task entails analyzing diverse data sources, including clinical trials, postmarketing surveillance, spontaneous reporting systems, and published scientific literature. Despite the wealth of available data, some ADEs are not always detected promptly, largely because of underreporting. In France, for instance, underreporting was estimated to range between 78% and 99% from 1997 to 2002 [ 1 ]. To tackle this challenge, several countries have implemented systems allowing patients to report ADEs.

Additional sources for detecting ADEs have been under exploration, such as electronic patient records [ 2 - 4 ] and social media platforms [ 5 - 9 ]. While some argue that social media alone cannot serve as a primary source for signal detection [ 10 ], it can be viewed as a valuable secondary source for monitoring emerging adverse drug reactions or reinforcing signals previously identified through spontaneous reports stored in traditional pharmacovigilance databases [ 11 ]. In a prior study by the authors, patient profiles and reported ADEs found in web forums were compared with those in the French Pharmacovigilance Database (FPVD). The forums tended to represent younger patients, more women, less severe cases, and a higher incidence of psychiatric disorder–related ADEs compared with the FPVD [ 12 ]. Moreover, forums reported a greater number of unexpected ADEs. Over the past decade, several tools for evaluating social media posts have been described in the literature [ 13 ]. Specifically, effective ADE detection in social media necessitates both quantitative and qualitative analyses of data [ 14 ].

Qualitative Approach for Individual Assessment of Posts

Qualitative assessment entails evaluating whether users’ messages contain pertinent information for an assessment akin to a pharmacovigilance case report. This includes details such as the patient’s age and gender, the severity of the case, the expectedness and timeline of the adverse event, time-to-onset, dechallenge (outcome upon drug withdrawal), and rechallenge (outcome upon drug reintroduction). For instance, GlaxoSmithKline Inc. implemented the qualitative approach Insight Explorer, which facilitates the collection of extensive data for causality and quality assessment. Users can input data including personal information (eg, age range, gender) and product details (eg, name, route of administration, duration of use, dosage). This approach was adapted for the WEB-RADR (Recognizing Adverse Drug Reactions) project to manually construct a gold standard of curated patient-authored text [ 15 ].

Quantitative Approach for Monitoring Adverse Drug Events on Social Media

Quantitative evaluation involves analyzing extracted data using descriptive and analytical statistics, such as signal detection and change-point analysis. Numerous projects have been undertaken to monitor ADEs on social media. One of the earliest projects is the PREDOSE (Prescription Drug Abuse Online Surveillance and Epidemiology) project [ 5 ], which investigates the illicit use of pharmaceutical opioids reported in web forums. While the PREDOSE project showcased the potential of leveraging social media for opioid monitoring, notable limitations are the lack of deidentification and signal detection methods. MedWatcher Social, a monitoring platform for health-related web forums, Twitter, and Facebook, represents a prototype application developed in 2014 [ 16 ]. Yeleswarapu et al [ 6 ] outlined a semiautomatic pipeline that applies natural language processing (NLP) tasks to extract ADEs from MEDLINE abstracts and user comments from health-related websites. However, this pipeline was not intended for routine use.

The Domino’s interface [ 17 ], developed in 2018 by the University of Bordeaux in France and funded by the French Medicines Agency (Agence nationale de sécurité du médicament et des produits de santé [ANSM]), was designed to analyze drug misuses in health-related web forums using NLP methods and the summary of product characteristics. Initially tailored for antidepressant drugs, this tool does not primarily focus on ADE surveillance.

Another pipeline, described by Nikfarjam et al in 2019 [ 7 ], used a neural network–based named entity recognition system specifically designed for user-generated content in social media. This platform is dedicated to identifying the association of cutaneous ADEs with cancer therapy drugs. The study focused on a selection of drugs and only examined 8 ADEs.

Magge et al [ 8 ] described a pipeline aimed at the extraction and normalization of adverse drug mentions on Twitter. Their pipeline consisted of an ADE classifier designed to identify tweets mentioning an ADE, which were then mapped to a MedDRA (Medical Dictionary for Regulatory Activities Terminology) code. However, the normalization process was confined to the ADEs present in the training set. Neither Nikfarjam’s nor Magge’s pipeline provides a graphical user interface.

Some private companies also offer tools for analyzing social media for pharmacovigilance purposes. For instance, the DETECT platform was developed as part of a collaborative project in France by Kappa Santé [ 18 ]. This system enabled the labeling of posts with known controlled vocabulary concepts, and signal detection was conducted [ 19 ]. Within the scope of this project, Expert System Company implemented BIOPHARMA Navigator to extract web forum posts, while the Luxid Annotation Server provided web services for the automatic annotation of posts.

An important finding from the studies of the last decade is that while regulatory agencies have begun using data sources beyond spontaneous reports, social media has yet to be fully leveraged in real-world settings due to the immaturity of available solutions. Primarily, these solutions are essentially proofs of concept that lack scalability and are challenging for experts to evaluate routinely, primarily due to the absence of a graphical user interface to present information.

Our aim was to assess the potential of integrating social media into pharmacovigilance and enhancing decision-making with this novel data source. To achieve this, our objective was to develop and assess a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. The goal was to create a user-friendly tool that enables regulatory authorities to make better-informed decisions effectively.

This article presents the design and implementation of our pipeline dedicated to harnessing posts from social media. In addition, we showcase the use of the pipeline through a specific use case, emphasizing the importance of monitoring drugs in social media to better address patients’ expectations.

The PHARES project (Pharmacovigilance in Social Networks), funded from 2017 to 2019 by the French ANSM, aimed to develop a software suite (a pipeline) enabling pharmacovigilance users to analyze social networks, particularly messages posted on forums. The objective of the pipeline is to facilitate routine use through continuous post extraction and quantitative data analysis from web forums, specifically tailored for the French language.

The pipeline is made up of 4 modules, each referring to its own methods ( Figure 1 ):

The Scraper module, which extracts posts from forums using a previously developed tool, Vigi4Med (V4M) scraper [ 9 ], and produces a comma-separated values (CSV) file filled with the texts extracted.

The Annotation module, which extracts elements of interest from the posts and registers annotations in CSV files, with each line representing an annotation of an ADE or a drug. When a causality relationship is identified, both an ADE and a drug are annotated on the same line.

The Statistical module, which performs quantitative analysis on the annotated posts, generating numerical data, tables, or figures.

The Interface module, which supports query definition and visualization of results.

The methodology used to evaluate the PHARES pipeline involved comparing its performance with existing platforms mentioned above, in accordance with a set of criteria established with prospective PHARES users. The criteria, specific to each module, are as follows:

• General level: focus on ADEs, designed for routine usage.
• Scraper: collects all posts of a selected website, performs deidentification, allows to extract posts from web forums, and is open source.
• Statistics: the temporal evolution of posts or annotations is displayed and a change-point analysis (detecting breakpoints) is possible.
• Signal detection: allows to apply at least one signal detection method, displays the temporal evolution of the proportional reporting ratio (PRR), and allows to perform a logistic regression–based signal detection method.
• Graphical user interface: has an interface for users.

Scraper Module

V4M Scraper is an open-source tool designed for data extraction from web forums [ 9 ]. Its primary functions are optimizing scraping time, filtering out posts primarily focused on advertisements, and structuring the extracted data semantically. The module operates by taking a configuration file as input, which contains the URL of the targeted forum. The algorithm navigates through forum pages and generates resource description framework (RDF) triplets for each extracted element, allowing for potential alignment with external semantic resources. A caching mechanism has been integrated into this tool to maintain a local copy of previously visited pages, thereby avoiding redundant requests to websites for already scraped web pages, particularly in cases of errors or testing, for example. Vigi4Med V4M Scraper was customized for the PHARES project, as indicated by the red elements in Figure S1 in Multimedia Appendix 1 . The database format (Figure S2 in Multimedia Appendix 1 ) was implemented to enhance interaction with the interface. Specifically, the main scraping script was adjusted to produce a simplified tabular format (CSV) of the extracted data and to store these data in a database. This modification aims to facilitate input to the subsequent module of the pipeline (annotation). V4M Scraper was customized to enable a continuous scraping routine, wherein data extracted from web forums are automatically and regularly annotated and registered. A log file was integrated into the scraper structure to maintain a record of the last scraped element. This log file ensures that the daily routine scraping always begins from the last scraped point. An automation tool (crontab) is used to schedule the execution of the pipeline for each forum on a daily basis at a specific time.

A total of 23 public French health-related web forums were selected through a combination of Google searches and from a list of certified health websites provided by the HON Foundation, in collaboration with the French National Health Authority (HAS). The selection criteria included the requirement for websites to be hosted in France, feature a discussion board or space for sharing experiences, and have more than 10 patient contributions. Furthermore, Twitter posts are collected and analyzed by the pipeline. This is achieved using the Twitter API for data collection, followed by employing the same modules used for processing web forum posts.

Annotation Module

Entities corresponding to drugs and pathological conditions in social media were identified and annotated using an NLP pipeline [ 20 ]. Initially, conditional random fields were used to account for global dependencies [ 21 ]. Specifically, the model considers the entire sequence when making predictions for individual tokens. This approach is advantageous for entity extraction tasks, as the presence of an entity in one part of the text can influence the likelihood of other entities in the vicinity. Second, a support vector machine is used to predict the causality relationship between an entity identified as a drug and another entity identified as an ADE. The annotation method used in this module was implemented at an early stage of the pipeline’s design. Currently, the named entity recognition task of this module is undergoing revision to incorporate more recent advancements in NLP algorithms [ 22 - 26 ].

In a third step, the detected annotations were normalized using codes from the MedDRA and the Anatomical Therapeutic Classification (ATC) to ensure they were suitable for signal detection purposes.

MedDRA is an international medical hierarchical terminology comprising 5 levels used to code potential ADEs in pharmacovigilance. The highest level is the system organ class, which is further divided into high-level group terms, then into high-level terms, preferred terms (PTs), and finally lowest level terms. Typically, the PT level is used in pharmacovigilance signal detection.

The ATC classification system is a drug classification used in France for pharmacovigilance purposes. It categorizes the active ingredients of drugs based on the organ system they primarily affect. The classification comprises 5 levels: the anatomical main group (consisting of 14 main groups), the therapeutic subgroup, the therapeutic/pharmacological subgroup, the chemical/therapeutic/pharmacological subgroup, and the chemical substance. Typically, the fifth level (chemical substance) is used in pharmacovigilance signal detection.

The outputs of the annotation module are CSV files with the following variables:

• Concerning the post: forum name, post ID, and date
• Concerning the ADE: verbatim, normalized term, unified medical language system’s concept unique identifier, and MedDRA code
• Concerning the drug: verbatim, normalized term, active ingredient, and ATC code

In these CSV files, each line can consist of either an adverse event (ADE) annotation, a drug annotation, or both when a causality relationship has been identified between the drug and the ADE. Table 1 provides a sample of the database.

In a prior study, we selected posts where at least one ADE associated with 6 drugs (agomelatine, baclofen, duloxetine, exenatide, strontium ranelate, and tetrazepam) had been detected by this algorithm. A manual review revealed that among 5149 posts, 1284 (24.94%) were validated as pharmacovigilance cases [ 12 ]. The fundamental metrics used to assess the performance of the annotation module were precision (P), recall (R), and their harmonic mean F 1 -score. To calculate these metrics, it is necessary to evaluate false negatives for nonrecognition of relevant terms, false positives for irrelevant recognitions, and true positives for correct recognitions. Precision, recall, and F 1 -score are defined as follows:

Precision = (true positive)/(true positive + false positive); recall = (true positive)/(true positive + false negative); F 1 -score = (2 × precision × recall)/(precision + recall) (1)

In the “Results” section, we present a comparison of the performance of the annotation module with the performance of state-of-the-art methods [ 8 , 22 , 25 , 26 ].

a ADE: adverse event.

b MedDRA: Medical Dictionary for Regulatory Activities Terminology.

c ATC: Anatomical Therapeutic Classification.

d No data are available for this slot.

Statistical Module

This module generates general statistics and diagrams for web forums or Twitter. It provides data such as the number of annotated posts (related to the drug, the ADE, or both), the count of drug-ADE pairs identified, and the distribution of ADEs’ MedDRA-PTs. In addition, a change-point analysis method was used to detect significant changes over time in the mean number of posts mentioning the drug and ADE [ 27 ].

Besides, several statistical signal detection methods were implemented to generate potential signals. Safety signals, which provide information on adverse events that may potentially be caused by a medicine, were further evaluated by pharmacovigilance experts to determine the causal relationship between the medicine and the reported adverse event.

The statistical module implements 3 signal detection methods, including 2 well-known and frequently used disproportionality signal detection methods: the PRR [ 28 ] and the reporting odds ratio (ROR) [ 29 ]. In addition, a complementary method, a logistic regression–based signal detection method known as the class imbalanced subsampling lasso [ 30 ], was used.

PRR and ROR are akin to a relative risk and an odds ratio, respectively. However, they differ in their denominators: as the number of exposed patients is typically unknown in pharmacovigilance databases, the denominator in PRR and ROR calculations is the number of cases reported in the pharmacovigilance database.

PRR and ROR are specific to each drug-ADE pair and can be directly computed from the contingency table ( Table 2 ).

The PRR compares the proportion of an ADE among all the ADEs reported for a specific drug with the same proportion for all other drugs in the database (Equation 2). A PRR significantly greater than 1 suggests that the ADE is more frequently reported for patients taking the drug of interest, while a PRR equal to 1 suggests independence between the 2 variables.

PRR = [a/(a + b)]/[c/(c + d)] (2)

The ROR quantifies the strength of the association between drug administration and the occurrence of the ADE. It represents the ratio of the odds of drug administration when the ADE is present to the odds of drug administration when the ADE is absent (Equation 3). When the 2 events are independent, the ROR equals 1. An ROR significantly greater than 1 suggests that drug administration is associated with the presence of the ADE.

ROR = ad / bc (3)

We considered events over posts for the calculation of disproportionality statistics. If the same drug-ADE pair was identified multiple times within a post, the pair was counted as many times as it occurred in the calculation.

Disproportionality analysis has certain limitations, including the confounding effect resulting from coreported drugs and the masking effect, where the background relative reporting rate of an ADE is distorted by extensive reporting on the ADE with a specific drug or drug group. Caster et al [ 31 ] demonstrated through 2 real case examples how multivariate regression–based approaches can address these issues. Harpaz et al also suggested that logistic regression could be used for safety surveillance [ 32 ]. Initially designed for pharmacovigilance case reports, we hypothesize that they may also be applicable to posts.

The logistic regression model specifically focuses on a particular ADE or a group of ADEs. It involves creating a vector that represents the presence (1) or absence (0) of the ADE of interest in the pharmacovigilance case (in our case, in the post). Additionally, a matrix is generated to represent the administration or nonadministration of all drugs in the database by the patient (1 for administration and 0 for nonadministration). Figure S3 in Multimedia Appendix 1 illustrates an example of using logistic regression. In our case, we assumed that if a drug was annotated in the post, it was taken by the patient. The logistic regression aims to predict the probability of the presence of the ADE (ADE=1) of interest based on the presence of all ( N m ) drugs in the database (Equation 4), where X represents the distribution of the presence/absence of the drugs. The adjusted factors included only concomitant medications, as patient-related factors are often missing in web forums’ posts. Therefore, we did not need to address the impact of missing data, which should be evaluated when necessary.

ln([P(X|ADE=1)]/[P(X|ADE=0)]) = a + b1 × Drug1 + ... + bi × Drug i + .. . + bNm × Drug Nm (4)

The selection of the drugs depends on the parameter b i . If b i <0, the drug i decreases the risk of the ADE, and if b i >0, the drug i increases the risk of the ADE.

Then, 2 sets are defined:

• S 1 : set of n 1 posts with an annotation of the ADEs of interest.
• S 0 : set of n 0 posts without an annotation of the ADEs of interest.

In our case n 0 >> n 1 , indicating a significant imbalance toward posts lacking annotations of the ADEs of interest. To address this issue, we took a subsample with a more favorable ratio of posts with annotated ADEs versus those without. Additionally, to enhance result stability, we conducted multiple draws instead of just one.

In practice, we generated B subsamples. Each subsample was constructed by randomly drawing, with replacement, n 1 posts from S 1 and R posts from S 0 , where R=max(4 n 1 , 4 N m ). The choice of 4 n 1 was inspired by case-control studies, while 4 N m was included to ensure an adequate number of observations considering the multitude of predictors.

We implemented a change-point analysis method described in [ 27 ] to detect whether there was a change in the evolution over time of a chosen statistic, such as the number of a specific drug-ADE pair, the number of ADEs associated with a specific drug, or the number of drugs associated with a specific ADE. The method uses the Cumulative Sum (CUSUM) algorithm to analyze the evolution of statistics over time, comparing current values with the period mean. It identifies breakpoints by calculating the highest difference in statistical values and comparing it with random samples. The process repeats for periods before and after detected breakpoints until no more are found.

User Interface Module

The user interface module facilitates user interaction with the pipeline in a user-friendly manner. The interface comprises a dashboard divided into 2 main parts. The left dark column ( Figure 2 ) serves as a control sidebar, where users can select parameters to filter the data, including the forum, period, drug(s) according to the ATC classification, and ADE(s) according to a level in the MedDRA hierarchy. On the right side of the interface, various visualizations are available, organized into several tabs such as “Forum Statistics” and “Consultation of Posts,” with additional tabs for statistics that become active upon querying.

Before applying a specific query, the interface provides general information about the currently available data ( Figure 2 ), including the total annotated posts since 2017 (n=2,081,296) and total annotations since 2017 (n=2,454,310). In addition, a “Consultation of Tweets” tab (not visible in the figure) displays the total annotated tweets since March 2020 (n=46,153).

Furthermore, several tabs corresponding to different types of statistics, including “Forums Statistics” and “Twitter Statistics,” provide general statistics and diagrams for web forums and Twitter. Examples of these are pie charts showing forum distribution, line charts depicting the evolution of drug and ADE mentions, histograms displaying ADE distribution by system organ class, and line charts illustrating the temporal trend of posts containing the drug and an ADE, as shown in Figures 3 and 4 . The “Annotations Plot” tab displays annotations of drugs and adverse effects selected by the user, along with forum information, PTs, high-level terms, high-level group terms, dates, and hours. The “Logistic Regression” tab allows users to choose parameters for applying logistic regression. In the “Disproportionality” tab, users can choose between the PRR and ROR methods, with the time evolution of the chosen method displayed. The “Change-Point” tab enables analysis of temporal evolution, with identified breakpoints indicated. The “Consultation of Posts” and “Consultation of Tweets” tabs provide details on annotated posts/tweets, including downloadable tables. The statistical module performs calculations based on user queries, updating the interface accordingly. If multiple drugs or adverse events are selected, they are treated as new entities for analysis.

The interface was implemented using the R language and environment (R Foundation) for statistical computing and graphics [ 33 ], leveraging the Shiny package [ 34 ] for development.

Ethical Considerations

A statement by an Institutional Review Board was not required because we used only publicly available data that do not necessitate Institutional Review Board review.

This study complied with the European General Data Protection Regulation (GDPR), which has been in force since 2018 in Europe [ 35 ]. The GDPR enhances the protection of individuals by introducing the right to be informed about the processing of personal data. However, informing each user individually may be impractical. Therefore, the GDPR introduces 2 legal conditions where informed consent is not mandatory, which can be interpreted as supporting the processing of web forum posts for pharmacovigilance (Article 9): “(e) processing relates to personal data which are manifestly made public by the data subject; [. . .] (i) processing is necessary for reasons of public interest in the area of public health, such as [. . .] ensuring high standards of quality and safety of health care and of medicinal products . . ..” The GDPR also requires data processing to “not permit or no longer permits the identification of data subjects” (Article 89). Deidentification was conducted during the extraction of posts from web forums to ensure privacy [ 9 ]. User identifiers in the main RDF file were encrypted using the SHA1 algorithm [ 36 ]. The correspondence between these encrypted identifiers and the original keys is presented in RDF triplets in a separate file, referred to as the “keys file.” Therefore, the only way to retrieve the original authors’ identities is by concatenating the main RDF containing the encrypted data with the keys file, which is kept in a secured location. Moreover, all our data processing was carried out on a secured server with restricted access.

General Results About the Pipeline

The primary outcome of this study is the operational PHARES pipeline itself. Daily extraction and annotation of posts are initiated and imported into the database linked to the user interface. In this paper, the platform’s use will be demonstrated through a specific use case on the analysis of Levothyrox ADE mentions in forums (discussed later). In addition, we conducted a comparative analysis of the PHARES pipeline with the existing platforms mentioned in the “Introduction” section, based on the criteria listed in the “Methods” section.

Of the 10 identified pipelines, half were public and half were private. While 8 out of 10 focused on ADEs, only 4 were designed for routine usage. Five scrapers were open source, and all posts from considered websites were extracted by only 6 of the scrapers (with others extracting posts under certain conditions). Six scraped web forum posts, but only 3 performed deidentification. Additionally, 4 pipelines focused on the French language. A total of 6 pipelines displayed the temporal evolution of the number of posts, but only 1 conducted a change-point analysis. Signal detection methods were performed by only 4 of them, with none displaying the temporal evolution of the PRR nor a logistic regression–based method. Finally, 6 of them had an interface ( Table 3 ).

a PHARES: Pharmacovigilance in Social Networks.

b The X symbol means that the characteristic is missing and the symbol ✓ means the characteristic is fulfilled.

c ADE: adverse drug event.

d PRR: proportional reporting ratio.

e PREDOSE: Prescription Drug Abuse Online Surveillance and Epidemiology.

f ADR-PRISM: Adverse Drug Reaction from Patient Reports in Social Media.

Annotation Module’s Comparison With Up-to-Date State-of-the-Art Methods

We also compared the performance of our annotation process with those of up-to-date state-of-the-art methods ( Table 4 ).

While the annotation module demonstrated good performance for named entity recognition ( F 1 -score=0.886), it remains slightly below the state of the art. Presently, in medical texts, the best performances are achieved by Hussain et al [ 25 ] and Ding et al [ 26 ] for the named entity recognition task, and by Xia [ 22 ] for the relationship extraction task. On Twitter, known for its notably more complex data, Hussain et al [ 25 ] achieved slightly better results than our annotator, while Ding et al [ 26 ] achieved slightly worse results.

a The 2 categories are entity recognition, which is the detection of a drug or ADE mention, and relationship extraction, which is the detection of a relation between a drug and an ADE.

b PHARES: Pharmacovigilance in Social Networks.

c BERT: Bidirectional Encoder Representations from Transformer.

d Not available.

e HAMLE: Historical Awareness Multi-Level Embedding.

f BGRU: Bidirectional Gated Recurrent Unit.

g LSTM: Long-Short-Term-Memory.

Summary of the Result

From January 1, 2017, to February 28, 2021, a total of 2,081,296 posts were extracted from 23 French web forums ( Table 5 ). We obtained 713,057 normalized annotations of drugs, 1,527,004 normalized annotations of ADEs, and 437,192 annotations of normalized drug-ADE couples. The number of posts annotated with at least one normalized drug-ADE couple was equal to 125,279 (6.02%). Table 4 summarizes the number of posts extracted per forum, the publication dates, and the description of the web forum. For 1 forum, the publication dates were not available. A total of 9 were generalist health forums, 3 were specialized for parents of a young baby, 2 for families, 3 for mothers, 2 specialized in thyroid issues, 1 for pregnant women, 1 for women, 1 for parents of a teenager or for teenagers, 1 for sports persons, and 1 specialized in rare diseases.

a Not available.

Use Case: Analysis of Levothyrox ADE Mentions in Forums

To demonstrate the usage of the pipeline, we chose to focus on Levothyrox as a case study. Levothyrox is a drug prescribed in France since 1980 for hypothyroidism and circumstances where it is necessary to limit the thyroid-stimulating hormone. In 2017, a new formula of Levothyrox, differing from the 30-year-old drug at the excipient level (with lactose being replaced by mannitol and citric acid in the new formula), was marketed with widespread media coverage. In parallel, an unexpected increase in notifications of ADEs for this drug was detected. Viard et al [ 37 ] were unable to find any pharmacological rationale to explain that signal. Approximately 32,000 adverse effects were reported by patients in France in 2017, representing 42% of all the ADEs collected yearly [ 38 ]. Most of these notifications concerned the new formulation of Levothyrox and led to the “French Levothyrox crisis.” In 2017, 1664 notifications of ADEs were spontaneously reported by patients to the Pharmacovigilance Center of Nice. Among the 1544 reviewed notifications, 1372 concerned Levothyrox while only 172 concerned other drugs [ 37 ].

In this use case, the study period was from January 1, 2017, to February 28, 2021, and the drugs included were 2 drugs from the “H03AA Thyroid hormones” ATC class: “Levothyroxine sodium” and “associations of levothyroxine and liothyronine.” A total of 17 forums were selected as they included at least one post with information about these drugs. Posts were extracted, annotated, and analyzed through the pipeline from several forums ( Table 6 ). Signal detection methods were applied to an ADE chosen as it frequently appeared with Levothyrox in our data: “tiredness.” A signal can be detected when the lower bound of the 95% CI of the logarithm of the PRR is greater than 0. For logistic regression, we applied the tenth quantile. A total of 11,340 posts contained an annotation concerning the drugs of interest. Figure S4 in Multimedia Appendix 1 illustrates the source and evolution over time of these posts. Out of a total of 50,127 annotations of Levothyrox, they principally originated from the Vivre sans thyroïde forum and were mostly posted in mid-2017 ( Figure 4 , Table 6 ). The results of the statistical analysis were displayed by the user interface.

ADEs annotated with Levothyrox were mainly from system organ classes: general disorders and administration site conditions (29.6%), metabolism and nutrition disorders (11.6%), and endocrine disorders (11.4%). The PTs mostly found in association with Levothyrox are listed in Table 7 . All this information is accessible in the interface module (Figure S5 in Multimedia Appendix 1 ).

We chose the PT “tiredness” for the signal detection analysis. A total of 85,976 posts were annotated with either one of the drugs of interest or the ADE tiredness. Among them, 1841 Levothyrox-tiredness couples were found, mostly in 2017 ( Table 7 ).

Figure 5 illustrates the time evolution of the PRR for the Levothyrox-tiredness couple. Figure S6 in Multimedia Appendix 1 displays the source and evolution over time of French web forums’ posts for this couple. A signal is consistently generated throughout the period as the logarithm of the PRR is always greater than 0.

A total of 11 drugs were found to be associated with tiredness using logistic regression: paclitaxel, pegfilgrastim, Levothyrox, glatiramer acetate, escitalopram ferrous sulfate, the combination of Levothyrox and liothyronine, secukinumab, methotrexate, bismuth potassium, tetracycline, and metronidazole.

Change-point analysis was conducted on the monthly evolution of the number of Levothyrox-ADE couples detected in web forums. Six breakpoints were identified ( Figure 6 ), and 3 of them correlated with an increase in the number of ADEs found with Levothyrox on web forums. These increases occurred in August 2017 and in September and December 2018.

This use case demonstrates that the results obtained through the pipeline, particularly in the context of Levothyrox, align with findings in the literature derived from more traditional data sources such as case reports in pharmacovigilance (see the “Discussion” section). It underscores the potential of leveraging such a pipeline to monitor a drug, not only retrospectively but also in real time using social media. Consequently, PHARES has the capability to potentially uncover new signals in pharmacovigilance.

Principal Findings

To align with our objective, we implemented and evaluated a pipeline that processes data from the extraction of web forum posts to the generation of indicators and alerts within a visual and interactive environment. Through this pipeline, we demonstrated that quantitative analysis can be conducted through the interface without requiring the user to code. We discovered the feasibility of acquiring information akin to the literature regarding a drug’s ADEs, as well as unexpected ADEs and significant event dates related to a drug. This underscores the relevance and utility of such a pipeline.

A conceptual contribution of this research was the proposal of a methodology for designing a pipeline to facilitate pharmacovigilance studies on web forums. This involved describing 4 independent modules and outlining their interactions. Additionally, another contribution was the adaptation of certain pharmacovigilance analysis methods for the examination of data extracted from web forum posts. The logistic regression–based method presented in this article was originally tailored for pharmacovigilance cases to consider co-prescriptions of drugs. We have adapted it to suit the analysis of pharmacovigilance data extracted from web forum posts.

Comparison With Prior Work

The PHARES pipeline offers added value compared with previous pipelines in terms of the criteria set, which reflects an analysis of experts’ needs for routine monitoring of ADEs in social media. Unlike previous approaches, the scrapers used in PHARES routinely perform deidentification, and the inclusion of change-point analysis, the evolution of PRRs over time, and a logistic regression–based signal detection method were previously unavailable. The temporal evolution of the number of posts and a signal detection method are also seldom supported. Designed for routine usage and focused on ADEs, all posts from selected web forums are scraped and deidentified using an open-source scraper.

The period and selected web forums differed between both studies: Audeh et al [ 38 ] covered the period from January 2015 to December 2017, while our study spanned from January 2017 to February 2021. Additionally, Audeh et al [ 38 ] included only 1 web forum specialized in thyroid issues, whereas we incorporated this specific forum along with 16 others. The main ADEs associated with Levothyrox in our study align with those found by Audeh et al [ 38 ] on similar data, albeit without using the interface. In our study, the 10 most frequent symptoms were pain, tiredness, faintness, hypothyroidism, dizziness, insomnia, palpitations, hyperthyroidism, malignant tumor, and anxiety. By contrast, Audeh et al [ 38 ] reported tiredness, weight gain, pain, ganglions, hot flush, chilly, inflammation, faintness, weight loss, and discomfort.

Furthermore, the PHARES pipeline surpasses previous efforts, particularly regarding several criteria. These include the annotation tool, where only 4 pipelines were identified using a French annotator tool. In terms of available statistics, only 1 pipeline met both criteria we identified. Regarding signal detection, among the 3 criteria identified, 5 pipelines matched with only 1, while the remaining 5 matched with none.

In the use case, a notable increase in the number of ADEs associated with Levothyrox was detected using the change-point analysis method a few months after the introduction of the new formula in March 2017, specifically in August 2017. This surge coincided with the initial declaration to the pharmacovigilance network and a petition initiated by patients to reintroduce the former formula in June 2017. We compared these findings with results from a pharmacovigilance study based on spontaneous reporting. Out of 1554 notifications spontaneously addressed by patients to the Pharmacovigilance Center of Nice from January 1, 2017, to December 31, 2017, 1372 were related to the new formula of Levothyrox, representing 7342 ADEs. Our comparison with these data clarified our findings. The 10 most frequently reported ADEs in these notifications closely resembled our own results [ 37 ]. These were asthenia, headache, dizziness, hair loss, insomnia, cramps, weight gain, nausea, muscle pain, and irritability. Consequently, our results demonstrate coherence with the existing literature. This study illustrates the feasibility of identifying the date of significant events related to a drug. However, it is noteworthy that the detection of such events is not necessarily expedited through social media compared with the traditional pharmacovigilance system.

Limitations

The method used in our annotation process was integrated at an early stage during the pipeline’s design. Regarding the identification of drugs and symptoms, our annotation process exhibited the following performances: precision=0.926, recall=0.845, and F 1 -score=0.886 [ 20 ]. Similarly, for discerning the relationship between the drug and the ADEs, the performances were precision=0.683, recall=0.956, and F 1 -score=0.797 [ 20 ]. This study marked the inaugural publication on using NLP methods to identify ADEs in French-language web forums. The annotation process was thus developed using contemporary state-of-the-art methodologies at the time. However, it would now stand to gain from the integration of more recent NLP algorithms for named entity recognition [ 8 , 23 , 24 ]. These newer algorithms offer comparable performances while effectively handling more complex data, thereby enhancing the efficacy of NLP analysis. However, because of our emphasis on the genericity of the approach and the interoperability between the different modules rather than solely focusing on the performance of each module, we opted not to use these algorithms. Nevertheless, contemporary state-of-the-art methods for annotating ADEs from social media posts encompass convolutional neural networks trained on top of pretrained word vectors for sentence-level classification [ 24 ] and transformers using the bidirectional encoder representations from transformers (BERT) language model [ 39 ]. Hussain et al [ 25 ] introduced a multitask neural network based on BERT with hyperparameter optimization capable of sentence classification and named entity recognition. This model achieved performances of precision=0.840, recall=0.861, and F 1 -score=0.896 on the Twitter (X)-TwiMed data set. Additionally, Magge et al [ 8 ] presented a pipeline consisting of 3 BERT neural networks designed to classify sentences, extract named entities, and normalize those entities to their respective MedDRA concepts. The performances of this model were as follows: precision=0.82, recall=0.76, and F 1 -score=0.78 on the SMM4H-2020 data set (Twitter/X). Thanks to our modular design, it will be straightforward to substitute our current annotation process with an enhanced model in the future.

Several limitations should be acknowledged for future work. First, the scraper relies on the HTML structure of web forums, necessitating updates to its configuration files if a forum alters its page design. Additionally, our interface lacks the capability to incorporate alternate identifiers for drugs or ADEs. For instance, patients may commonly refer to the drug “baclofen” as “baclo” on social media platforms. Consequently, the number of posts pertaining to a drug or ADE could potentially be underestimated.

Forums must be selected before query execution to mitigate calculation time. However, selecting forums based on the presence of information related to a particular drug or ADE can introduce bias into signal detection methods, particularly in disproportionality analysis, where the drug-ADE pair may be overrepresented. Another limitation in qualitative analysis of posts is the inability of users to edit annotations or record typical pharmacovigilance qualitative data.

The assumption that all drugs mentioned in a post were consumed simultaneously by the user, as applied in the logistic regression–based method, introduces an evident bias.

One limitation associated with the use of social media data pertains to fraudulent posts. The pseudonymity inherent in these platforms provides malevolent individuals with the opportunity to disseminate false rumors. Additionally, patients might post identical or similar messages across multiple discussion boards, or even multiple times on the same board. Thus, it is crucial to consider these factors to mitigate biases in signal detection.

Perspectives

In the short to medium term, our objectives are updating the annotation module to enhance accuracy, improving the qualitative analysis by enabling users to edit and correct annotations, and expanding the range of signal detection methods available in the statistics module.

This method could indeed be beneficial for identifying potential drug misuse and unknown ADEs [ 40 ]. By categorizing pathological terms found in web forums based on their presence in the summary of product characteristics, we can distinguish between indications, known ADEs, and potential instances of drug misuse or unexpected ADEs. However, it is important to note that considering all pathological terms found in the summary of product characteristics as indications might obscure cases of drug inefficiency. Therefore, a nuanced approach is necessary to ensure comprehensive and accurate analysis.

We next tested our pipeline from the perspective of end users. However, the hypothesis was only partially confirmed, indicating the need for further studies. These studies should include evaluations with ergonomic criteria.

In the long term, our vision is to expand this tool to encompass other languages and themes beyond pharmacovigilance. This includes areas such as drug misuse, the consumption of food supplements, and the use of illegal drugs. French web forums dedicated to recreational drug use already exist, providing a valuable source of data for such endeavors.

Conclusions

Our hypothesis focused on the challenge encountered by regulatory agencies in using social media, primarily because of the lack of appropriate decision-making tools. To tackle this challenge, we devised a pipeline consisting of 4 editable modules aimed at effectively analyzing health-related French web forums for pharmacovigilance purposes. Using this pipeline and its user-friendly interface, we successfully demonstrated the feasibility of conducting quantitative analyses without the need for coding. This approach yielded coherent results and holds the potential to reveal new insights about drugs.

A practical implication of our pipeline is its potential application in health surveillance by regulatory agencies such as the ANSM or pharmaceutical companies. It can be instrumental in detecting issues related to drug safety and efficacy in real time. Furthermore, research teams can leverage this tool to retrospectively analyze events and gain valuable insights into pharmacovigilance trends.

Acknowledgments

The annotation module was developed by François Morlane-Hondère, Cyril Grouin, Pierre Zweigenbaum, and Leonardo Campillos-Llanos from the Computer Science Laboratory for Mechanics and Engineering Sciences (LIMSI). Code review for the graphical user interface in R language was performed by Stevenn Volant under a contract with the Stat4Decision company. Stat4Decision was not involved in designing the study and writing this article. This work was funded by the Agence nationale de sécurité du médicament et des produits de santé (ANSM) through Convention No. 2016S076 and was supported by a PhD contract with Sorbonne Université.

Data Availability

Our data were extracted from web forums that do not allow data sharing. Thus, as we are not the owners of the data we cannot make the data available. The scrapper we developed to extract these data is open source and can be used to extract data from web forum posts. The tool as well as full documentation (in English and French) of the code and configuration file are available online [ 41 ].

Conflicts of Interest

None declared.

Vigi4Med Scraper structure, PHARES database structure, example of data representation, and source and evolution over time of web forum posts. PHARES: Pharmacovigilance in Social Networks.

• Hazell L, Shakir SAW. Under-reporting of adverse drug reactions : a systematic review. Drug Saf. 2006;29(5):385-396. [ CrossRef ] [ Medline ]
• Liu F, Jagannatha A, Yu H. Towards drug safety surveillance and pharmacovigilance: current progress in detecting medication and adverse drug events from electronic health records. Drug Saf. Jan 2019;42(1):95-97. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Warrer P, Hansen EH, Juhl-Jensen L, Aagaard L. Using text-mining techniques in electronic patient records to identify ADRs from medicine use. Br J Clin Pharmacol. May 2012;73(5):674-684. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Black C, Tagiyeva‐Milne N, Helms P, Moir D. Pharmacovigilance in children: detecting adverse drug reactions in routine electronic healthcare records. A systematic review. Brit J Clinical Pharma. May 28, 2015;80(4):844-854. [ CrossRef ] [ Medline ]
• Cameron D, Smith GA, Daniulaityte R, Sheth AP, Dave D, Chen L, et al. PREDOSE: a semantic web platform for drug abuse epidemiology using social media. Journal of Biomedical Informatics. Dec 2013;46(6):985-997. [ CrossRef ] [ Medline ]
• Yeleswarapu S, Rao A, Joseph T, Saipradeep VG, Srinivasan R. A pipeline to extract drug-adverse event pairs from multiple data sources. BMC Med Inform Decis Mak. Feb 24, 2014;14(1):1-16. [ CrossRef ]
• Nikfarjam A, Ransohoff JD, Callahan A, Jones E, Loew B, Kwong BY, et al. Early detection of adverse drug reactions in social health networks: a natural language processing pipeline for signal detection. JMIR Public Health Surveill. Jun 03, 2019;5(2):e11264. [ CrossRef ] [ Medline ]
• Magge A, Tutubalina E, Miftahutdinov Z, Alimova I, Dirkson A, Verberne S. DeepADEMiner: a deep learning pharmacovigilance pipeline for extraction and normalization of adverse drug event mentions on Twitter. J Am Med Inform Assoc. Sep 18, 2021;28(10):2184-2192. [ CrossRef ] [ Medline ]
• Audeh B, Beigbeder M, Zimmermann A, Jaillon P, Bousquet C. Vigi4Med scraper: a framework for web forum structured data extraction and semantic representation. PLoS One. Jan 25, 2017;12(1):e0169658. [ CrossRef ] [ Medline ]
• Caster O, Dietrich J, Kürzinger ML, Lerch M, Maskell S, Norén GN, et al. Assessment of the utility of social media for broad-ranging statistical signal detection in pharmacovigilance: results from the WEB-RADR project. Drug Saf. Dec 2018;41(12):1355-1369. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bousquet C, Audeh B, Bellet F, Lillo-Le Louët A. Comment on "Assessment of the utility of social media for broad-ranging statistical signal detection in pharmacovigilance: results from the WEB-RADR project". Drug Saf. Dec 19, 2018;41(12):1371-1373. [ CrossRef ] [ Medline ]
• Karapetiantz P, Bellet F, Audeh B, Lardon J, Leprovost D, Aboukhamis R, et al. Descriptions of adverse drug reactions are less informative in forums than in the French pharmacovigilance database but provide more unexpected reactions. Front Pharmacol. May 1, 2018;9:439-411. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lardon J, Abdellaoui R, Bellet F, Asfari H, Souvignet J, Texier N, et al. Adverse drug reaction identification and extraction in social media: a scoping review. J Med Internet Res. Jul 10, 2015;17(7):e171. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Karapetiantz P, Audeh B, Faille J, Lillo-Le Louët A, Bousquet C. Qualitative and quantitative analysis of web forums for adverse events detection: "strontium ranelate" case study. Stud Health Technol Inform. Aug 21, 2019;264:964-968. [ CrossRef ] [ Medline ]
• Casperson T, Painter J, Dietrich J. Strategies for distributed curation of social media data for safety and pharmacovigilance. 2016. Presented at: International Conference on Data Science (ICDATA); October 1, 2016:118-124; Barcelona, Spain.
• Freifeld CC. Digital pharmacovigilance: The medwatcher system for monitoring adverse events through automated processing of internet social media and crowdsourcing. OpenBU Libraries. Boston University. OpenBU; 2014. URL: https://open.bu.edu/handle/2144/10995
• Cossin S, Lebrun L, Lobre G, Loustau R, Jouhet V, Griffier R, et al. Romedi: an open data source about French drugs on the semantic web. Stud Health Technol Inform. Aug 21, 2019;264:79-82. [ CrossRef ] [ Medline ]
• Abdellaoui R, Schück S, Texier N, Burgun A. Filtering entities to optimize identification of adverse drug reaction from social media: how can the number of words between entities in the messages help? JMIR Public Health Surveill. Jun 22, 2017;3(2):e36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bousquet C, Dahamna B, Guillemin-Lanne S, Darmoni SJ, Faviez C, Huot C, et al. The adverse drug reactions from patient reports in social media project: five major challenges to overcome to operationalize analysis and efficiently support pharmacovigilance process. JMIR Res Protoc. Sep 21, 2017;6(9):e179. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morlane-Hondère F, Grouin C, Zweigenbaum P. Identification of drug-related medical conditions in social media. 2016. Presented at: The Tenth International Conference on Language Resources and Evaluation (LREC'16); May 2, 2016:2022-2028; Portoroz, Slovenia.
• Lafferty J, McCallum A, Pereira F. Conditional random fields: probabilistic models for segmenting and labeling sequence data. San Francisco, CA. Morgan Kaufmann Publishers; 2001. Presented at: Eighteenth International Conference on Machine Learning (ICML 2001); June 28, 2001 to July 1, 2001:282-289; Williamstown, MA.
• Xia L. Historical profile will tell? A deep learning-based multi-level embedding framework for adverse drug event detection and extraction. Decision Support Systems. Sep 2022;160:113832. [ CrossRef ]
• Yu D, Vydiswaran VGV. An assessment of mentions of adverse drug events on social media with natural language processing: model development and analysis. JMIR Med Inform. Sep 28, 2022;10(9):e38140. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rezaei Z, Ebrahimpour-Komleh H, Eslami B, Chavoshinejad R, Totonchi M. Adverse drug reaction detection in social media by deep learning methods. Cell J. Oct 2020;22(3):319-324. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hussain S, Afzal H, Saeed R, Iltaf N, Umair MY. Pharmacovigilance with transformers: a framework to detect adverse drug reactions using BERT fine-tuned with farm. Comput Math Methods Med. 2021;2021:5589829. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ding P, Zhou X, Zhang X, Wang J, Lei Z. An attentive neural sequence labeling model for adverse drug reactions mentions extraction. IEEE Access. 2018;6:73305-73315. [ CrossRef ]
• Xu Z, Kass-Hout T, Anderson-Smits C, Gray G. Signal detection using change point analysis in postmarket surveillance. Pharmacoepidemiol Drug Saf. Jun 22, 2015;24(6):663-668. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. Dec 10, 2001;10(6):483-486. [ CrossRef ] [ Medline ]
• van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. Feb 06, 2002;11(1):3-10. [ CrossRef ] [ Medline ]
• Ahmed I, Pariente A, Tubert-Bitter P. Class-imbalanced subsampling lasso algorithm for discovering adverse drug reactions. Stat Methods Med Res. Mar 25, 2018;27(3):785-797. [ CrossRef ] [ Medline ]
• Caster O, Norén GN, Madigan D, Bate A. Large‐scale regression‐based pattern discovery: the example of screening the WHO global drug safety database. Statistical Analysis. Jul 20, 2010;3(4):197-208. [ CrossRef ]
• Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. Jun 11, 2013;93(6):539-546. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Team R. The R Project for Statistical Computing. R Foundation. URL: http://www.R-project.org/ [accessed 2024-04-26]
• Chang W, Cheng J, Allaire J, Sievert C, Schloerke B, Xie Y. shiny: web application framework for R. Comprehensive R Archive Network. URL: https://CRAN.R-project.org/package=shiny [accessed 2023-01-30]
• Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). EUR-Lex. URL: https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:32016R0679 [accessed 2024-04-26]
• SHA-1. Wikipedia. 2023. URL: https://en.wikipedia.org/w/index.php?title=SHA-1&oldid=1135933131 [accessed 2023-01-30]
• Viard D, Parassol-Girard N, Romani S, Van Obberghen E, Rocher F, Berriri S, et al. Spontaneous adverse event notifications by patients subsequent to the marketing of a new formulation of Levothyrox amidst a drug media crisis: atypical profile as compared with other drugs. Fundam Clin Pharmacol. Aug 07, 2019;33(4):463-470. [ CrossRef ] [ Medline ]
• Audeh B, Grouin C, Zweigenbaum P, Bousquet C, Jaulent M, Benkhebil M. French Levothyrox® crisis: retrospective analysis of social media. Bogota, Colombia. Springer International Publishing; 2019. Presented at: Conference ISOP - International Society of Pharmacovigilance; October 1, 2019; Bogota, Colombie. URL: https://hal.archives-ouvertes.fr/hal-02411632
• Devlin J, Chang M, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT 2019. 2019. Presented at: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; June 2-7, 2019:4171-4186; Minneapolis, MN. URL: https://aclanthology.org/N19-1423.pdf
• Campillos-Llanos L, Grouin C, Lillo-Le Louët A, Zweigenbaum P. Initial experiments for pharmacovigilance analysis in social media using summaries of product characteristics. Stud Health Technol Inform. Aug 21, 2019;264:60-64. [ CrossRef ] [ Medline ]
• Vigi4Med Scraper. GitHub. URL: https://github.com/bissana/Vigi4Med-Scraper [accessed 2024-04-22]

Abbreviations

Edited by A Mavragani; submitted 01.02.23; peer-reviewed by S Matsuda, L Shang; comments to author 06.07.23; revised version received 20.10.23; accepted 12.03.24; published 18.06.24.

©Pierre Karapetiantz, Bissan Audeh, Akram Redjdal, Théophile Tiffet, Cédric Bousquet, Marie-Christine Jaulent. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.06.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Synonyms for Case-study

0 other terms for case-study - words and phrases with similar meaning.