quantitative research on mental health

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 28 February 2018

Quantitative account of social interactions in a mental health care ecosystem: cooperation, trust and collective action

• Anna Cigarini 1 , 2 ,
• Julián Vicens   ORCID: orcid.org/0000-0003-0643-0469 1 , 2 , 3 ,
• Jordi Duch   ORCID: orcid.org/0000-0003-2639-6333 3 , 4 ,
• Angel Sánchez 5 , 6 , 7 , 8 &
• Josep Perelló   ORCID: orcid.org/0000-0001-8533-6539 1 , 2  

Scientific Reports volume  8 , Article number:  3794 ( 2018 ) Cite this article

9469 Accesses

10 Citations

51 Altmetric

Metrics details

• Applied mathematics
• Human behaviour
• Psychology and behaviour
• Public health

An Author Correction to this article was published on 26 September 2018

This article has been updated

Mental disorders have an enormous impact in our society, both in personal terms and in the economic costs associated with their treatment. In order to scale up services and bring down costs, administrations are starting to promote social interactions as key to care provision. We analyze quantitatively the importance of communities for effective mental health care, considering all community members involved. By means of citizen science practices, we have designed a suite of games that allow to probe into different behavioral traits of the role groups of the ecosystem. The evidence reinforces the idea of community social capital, with caregivers and professionals playing a leading role. Yet, the cost of collective action is mainly supported by individuals with a mental condition - which unveils their vulnerability. The results are in general agreement with previous findings but, since we broaden the perspective of previous studies, we are also able to find marked differences in the social behavior of certain groups of mental disorders. We finally point to the conditions under which cooperation among members of the ecosystem is better sustained, suggesting how virtuous cycles of inclusion and participation can be promoted in a ‘care in the community’ framework.

Similar content being viewed by others

Reformulating computational social science with citizen social science: the case of a community-based mental health care research

Investigation of turning points in the effectiveness of Covid-19 social distancing

An agent-based model of social care provision during the early stages of Covid-19

Introduction.

Approximately one fifth of the world population will suffer some mental disorder (MD) at some point in their lives, such as anxiety or depression 1 . The direct economic costs of MD, including care and indirect effects, is estimated to reach $6 trillion in 2030, which is more than cancer, diabetes, and respiratory diseases combined 2 . As part of a global effort to scale up services and bring down costs, reliance is increasingly made upon informal social networks 3 . A holistic approach to mental health promotion and care provision is then necessary, and emphasis is placed on the idea of individuals-in-community: individuals with MD are defined not just alone but in relationship to others 4 . Such a paradigm shift implies superseding the traditional physician-patient dyad to include caregivers, relatives, social workers, and the community as a whole, recognizing their crucial role in the recovery process.

A key aspect in the definition and aetiology of MD has to do with social behavior 5 : behavioral symptoms, or consequences at the behavioral level, characterize most MD. For instance, autism, social phobia, or personality disorders are determined by the presence of impairments in social interaction. Other disorders result in significant difficulties in the social domain, such as depression or psychotic disorders. Further, conditions that are intrinsically behavioral (as for eating disorders or substance abuse) seem to be exacerbated by the influence of social peers. A large body of research has therefore looked at the neural basis of social decision-making among individuals with MD to identify objective biomarkers that may prove useful for its diagnosis, therapy evaluation, and understanding 6 , 7 , 8 . However, such a methodology does not well fit into the individuals-in-community paradigm. We argue that an agent-based approach which draws upon experimental game theory might prove insightful and ecologically valid for the study of behavior in a given social environment.

Within the mental health literature, the use of game theory as a way to understand the multi-faceted dimensions of behavior has received already quite some attention 9 , 10 . Most research addressed the issue of behavioral differences between individuals with MD and healthy populations 6 , 7 , 11 , 12 , 13 , 14 , 15 , 16 . These works, that point to cognitive and affective processing impairments 6 , 16 , 17 , further support the idea that MDs are associated with significant and pervasive difficulties in social cognition and altered decision-making at various levels. Yet, despite these studies are of very much interest, they are primarly concerned with dyadic interactions among people with specific MDs. That is, they lack insights into the complexity of individual behaviors of MD within a specific social context.

Here we adopt a novel community perspective. Our objective is twofold: First, we aim to develop a thorough taxonomy of the behavioral traits of role groups within the collective. We thus account for both the heterogeneity of actors, and for multiple types of social interactions. We strongly believe that to predict and understand behavior is necessary to consider the relationship context in which individuals are embedded. Therefore diversity of roles, motivations or capabilities, must be taken into account. Also, real life social interactions occur in different forms; sometimes people must work together, some others they have to coordinate or anti-coordinate their behaviors, yet in other situations they find themselves in more or less disadvantaged positions. It is therefore of crucial importance to encompass a comprehensive range of strategic situations if we are to appreciate behavior. That is, traits such as trust, altruism, or reciprocity, along with the person’s own expectations, all play a role in the process of decision making in social contexts. This calls for an experimental approach in which participants face several strategic settings. Our second objective is to provide quantitative accounts of social capital within the mental health community, bringing the notion of social capital into the forefront of mental health care. Far from being universally defined, its core contention is that social networks are a valuable asset, providing a basis for social cohesion and cooperation towards a common goal 18 (which is, in our case, mental care provision). It thus encompasses those norms and forces that shape social interactions, serving as the glue that holds society together 19 .

For these purposes, we have designed an experimental setup that probes into the complexity of the interdependencies at play within the mental health ecosystem. Accordingly, our experiments take place in a socialized, lab-in-the-field setting 20 , in order to be as close as possible to the dynamic and unique nature of real-life social interactions. The design of our socialized setup is based on a participatory process and citizen science practices 20 which counted on the collaboration of all stakeholders of the mental health ecosystem. By combining all these ingredients, we have developed a framework that, as will be shown below, allows to capture some difficult-to-observe aspects of behavior and social capital within mental health ecosystems as a way to understand how communities contribute to care and resocialization.

A full description of the games we implemented can be found in the Methods section below, but for clarity we briefly describe here the games we used. We had participants play two dyadic games, namely the Trust game, in which they had to lend money to another player who then obtains a return, and has the option to send some money back to the lender; players played in both roles. They also played the well known Prisoner’s Dilemma, in which they had to choose to cooperate or to try to benefit from the other’s cooperation. Finally, they played a collective risk dilemma, in which the whole group had to reach a common goal to avert a catastrophe that most likely would wipe out their money. Participants belonging to the mental health ecosystem played with each other in group of six players. However, they could by no means guess with whom they were actually playing.

We begin the presentation of our results from the dyadic games of our suite of strategic interactions. Aggregate behavioral measures point to systematic deviations from self-interested predictions which are in line with previous literature on experimental game play 21 . In the Prisoner’s Dilemma (PD), the average cooperation rate across all individuals is c  = 0.61 ± 0.03 (standard error of the mean), which is notoriously well above the Nash equilibrium prediction of c  = 0. Participants behavior in the PD is also significantly associated with their estimates about the likely cooperation of the partner ( \({\chi }^{2}=32.48\) , p  = 1.2 · 10 8 ), with 44% of all participants expecting the partner to cooperate, and thus cooperating themselves. This points to the crucial role of positive expectations on cooperative behavior 22 . Further, participants trust and reciprocate positive amounts in the Trust Game (5.79 ± 0.15 monetary units (MU) and 41.3 ± 1.37% of the amount available to return, respectively), again departing largely from Nash equilibrium conjectures of 0 MU transferred. The results also suggest that in considering the mental health community in its whole, thus accounting for the diversity of actors and roles, the global picture does not substantially differ from society at large.

Sectorial and dyadic behavior

As we stated above, our main interest is to delve into the behavior of the different actors who make up the mental health ecosystem Fig.  1 summarizes the results for the five groups of individuals concerned. The heatmap yields several insights that are worth commenting upon.

Heatmap of behavioural traits’ average and deviation of the mean across games. Collectivity refers to the ratio of contribution in the Collective-Risk Social Dilemma. Cooperation and Optimism refers to the ratio of cooperation and expected cooperation, respectively, in Prisoner’s Dilemma. Trust and Reciprocity refers to the ratio of capital trusted and reciprocated in Trust Game. The left part shows the ratio of individuals without mental conditions: caregivers (professionals and relatives with caregiving tasks) and non-caregivers (relatives without caregiving tasks, friends and others). The right part shows the actions of individuals with mental conditions. Therefore, the number in each cell indicates the ratio of social preferences per subjects in each social dilemma and the color scale shows the deviation of the mean measured in SD units.

In one-shot dyadic interactions some marked differences in the frequency of cooperative behaviors (PD) arise within the collective formed by affected with MD, caregivers, non-caregivers (Kruskal-Wallis rank sum test, \(H=6.04,df=2,p=0.0488\) ). Further pairwise comparisons (see Supplementary Table  S1 ) show that participants with anxiety and caregivers are more likely to opt for the cooperative strategy compared to participants with bipolar disorder, psychosis or other members of the collective. Participants with anxiety are also the ones with the most positive expectations about the partner’s behavior compared to all but caregivers (see Supplementary Table  S2 ). Also, relatives, friends and other members with no MD defect more than caregivers (Mann-Whitney U test, \(U=1352,p=0.02839\) ), being relatives remarkably less cooperative than the rest of the collective c  = 0.33 ± 0.16. This suggests that cooperation among members of the mental health ecosystem is contextually based, depending on the role that actors play in the recovery process. It also varies across diagnostics, revealing a marked cooperativeness and optimism of individuals with anxiety disorders.

On the other hand, in sequential dyadic interactions (TG) all participants trust more than half of their endowment, being the distribution of initial transfers similar across groups. No variation is indeed found in trust levels between participants with MD, caregivers and non caregivers (Kruskal-Wallis rank sum test, \(H=2.75,df=2,p=0.25\) ). Yet, at the time of reciprocating the partner’s behavior, participants with anxiety and depression return the least (37.5 ± 3.3%). The difference is significant if compared to return transfers of participants with psychosis or other diagnostics (see Supplementary Table  S4 ).

Group interaction

Our experimental setup has proven extremely informative in its most novel section, namely the analysis of group interactions framed within the Collective Risk Dilemma (CRD), with no prior result within the mental health literature. In global terms, the average amount contributed to the public good (22.6 MUs) is much more than the fair contribution of 20 MUs, where by fair we understand sharing equally the total amount needed for the threshold (120 MUs) among all six participants. Here it is important to keep in mind that participants were told that all money contributed would go to reforestation projects, so it is not irrational to keep contributing beyond the threshold as many of our subjects did. The key result in the CRD is that large, significant differences (t-test, \(t=2.85,df=242,p=0.0047\) ) are found between participants with and without mental disorders. The former contribute with 22.95 ± 0.63 MUs compared to 20.34 ± 0.68 MUs from the latter, and therefore it appears that when repeated interaction and sustained teamwork (CRD) are required, people with MD contribute much more to the common goal (See Supplementary Section 1.6.2).

Contribution dynamics vary according to group composition in terms of number of participants with mental disorder conditions and other actors involved in the recovery process. All groups successfully reach the target collecting on average 135.64 ± 1.75 MUs (see Supplementary Section 1.6.1). Similarly to other public good experiments, contributions decrease over time 23 . While in the first round participants contribute around 56.3% of the allowed contribution per round (2.2 ± 0.07 MUs, where the social optimum is 2 MU), contributions drop when the endgame effect sets in. A Spearman’s rank-order correlation of contributions over rounds corroborates this negative time trend ( \(\rho =-0.757,p < 0.05\) ). Both patients and actors involved in the recovery process reduce their contributions by the end of the game. However, in almost all rounds, participants with a mental condition contribute more than caregivers and non caregivers, for whom motivations to contribute decline steadily (see Fig.  2 ).

(a) Individual contribution over rounds. Evolution of contributions (mean and standard error of the mean) during the game between participants with mental disorder conditions, caregivers and non-caregivers. We can see that all groups behave similarly and in an identical way to a previous experiment run outside the mental health ecosystem 40 . (b) Average individual contribution per round. Average contribution and standard error of the mean in the mental health ecosystem. There are significant differences between participant with MD and the rest of actors, caregivers (t-test, \(t=2.107,df=155,p < 0.0294\) ) and non-caregivers (t-test, \(t=2.499,df=48,p=0.01588\) ). Distribution of choices by participants with MD ( c ), caregivers (d) and non-caregivers ( e ). The most of participants with MD (43.6%) selected the maximum contribution (4), while the caregivers (46.5%) and non-caregivers (48.9%) mostly selected the fair contribution (2).

In terms of the group composition, groups where individuals with MD conditions constitute half or the majority of the group (n = 36) do much better in sustaining cooperation compared to groups where firsthand affected are the minority (n = 9). It is here worth to mention that participants may see who the rest of the members are but ignore who is exactly making the choice in the game (see Methods for further details). As Fig.  3b shows, while average individual contributions are similar in the last periods (rounds 6–10 t-test, t  = 0.19, p  = 0.85), groups with half or more individuals with MD contribute significantly more at the beginning of the game (rounds 1–5 t-test, t  = 2.79, p  = 0.0054). Hence, the presence of three or more individuals with a mental condition in the group has a positive and stabilizing effect on average individual contributions. Likewise, in games with a low proportion of participants affected with MD the group achieved the goal, on average, later than in games with more than 50% of participants affected with MD (see Fig.  3a ).

(a) Average round of achievement. Round (mean and standard error of the mean) in which the group of six achieved the target. (b) Aggregated contributions per group composition. Contributions (mean and standard error of the mean) in the first and last five-rounds per number of individuals with MD in a group. There are significant differences (t-test p  < 0.01) in contributions in the first part of the game. (c ) Contributions per group of six. Total group contributions by number of individuals with mental conditions in the group. (d ) Gini index of final payoff within groups. Level of inequality in final payoff based on the number of individuals with MD in each group.

If we then break down the analysis by group type, we find that group members contribute and benefit differently from cooperation (see Fig.  3c ). Indeed, final payoffs within groups are far from being equally distributed (see Fig.  3d ), with the highest inequality found in the group where the number of patients equals the number of actors involved in the recovery process (Gini coefficient = 0.289). We thus see clearly that the cost of collective action is mainly supported by individuals with a mental disorder. Given that they contribute the most within all groups, lower investments are needed for other members of the collective to reach the common target. Yet, in 4/6 and 5/6 groups caregivers reduce average individual contributions while non-caregivers pay more than their fair share. In 1/6 and 2/6 groups, on the other hand, caregivers are the ones who compensate the unfair contributions of other members. These last groups are the ones that ensure the lowest inequality in final payoffs. Therefore, while our results are unambiguous about the larger readiness for collective action among people with MD, we cannot claim nothing about the rest of the collective.

Let us now turn to the discussion of the above results and their implications (see Table  1 for a summary of the key findings). As a first general remark, through our lab-in-the field experiment we found that an ecosystem approach to mental health care brings with it a quite complex scenario with several interesting insights. To begin with, participants with anxiety symptoms display a markedly different behavior compared to other diagnostics: they are more likely to opt for the cooperative strategy compared to individuals with bipolar disorder or depression, and return significantly less than participants with psychosis or other disorders. Since the current study is the first to investigate social decision-making within a heterogeneous population of individuals diagnosed with MD, a comparison with previous research is only possible referring to studies focusing on specific clinical and quite homogeneous populations. Several experiments have demonstrated deficits in cooperative behavior among individuals with anxiety or depression when playing iterated versions of the PD 11 , 17 , 24 , 25 , but results about altruism (Ultimatum Game) and trust are inconsistent between studies 6 , 7 , 11 , 12 , 17 , 26 . Individuals with major depressive disorders (which include anxiety and depressive symptoms) have also been found to systematically differ when their emotional responses to fairness are compared 6 , 17 , showing higher levels of negative feelings when faced with unfair treatments. One of the hypothesis advanced to explain the systematic behavioral differences of individuals with anxiety relates to a potentiated sensitivity to negative stimuli as well as a tendency to treat neutral or ambiguous stimuli as negative or as less positive 6 , 12 , 17 , 27 . This hypothesis might find support in our results as for the low returns in the Trust Game, despite displaying relatively high trust in the partner’s behavior and very high expectations. Indeed, participants with depressive or anxiety symptoms in our experiment significantly over-punish trustee transfers, but the low returns are independent of the amount received. This seems to imply that participants with mood disorders respond negatively to their partner behavior, as if they interpret their partner’s choice in a negative sense. Alternatively, fairness considerations may be playing a role: low returns of participants with mood disorders might therefore be due to different fairness perceptions 6 , 12 , 17 , which result in a bias towards negative reactions rather than positive rewarding.

Deficits in economic game play have also been documented for individuals with bipolar disorder. Studies report low and decreasing trust levels over sequential interactions, skeptical beliefs about the partner’s behavior and a tendency to break cooperative interactions 28 , 29 . Again, this is partly supported by our results. Negative expectations of participants with bipolar disorder indeed agree with a low frequency of cooperative choices, little amounts of money sent to trustees, and low contributions to collective action. In line with King-Casas et al . results 29 , while individuals with depression trust in the cooperativeness of other people, those with bipolar personality disorders do not. Cognitive dysfunctions (insula response) might possibly reflect an atypical social norm in this group 29 . Consequently, defection by partners might not violate the social expectations of individuals with BPD. In contrast, in our experiment, participants with bipolar disorder return the most within the group of individuals with a mental disorder. That is, they report a strong willingness to positively respond to a norm of trust as to signal their partner trustworthiness. Therefore, conditioned on the previous action of the partner, it seems that individuals with BPD are willing to show cooperative behavior. Considering now individuals with high levels of psychopathy, they have been found to make less fair offers, accept less fair offers, and show very high levels of defection 15 , 16 , 30 . Major explanations for such behavior point to deficits in emotion regulations (amygdala dysfunctions), which would lead to lack of anxiety, empathy, and guilt, coupled with exaggerated levels of anger and frustration 30 and to the absence of prepotent biases toward minimizing the distress of others 16 . In this case, our experiments do not confirm those previous results: Indeed, participants with psychosis are the ones who trust, contribute the most to the public good, and are willing to take costly actions to reciprocate their partner’s behavior. It could be possible that, as psychopathic disorders are in fact a large group of different ones, behavioral differences among subgroups may lead to this discrepancy. In connection with these results, it is interesting to note that recent results on a large population of patients with paranoia suggest that distrust is not the best explanation for reduced cooperation and alternative explanations incorporating self-interest might be more relevant 31 , 32 . This calls for further research into this particular family of MD to clarify whether or not the behavioral characterization applies to all or to a subclass of them.

However, pointing to deficits in social cognition can only account for a partial explanation of individual behavior, and does not contribute to community care narratives. The fact that nothing in this direction has been reported before also reinforces the need to adopt a more holistic view on the interdependencies at play within the mental health collective. Indeed, if statistically relevant differences in cooperative behavior are found across diagnostics, they also depend on the role that actors play in the recovery process. That is, caregivers display exceedingly large degrees of cooperativeness and optimism in one-shot interactions. Caregivers can be thus considered the strong ties of the mental health ecosystem, of particular value when one seeks emotional support. With the de-institutionalization of health systems, caregivers have indeed become key players in care provision. Taking into account their behavior and expectations is therefore of particular interest to extend the support tailored to their needs. These actions should improve the effectiveness of their role by guiding them 33 . Yet, relatives who do not strictly contribute to caregiving practices turn out to be the weak links. It is thus likely that interventions designed to increase their participation in the community might help improve the recovery process.

Also, members of the mental health ecosystem do not equally contribute and benefit from collective action. Rather, systematic behavioral differences arise as the number of social interactions increase, i.e., when teamwork is required for the collective to benefit as a whole. This suggests that considering repeated games may prove extremely insightful for the purpose of the research. Indeed, our experiments show that individuals with MD are the ones who contribute the most to the public good: they make larger efforts towards reaching the collective goal, thus playing a leading role for the functioning of the ecosystem. As a consequence, groups with half or more participants with MD do better in sustaining cooperation in the first rounds, which implies that a community care setting might prove successful for capability building. Yet, large proportions of individuals with MD in a group result in higher inequalities in final gains, which reach the maximum when the number of individuals with MD equals the number of caregivers or relatives. This means that community care perspectives might also take account of group composition to deal with potential inequalities arising from differential capabilities. In summary, we have explored the behavior of all individuals and role groups who make up the mental health ecosystem through an extensive suite of games that simulate strategic social situations. Overall, the results point to the availability of large social capital in the mental health community that can make a difference in the welfare and recovery process of firsthand affected, and suggest that the community-centered approach to mental care may turn out to be very beneficial. Indeed, the behavior of individuals with MD can be better explained by examining not only their cognitive abilities, but also the web of relationships in which they are embedded. Yet, that web of relationships presents opportunities and imposes constraints.

Though we depicted some behavioral differences in dyadic interactions, most importantly we found that individuals with MD show a remarkably larger disposition towards sustaining cooperation within groups. The larger readiness of individuals with MD to contribute to the collective action problem can thus be seen as a way to claim their place in the community. By having participants unaware of their partner’s identity, we could indeed measure participants decisions based solely on the value they placed on the group’s welfare, independently of its composition or other factors. Yet, the fact that participants with MD contribute the most implies for other members of the group lower investments to reach the common target. This, on the other hand, unveils the vulnerability of individuals with a diagnosis of MD. Repeated or periodic and more situated experiments with digital platforms 34 , in the future, can surely provide further valuable insights into the effect of participants prior knowledge of and relation with the partner on their behavior. We are indeed sure that our experimental setup can prove helpful in complementing the diagnostic process of physicians and health professionals and even to evaluate care service providers. On the other hand, other possible application of this approach arises in the realm of behavior change interventions 35 , that should focus on the aspects that are more specific of every disorder.

In conclusion, the results reinforce the idea of community social capital as a key approach to the recovery process based on an ecosystem paradigm (see also the recent results in ref. 36 about the role and impact of family and community social capital on MD in children and adolescents). Also, if on the one hand the fact that the results of our dyadic games are in general agreement with previous studies validates our procedure; on the other hand it supports the validity and contributions of neuroeconomics and experimental approaches to the study of MD. Finally, given that our work has been carried out in a fully socialized context, this approach can be applied to any similar’ ‘care in the community’ initiative. The adoption of our setup could lead to the identification of core groups that can boost and sustain cooperation within a given community. It can also help in discriminating among different communities in order to identify best practices and optimize resource allocation 37 .

All participants were fully informed about the purpose, methods and intended uses of the research. No participant could approach any experimental station without having signed a written informed consent. The use of pseudonyms ensured the anonymity of participants’ identity, in agreement with the Spanish Law for Personal Data Protection. No association was ever made between the participants’ real names and the results. The whole procedure was approved by the Ethics Committee of Universitat de Barcelona. All methods were performed in accordance with the relevant guidelines and regulations.

Experimental design

As indicated in the main text, the dialogue with the main stakeholders of the mental health ecosystem was at the centre of the project. Around 20 representatives including members of the Catalonia Federation of Mental Health (Federació Salut Mental Catalunya), firsthand affected, relatives, caregivers, and other professionals related to both the health and social sector, informed and validated the whole research through focus groups and further discussions, leading to the largest experiment of this kind ever carried out. Citizen science principles guided the whole experimental design process in order to raise concerns grounded in the daily life of mental health professionals and service users, and to increase public awareness. The experimental dilemmas being proposed served both to advance in knowledge on the social dynamics at play within ‘care in the community’ settings and as a self-reflection experience for all participants. The experimental design process developed in four main phases: (i) identification of the behavioral traits perceived as of fundamental importance within the community, (ii) operationalization of those same behavioral traits thorugh game theoretical paradigms and literature reviews, (iii) definition of the socio-demographic information relevant for the analysis, and (iv) a beta testing of the digital interface (including contents, time duration, and language used). The locations where the experiments took place were accorded with the Catalonia Federation of Mental Health in an attempt to explore the functioning of some communities of interest for inclusive and effective policy making. The Federation provided a fundamental support throughout the whole experiments’ implementation, serving as a crucial intermediaire between the scientists and different mental health collectives. It also provided valuable insights to better interpret the data obtained.

Participants and procedure

To our knowledge, experimental work on this issue has been conducted only recently and on specific collectives of orders of magnitude smaller. A total of 270 individuals participated in the experiments, that were run over 45 sessions between October 2016 and March 2017. The experiments were carried out in Girona (n = 60), Lleida (n = 120), Sabadell (n = 48) and Valls (n = 42). Participants were either diagnosed with a mental condition (n = 169) or members of the broader mental health ecosystem (n = 101), including professionals of the health and social sector (n = 52), formal and informal caregivers (n = 17), relatives (n = 9), friends (n = 4) and other members of the collective (n = 19). Individuals with a mental condition had to self-assess their diagnosis selecting one from a spectrum of options agreed upon with representatives of the mental health ecosystem during the co-design phases of the experiment. Those participants who had received more than one diagnosis had to select the one they considered to be the most relevant. Overall, they had received a diagnosis of psychosis (n = 63), depression (n = 33), anxiety (n = 31), bipolar disorder (n = 17) or other unspecified diagnosis (n = 25). They ranged in age from 21 to 77 years old (these are weighted values since for ethical and privacy reasons participants were only asked to choose among different age ranges) with 47.2 years on average. Further, 55.6% were men and 44.4% were women. Yet, actors involved in the recovery process were predominantly women (76.2%), and up to 21.8% of them was over 60 years old (see Supplementary Section 1.1). Participants were told that they would play against each others a set of games meant to explore human decision-making processes. They played in random groups of six players through a web interface specifically developed for the research. They were informed that they had to make a decision under different conditions and against different opponents in every round. Every game represented an interactive situation requiring the participants to make a decision, the result of which depended also on the opponent’s behavior. To incentivize the participation, they would earn a voucher worth their final score (the experimental settings and instructions, can be found in the Supplementary Section 1.2 and 1.3 respectively). First, participants participated in a Collective Risk Dilemma 23 against five opponents. Briefly, the game is a public goods game with threshold: If the participants’ total contribution after 10 rounds is lower than a given threshold, they loose all the money they kept with a probability of 90%. Otherwise, they are told that the money collected in the common fund are spent in reforesting land plots in Catalonia, where the experimental sessions took place, and each participants earns the money left in the personal account. After completing the task, participants played one round of the Trust Game 38 in both roles: as trustors and as trustees. They played against different partners in each role. Finally, they played one round of a Prisoner Dilemma 39 with (unincentivized) belief elicitation about their counterpart’s behavior prior to playing. Before starting the games, participants had to complete a brief survey covering some key dimensions of their sociodemographic background. The assignment of players’ partners in the dyadic games was completely random and every action was made with a different partner. The average (standard error of the mean) time for completing the three experiments (CRD, PD and TG and tutorials) is around 12 minutes, 705.86 ± 17.93 s. At the end of each session, participants received a gift card worth their earnings. The average individual earning is 46.84 ± 0.77 MUs equivalent to a 4.04 ± 0.077 EUR voucher. The behavioral patterns that emerged do not reveal significant variation across the different experiments, which may suggest that our results are robust to generalizations (see Supplementary Section 1.7).

Statistical analysis

Results were analyzed at two levels: first, we tested for behavioral differences between the whole group of individuals with mental condition compared to members of the mental health ecosystem; we then checked for systematic behavioral variation across diagnostics and role played in the recovery process. In one shot, two-person dyadic interactions we performed Mann-Whitney-U tests for independent groups to compare the distributions of cooperative choices (PD), and initial and back transfers (TG), between individuals with and without a mental condition. We then checked for marginal differences within groups using Kruskal-Wallis tests, and post-hoc comparisons were run with Mann-Whitney-U tests adjusting for p-values with the Holm-Bonferroni method. Welch’s two-tailed t-tests were performed to check for differences in average contributions (CRD) between participants with and without a MD, controlling for unequal variances and sample sizes. Finally, ANOVA and further Tukey HSD post-hoc comparisons served to check for differences in average contributions over round across diagnostics and members of the mental health community.

Accession codes

Data is available in an structured way at Zenodo public repository with DOI 10.5281/zenodo.1175627.

Change history

26 september 2018.

A correction to this article has been published and is linked from the HTML and PDF versions of this paper. The error has not been fixed in the paper.

Steel, Z. et al . The global prevalence of common mental disorders: A systematic review and meta-analysis 1980–2013. Int. J. Epidemiol. 43 , 476–493 (2014).

Article   Google Scholar  

Insel, T. R., Collins, P. Y. & Hyman, S. E. Darkness invisible: The hidden global costs of mental illness. Foreign Affairs 94 (2015).

White, R. G., Jain, S., Orr, D. M. R. & Read, U. M. The Palgrave handbook of sociocultural perspectives on global mental health. The Palgrave handbook of sociocultural perspectives on global mental health. (2017).

Chapter   Google Scholar  

World Health Organisation. The World Health Report 2001: mental health, new understanding, new hope. World Health Report 1–169 (2001).

American Psychiatric Association. Diagnostic and statistical manual of mental disorders. Diagnostic and Statistical Manual of Mental Disorders 4th edition TR. (2000).

Gradin, V. B. et al . Abnormal brain responses to social fairness in depression: an fMRI study using the Ultimatum Game. Psychol. Med. 45 , 1241–51 (2015).

Article   CAS   Google Scholar  

Shao, R., Zhang, H. & Lee, T. M. C. The neural basis of social risky decision making in females with major depressive disorder. Neuropsychologia 67 , 100–10 (2015).

Guroglu, B., van den Bos, W., Rombouts, S. A. R. B. & Crone, E. A. Unfair? It depends: Neural correlates of fairness in social context. Soc. Cogn. Affect. Neurosci. 5 , 414–423 (2010).

Wang, Y., Yang, L. –Q., Li, S. & Zhou, Y. Game Theory Paradigm: A New Tool for Investigating Social Dysfunction in Major Depressive Disorders. Front. Psychiatry 6 , 128 (2015).

PubMed   PubMed Central   Google Scholar  

King–Casas, B. & Chiu, P. H. Understanding interpersonal function in psychiatric illness through multiplayer economic games. Biol. Psychiatry 72 , 119–125 (2012).

Pulcu, E. et al . Social–economical decision making in current and remitted major depression. Psychol. Med. 45 , 1–13 (2015).

Wang, Y. et al . Impaired social decision making in patients with major depressive disorder. BMC Psychiatry 14 , 18 (2014).

Csukly, G., Polgár, P., Tombor, L., Réthelyi, J. & Kéri, S. Are patients with schizophrenia rational maximizers? Evidence from an ultimatum game study. Psychiatry Res. 187 , 11–17 (2011).

Agay, N., Kron, S., Carmel, Z., Mendlovic, S. & Levkovitz, Y. Ultimatum bargaining behavior of people affected by schizophrenia. Psychiatry Res. 157 , 39–46 (2008).

Mokros, A. et al . Diminished cooperativeness of psychopaths in a prisoner’s dilemma game yields higher rewards. J. Abnorm. Psychol. 117 , 406–413 (2008).

Rilling, J. K. et al . Neural Correlates of Social Cooperation and Non-Cooperation as a Function of Psychopathy. Biol. Psychiatry 61 , 1260–1271 (2007).

Scheele, D., Mihov, Y., Schwederski, O., Maier, W. & Hurlemann, R. A negative emotional and economic judgment bias in major depression. Eur. Arch. Psychiatry Clin. Neurosci. 263 , 675–683 (2013).

Putnam, R. D. Bowling Aalone: The Collapse and Revival of American Community. J. Democr. 6 , 1–18 (2000).

Google Scholar  

Ostrom, E. In Handbook of Social Capital: The troika of sociology, political science and economics, pp. 17–35 (2009).

Sagarra, O., Gutiérrez-Roig, M., Bonhoure, I. & Perelló, J. Citizen science practices for computational social science research: The conceptualization of pop-up experiments. Front. Phys. 3 , 93 (2016).

Camerer, C.F., Behavioral Game Theory: Experiments in Strategic Interaction (Princeton University Press, 2003).

Brañas-Garza, P., Rodríguez-Lara, I. & Sánchez, A. Humans expect generosity. Sci. Rep. 7 , 42446 (2017).

Article   ADS   Google Scholar  

Milinski, M., Sommerfeld, R. D., Krambeck, H.-J., Reed, F. A. & Marotzke, J. The collective-risk social dilemma and the prevention of simulated dangerous climate change. Proc. Nat’l. Acad. Sci. USA 105 , 2291–2294 (2008).

Article   ADS   CAS   Google Scholar  

Surbey, M. K. Adaptive significance of low levels of self-deception and cooperation in depression. Evol. Hum. Behav. 32 , 29–40 (2011).

Clark, C. B., Thorne, C. B., Hardy, S. & Cropsey, K. L. Cooperation and depressive symptoms. J. Affect. Disord. 150 , 1184–1187 (2013).

Destoop, M., Schrijvers, D., De Grave, C., Sabbe, B. & De Bruijn, E. R. A. Better to give than to take? Interactive social decision–making in severe major depressive disorder. J. Affect. Disord. 137 , 98–105 (2012).

Harlé, K. M., Allen, J. J. B., Sanfey, A. G., Harlé, K. M. & Sanfey, A. G. The impact of depression on social economic decision making. J. Abnorm. Psychol. 119 , 440–446 (2010).

Unoka, Z., Seres, I., Aspán, N., Bódi, N. & Kéri, S. Trust game reveals restricted interpersonal transactions in patients with borderline personality disorder. J. Pers. Disord. 23 , 399–409 (2009).

King-Casas, B. et al . The rupture and repair of cooperation in borderline personality disorder. Science 321 , 806–810 (2008).

Koenigs, M., Kruepke, M. & Newman, J. P. Economic decision-making in psychopathy: A comparison with ventromedial prefrontal lesion patients. Neuropsychologia 48 , 2198–2204 (2010).

Raihani, N. J. & Bell, V. Paranoia and the social representation of others: a large-scale game theory approach. Scientific Reports 7 , 4544, https://doi.org/10.1038/s41598-017-04805-3 (2017).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Raihani, N. J. & Bell, V. Conflict and cooperation in paranoia: a large-scale behavioural experiment. Psychological Medicine 1–11, https://doi.org/10.1017/S0033291717003075 (2017).

Collins, L. G. & Swartz, K. Caregiver care. American family physician 83 , 1309–1317 (2011).

PubMed   Google Scholar  

Aledwood, T. et al . Data collection for mental health studies through digital platforms: requirements and design of a prototype. JMIR research protocols 6 (6), 1–11 (2017).

Blaga, O. M., Vasilescu, L. & Chereches, R. M. Use and effectiveness of behavioural economics in interventions for lifestyle risk factors of non-communicable diseases: a systematic review with policy implications. Perspectives in Public Health 20 (10), 1–11 (2017).

McPherson, K. E. et al . The association between social capital and mental health and behavioural problems in children and adolescents: an integrative systematic review. BMC psychology 2 , 7, https://doi.org/10.1186/2050-7283-2-7 (2014).

Article   PubMed   PubMed Central   Google Scholar  

Almedom, A. M. Social capital and mental health: An interdisciplinary review of primary evidence. Social Science and Medicine 61 , 943–964 (2005).

Berg, J., Dickhaut, J. & McCabe, K. Trust, reciprocity, and social history. Games Econ. Behav. 10 , 122–142 (1995).

Rapoport, A. & Chammah, A. M. Prisoner’s Dilemma (University of Michigan Press, 1965).

Vicens, J. et al . Resource heterogeneity leads to unjust effort distribution in climate change mitigation. Preprint at https://arxiv.org/abs/1709.02857 (2017).

Download references

Acknowledgements

We thank the community of patients, caregivers and families working within the Federació de Salut Mental Catalunya (Catalonia Mental Health Federation) for the enthusiasm and for their invaluable help in the design and realization of the experiments. We are also especially thankful to I Bonhoure for the necessary logistics to make the experiments possible, to F Español for contributing in the first steps in the experimental design, to M Poll for always giving us the institutional support from inside the Federation, to both E Ferrer and F Muñoz for building the bridge between us and the mental health ecosystem and to X Trabado for encouraging us to run this research. This work was partially supported by Federació de Salut Mental Catalunya; by MINEICO (Spain), Agencia Estatal de Investigación (AEI) and Fondo Europeo de Desarrollo Regional (FEDER) through grants FIS2013-47532-C3-1-P (JD), FIS2016-78904-C3-1-P (JD), FIS2013-47532-C3-2-P (JP), FIS2016-78904-C3-2-P (JP, AC); by Generalitat de Catalunya (Spain) through Complexity Lab Barcelona (contract no. 2014 SGR 608, JP) and through Secretaria d’Universitats i Recerca (contract no. 2013 DI 49, JD, JV); and by the EU through FET Open Project IBSEN (contract no. 662725, AS) and FET-Proactive Project DOLFINS (contract no. 640772, AS).

Author information

Authors and affiliations.

Departament de Física de la Matèria Condensada, Universitat de Barcelona, 08028, Barcelona, Spain

Anna Cigarini, Julián Vicens & Josep Perelló

Universitat de Barcelona Institute of Complex Systems UBICS, 08028, Barcelona, Spain

Departament d’Enginyeria Informàtica i Matemàtiques, Universitat Rovira i Virgili, 43007, Tarragona, Spain

Julián Vicens & Jordi Duch

Northwestern Institute on Complex Systems (NICO), Northwestern University, 60208, Evanston, IL, USA

Grupo Interdisciplinar de Sistemas Complejos (GISC), Unidad de Matemática, Modelización y Ciencia Computacional, Universidad Carlos III de Madrid, 28911, Leganés, Spain

Angel Sánchez

Unidad Mixta Interdisciplinar de Comportamiento y Complejidad Social (UMICCS) UC3M-UV-UZ, Universidad Carlos III de Madrid, 28911, Leganés, Spain

Institute UC3M-BS of Financial Big Data, Universidad Carlos III de Madrid, 28903, Getafe, Spain

Instituto de Biocomputación y Física de Sistemas Complejos (BIFI), Universidad de Zaragoza, 50009, Zaragoza, Spain

You can also search for this author in PubMed   Google Scholar

Contributions

J.D., A.S., and J.P. conceived the original idea for the experiment; J.V. and J.D. prepared the software for the final experimental setup; A.C. and J.V. analyzed the data; and all authors carried out the experiments, discussed the analysis results, and wrote the paper.

Corresponding author

Correspondence to Josep Perelló .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Supplementary information, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Cigarini, A., Vicens, J., Duch, J. et al. Quantitative account of social interactions in a mental health care ecosystem: cooperation, trust and collective action. Sci Rep 8 , 3794 (2018). https://doi.org/10.1038/s41598-018-21900-1

Download citation

Received : 15 November 2017

Accepted : 01 February 2018

Published : 28 February 2018

DOI : https://doi.org/10.1038/s41598-018-21900-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mental Health
• Community Social Capital
• Group Roles
• Dyadic Games
• Average Individual Contribution

This article is cited by

Gender-based pairings influence cooperative expectations and behaviours.

• Anna Cigarini
• Julián Vicens
• Josep Perelló

Scientific Reports (2020)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Advertisement

Quantitative needs assessment tools for people with mental health problems: a systematic scoping review

• Open access
• Published: 12 March 2024

Cite this article

You have full access to this open access article

• Irena Makivić   ORCID: orcid.org/0000-0003-2748-5522 1 ,
• Anja Kragelj 1 &
• Antonio Lasalvia   ORCID: orcid.org/0000-0001-9963-6081 2  

779 Accesses

Explore all metrics

Needs assessment in mental health is a complex and multifaceted process that involves different steps, from assessing mental health needs at the population or individual level to assessing the different needs of individuals or groups of people. This review focuses on quantitative needs assessment tools for people with mental health problems. Our aim was to find all possible tools that can be used to assess different needs within different populations, according to their diverse uses. A comprehensive literature search with the Boolean operators “Mental health” AND “Needs assessment” was conducted in the PubMed and PsychINFO electronic databases. The search was performed with the inclusion of all results without time or other limits. Only papers addressing quantitative studies on needs assessment in people with mental health problems were included. Additional articles were added through a review of previous review articles that focused on a narrower range of such needs and their assessment. Twenty-nine different need-assessment tools specifically designed for people with mental health problems were found. Some tools can only be used by professionals, some by patients, some even by caregivers, or a combination of all three. Within each recognized tool, there are different fields of needs, so they can be used for different purposes within the needs assessment process, according to the final research or clinical aims. The added value of this review is that the retrieved tools can be used for assessment at the individual level, research purposes or evaluation at the outcome level. Therefore, best needs assessment tool can be chosen based on the specific goals or focus of the related needs assessment.

Similar content being viewed by others

Access to health services among culturally and linguistically diverse populations in the Australian universal health care system: issues and challenges

Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period

The Biopsychosocial Approach: Towards Holistic, Person-Centred Psychiatric/Mental Health Nursing Practice

Avoid common mistakes on your manuscript.

Mental disorders are the largest contributor to the disease burden in Europe (Wykes et al., 2021 ), and mortality related to such conditions increases the overall economic burden (McGorry & Hamilton, 2016 ). Mental disorders affect various life domains, from physical health to daily living, friends, family situations, and education, and are associated with greater unemployment and economic problems (Wykes et al., 2021 ).

In order to plan and carry out successful mental health care, it is necessary to have a good mental health information system that also includes data about related needs (Wykes et al., 2021 ). When a need is identified, an action can be (re)organized to address it. Such action, based on the needs identified by the affected individuals, professionals or society, results in either satisfaction or dissatisfaction if the needs continue to be present (Endacott, 1997 ). Assessing needs might also be used to assess the adequacy and prioritization of mental health services at the population level (Ashaye et al., 2003 ; Hamid et al., 2009 ) as well as for the evaluation of mental health care (Hamid et al., 2009 ).

When considering mental health, a need represents a gap between what is and what should be (Witkin & Altschuld, 1995 ), and any changes that are made to the system should thus work to reduce this gap. There are various definitions of both “need” and “assessment” (Royse & Drude, 1982 ). Kahn (1969) considered needs from a social perspective to represent what someone requires in a broader bio-psycho-social context to be able to fully and productively participate in a social process (Royse & Drude, 1982 ). Brewin conceptualised needs (Lesage, 2017 ) as assessing what kind of social disability an individual has for professionals to be able to use an adequate model of care. Disability in this context is the result of interactions between people and the environment, and thus a disability can be seen as a lack of appropriate care models in relation to recognized needs. The concept of “need” in mental health care may be defined according to different points of view: a “normative need” is defined by professionals, while a “felt need” is what people with mental health problems experience and ask to be met (Endacott, 1997 ). What patients request and what they really need may differ, as they can only get what is available and provided at the system level, and what is the most beneficial for them in the current situation. Moreover, what they ask for is not always feasible. However, according to Bradshaw, what an individual requests is important and should be considered as felt needs (Endacott, 1997 ). Bearing in mind Maslow’s hierarchy of needs, only a combination of assessments from different points of view can provide a comprehensive needs assessment: needs assessed at the individual level from service users, their family members, caregivers, practitioners, and other professionals (Endacott, 1997 ). Indicators of needs at the individual level include functioning on different levels, symptoms, diagnoses, quality of life and, access to services (Aoun et al., 2004 ). Patient-centredness is vital to ensure the highest quality of care through monitoring performance (Kilbourne et al., 2018 ). Taking into account the patients’ perspective is also important to assess needs correctly, since such an assessment is more than just the professionals’ perception. An assessment of needs, as Thornicroft ( 1991 ) pointed out, provides care in the community with an emphasis on the provider-user relationship as a key component through which effective care is organized (Carter et al., 1995 ). According to Slade ( 1994 ), the concept of a need in mental health has no single correct definition, but it should rather be seen s a “socially-negotiated concept” (Thornicroft & Slade, 2002 ). Additionally, needs have to be assessed through the bio-psycho-social model (Makivić & Klemenc-Ketiš, 2022 ), including not just medical needs but also a wide array of social needs.

Initially, the assessment of needs (Balacki, 1988 ) in the community was seen as an approach using different forms of analysis to gain insights into the use of services, characteristics of people, incidence and prevalence rates and indicators to recognize crucial determinants that lead to the worsening of mental health. The assessment of mental health needs in Western societies began in 1775 with the analysis of public health data contained in the case registers (Royse & Drude, 1982 ). In the mid-1970s, with the beginning of the transition to care for mental health in the community (and the launch of community mental health service organizations), needs assessment was required within the evaluation process to help meet the patients’ needs. Needs assessment also represents a crucial part of mental health planning (Royse & Drude, 1982 ), where different needs must be considered, especially those felt by individuals. At the end of seventies, Kimmel pointed out that this area of needs assessment had no systematic procedures (Royse & Drude, 1982 ). However, several mental health needs assessment tools have been developed over the last thirty years.

The MRC Needs for Care Assessment (NFCAS) (by Brewin, 1987) was the first attempt to introduce a standardized assessment of the needs of the severely mentally ill (Lesage, 2017 ). Subsequently, a reduced version of the instrument applicable to common mental disorders was developed – i.e., the Needs for Care Assessment Schedule-Community version (NFCAS-C) (Bebbington et al., 1996 ). The shortened version of NFCAS was the Cardinal Needs Schedule (CNS), which is used to assess needs to address them with appropriate interventions (Marshall et al., 1995 ). Later the self-administered Perceived Needs for Care Questionnaire (PNCQ) was developed for use at the population level (Meadows et al., 2000 ), while in 1995 the Camberwell Assessment of Need (CAN) (Phelan et al., 1995 ) was published. After this time the focus shifted more to people-centred approaches, and therefore the assessment of needs also moved beyond psychiatric symptomatology to bring in “consumers”, i.e. patients and their caregivers. Other scales have also been used as needs assessment tools, such as the HoNOS scale (Joska & Flisher, 2005 ) which was designed to evaluate the clinical and social outcomes of mental health care.

Needs assessment is not always a clear and straightforward process with one approach and one goal. Therefore, different tools and approaches may be used to assess needs from different perspectives at different levels and with the help of different tools. The problem with using different techniques is that there is a lack of comparability and a consequent danger of not using the needs assessment outcome data as intended (Stewart, 1979 ); thus, it is important to have a good overview of the available tools.

To the best of our knowledge, only six reviews on needs assessment in people with mental health problems have been published to date (Davies et al., 2018 , 2019 ; Dobrzyńska et al., 2008b ; Joska & Flisher, 2005 ; Keulen-de Vos & Schepers, 2016 ; Lasalvia et al., 2000b ). Four additional reviews focused on the general needs or general health needs of people without mental health problems (Asadi-Lari & Gray, 2005 ; Carvacho et al., 2021 ; Lasalvia et al., 2000a ; Ravaghi et al., 2023 ), which was not focus group of our review. Finally, another article was considered inadequate for this study’s purposes, as it was published in Polish (as the one above) and is not a review paper (Dobrzyńska et al., 2008a ). None of the reviews published thus far have focused on the different assessment tools available for assessing the needs of people with different mental disorders. To date, no study has attempted to review all the available published studies on the various needs assessment processes to systematize the topic. The reviews mentioned above deal with only one specific population (patients with first-episode psychosis; forensic patients), or with specific needs (need for mental health services, supportive care needs, or individual needs for care). Thus, this study aimed to review all studies addressing needs assessment tools specifically designed for people with mental health problems, regardless of their diagnoses. The added value of this study is especially because of its wholeness in presenting different tools that can be used on different populations and by different groups. Thus this study may serve as a framework for starting different needs-assessment processes.

Search strategy

A comprehensive literature search using the Boolean operators “Mental health” AND “Needs assessment” was conducted in electronic bibliographic databases PubMed [Needs Assessment (Mesh Terms) AND Mental Health (Mesh Terms); Mental Health (Title/Abstract) AND Needs assessment (Title/Abstract);] and PsychINFO [Needs assessment AND Mental health in keywords; Needs assessment AND Mental health in Title; Needs assessment AND Mental health in Abstract]. Searching was carried out with the inclusion of all results without time or other limits in August 2021. The search strategy was based on the needs from a clinical context as well as some research priorities in the field of mental health. After the first systematic search we collected additional papers with an overview of six review articles (Davies et al., 2018 , 2019 ; Dobrzyńska et al., 2008b ; Joska & Flisher, 2005 ; Keulen-de Vos & Schepers, 2016 ; Lasalvia et al., 2000b ) and their results, and by searching PubMed within all connected articles. This was important since keywords changed over all this broad timeframe.

Inclusion and exclusion criteria

Our research exclusively focused on quantitative studies. We thus excluded all theoretical/conceptual articles, editorials, books, book commentaries or dissertations. Studies assessing the needs of patients with dementia and groups of people with physical and psychological disabilities were also excluded. We did not include papers related to 1) only general health (care), 2) other needs of the general population, 3) screening, prevalence, general diagnostic tools, and 4) tools for assessing caregivers’ needs. All those steps were done comprehensively by two researchers (IM, AK) independently. When there was a disagreement on the inclusion or exclusion of an article, both researchers looked at it again before reaching a consensus. We then manually added all relevant articles that could have been missed during the electronic search. We added articles that were cited within or were related with all the six mentioned reviews, but were not yet retrieved in the first search. These review articles were not included in the final number of all the articles examined in this study with the aim of exploring the different tools used for needs assessment of people with mental health problems. The aim of this process is to first obtain an overview of all the tools available, as this will make it possible to better use them within clinical settings, as well as for research and development purposes in order to plan a system or intervention that addresses the recognized needs (Fig.  1 ).

Concept of patient-centred care based on needs

Scoping studies, as Arksey and O'Malley ( 2005 ) mentioned, follow five steps, which we also took into consideration. First (step one) we identified the research question, which was “What are all different needs assessment tools that have been used in the population of people with mental health problems within different studies”. We then identified the relevant studies within recognised databases, as well as manually searching and adding the relevant articles (step two). We selected the appropriate studies (step three) as described within the search strategy process, with all inclusion and exclusion criteria. Finally, we presented the results (step four) in the chart flow in Fig.  2 , and Tables  1 , 2 and 3 , which corresponds to the concept of patient-centred care based on needs (Fig.  1 ). Because our focus was on different tools, we prepared the tables accordingly. There was no other relevant information in the original 242 articles to be presented at this occasion, other than those about the usage of different needs assessment tools, as this was the goal of the scoping review. The presentation of the results is based on the use of all recognized needs assessment tools, since geographical studies have been presented elsewhere (Makivić & Kragelj, 2023 ).

Research process within the databases

The analysis was multi-structured to provide an overview of all the recognized tools and the related time trends, country use and population of the most frequently used assessment methods.

The study selection process is shown in Fig.  2 . PubMed provided 578 records within the Mesh search and 537 within the title/abstract search, with after duplicates were removed this gave 1,090 results. Searching in PsychINFO provided 650 results from a search within the Abstract, 232 within Keywords and 1450 within Title; after combining these and removing duplicates, a total of 1,548 results were obtained.

The first selection was made within the final database (n = 2,638) by reading the abstracts and excluding all studies covering topics not relevant for this review. After this was completed, 166 articles remained. These were reviews and research articles covering the needs assessment of people with mental disorders (MD). After this, we eliminated review articles (n = 6) and used them for additional search to manually add all relevant articles that could have been missed during the electronic search, mainly because of the use of different keywords. Specifically, we added the articles that were cited within or were related to all the six mentioned reviews, but were not found in the first search (n = 82). After this process, a total of 242 articles were included in the final review.

Most studies addressing needs assessment tools retrieved with both electronic and manual searches were published in English (n = 231), although some were published in German (n = 3), Spanish (n = 3), and Italian (n = 2). Only one article each was published in Dutch, French and Turkish. Regarding the geographical distribution, most studies were published from European groups (n = 163), while 43 studies were conducted in America, 22 in Australia or New Zealand, 11 in Asia and only three in Africa. Some of the studies were published in collaboration among researchers from different countries. Regarding the publication period, the first studies on this issue were published in 1978, 52.9% of the studies were published from 2000 to 2012, and 66.1% had been published further by 2016.

Through the search performed in this study we found 29 different needs assessment tools, as shown in Table  1 in alphabetical order. We have made and additional search in order to find original sources and the information about the validation. Original sources for each of the recognized tools are listed in Supplementary information ( SI 1 ). Some tools, additional to those 29, were developed for the purposes of a single research study and its specific aims and the information about the validation were not available (n = 11), and thus we eliminated those tools at this point, although they will later be presented elsewhere in another study.

The retrieved tools and their respective constructs of need are presented in Table  2 . The various needs assessment tools are listed in alphabetical order. The tools are presented with regard to (1) who can answer the scale, (2) who the target population is, and (3) the domains addressed. Table 2 provides information on the various needs assessment tools, listed in alphabetical order. The tools are presented with regard to (1) who can answer the scale, (2) who the target population is, and (3) the domains addressed.

Service needs (Hamid et al., 2009 ) are defined as care requirements for prevention, treatment and rehabilitation. These needs can either be assessed by waiting lists or by only asking a simple question (e.g. “Do you think that you require any professional mental health services?”) along with the screening for mental and physical health problems (Yu et al., 2019 ) or social problems, with the help of the tools listed below. Moreover, there are different bio-psycho-social needs that are related to various mental health, physical health, and quality of life factors, as well as personal interests or abilities and social factors (Keulen-de Vos & Schepers, 2016 ), and these can be measured for different purposes. Social needs can be assessed by tools such as the Social Behavioral Schedule or REHAB Schedules, and therefore the need for rehabilitation can also be assessed (Hamid et al., 2009 ) using the comprehensive tools mentioned in our review.

Most of the needs assessment tools were self-completed by the patients (n = 85), completed by professionals (n = 41), or by combination of both (n = 78). Some tools were also completed by the patients and their caregivers (n = 12) or by the patients, caregivers, and professionals at the same time (n = 12). There were few studies where the researchers completed the needs assessment tool (n = 5). The majority of the tools were developed for assessing needs in an adult population with mental health problems (n = 193), either with severe mental disorders or with some other mental health diagnosis. Seventeen studies focused on an elderly population with mental health problems, and six on children with mental health problems. Some needs assessment tools for specific populations were found, such as tools for assessing the needs of forensic patients with mental health problems (n = 18), homeless people and migrants with a mental health diagnosis (n = 4), and mothers or pregnant women with a severe mental disorder (n = 1). In some studies, there was a combination of all these different populations and even people without a diagnosis, which we assigned to each of the mentioned groups.

In the second Supplementary information ( SI 2 ) there are reported the studies found in the literature search that used recognized needs assessment tools (n = 227). In this presentation some of the studies are not presented, namely those without validated tools (n = 11) as already mentioned and all articles using mentioned three different models (n = 4). In some studies, more tools have been used and in this case the study is counted within each tool in the total number of studies. Among the different needs-assessment tool, the CAN is mentioned as the most frequently used scale and, to the best of our knowledge, it has the highest number of different versions. The tools are presented based on their frequency of recognized use within this scoping review, from the most frequent to the least.

The recognized tools can be used in different contexts. Table 3 , groups the needs assessment tools according to their use at the care, research, and system levels.

This scoping review addressed all the published needs assessment tools specifically designed for use in mental health field. Nevertheless, some of the reviewed tools had also been used on the populations without a mental health diagnosis (Carvacho et al., 2021 ). Overall, we found twenty-nine different tools measuring needs in various mental health populations. The list of authors of the originally developed scales mentioned below are provided in the Supplementary information ( SI 1 ).

The reviewed literature highlights that the majority of needs assessment tools have been developed and used in Europe as the adoption of a community psychiatry model is relatively more widespread in this region than in other world regions; some tools, however, have been also used in America, Australia, and New Zealand.

Some scales had been developed with the aim to simplify or shorten previously published needs assessment tools, such as the Camberwell Assessment of Need (CAN) derived from the MRC Needs for Care Assessment Schedule. Similarly, the Difficulties and Needs Self-Assessment Tool was derived from the CAN, where some items are identical, some are a combination of several items of the CAN and some were added as new ones (on work, public places, family and friendship). Some tools, like the Montreal Assessment of Needs Questionnaire, were also developed from the CAN and had different aims, like enhancing data variability to broaden outcome measures for service planning, or simply because the organization of the related system is different and other tools are more appropriate. On the other hand, some tools are based on the CAN, but have been designed for use on a larger scale at the population level, like the Needs Assessment Scale. While most of the tools are used within health care services, the Resident Assessment Instrument Mental Health is a tool developed to support a seamless approach to person-centred health and social care. Some of the tools can also be used outside of the mental health field – such as the Child and Adolescent Needs and Strengths, which can be used in juvenile justice, intervention applications and child welfare – and the abovementioned CAN and others.

There are slightly different ideas regarding the needs and concepts about measuring needs. Many tools include a combination of needs assessed from different perspectives, such as the Bangor Assessment of Need Profile and the CAN. In some tools, like the Community Placement Questionnaire, it is predicted that various people rate the situation for one patient to eliminate any inaccuracies. On the other hand, some tools presented here, like the Self-Sufficiency Matrix, measure needs indirectly through self-sufficiency. When there is higher self-sufficiency for a certain life domain then there is less need presented for this area. Some tools, like Services Needed, Available, Planned, Offered, are complicated to use, since they include an investigation method with the review of the tool and assessment of the service use after the needs have been recognized. But this can be a good approach for the evaluation of the performance of community mental health centres about meeting the needs of their patients. Although we must bear in mind that such a tool is not directly transferable to every community mental health centre, as this depends on how each system is organized.

Needs can be evaluated according to different points of view, from patients themselves and their caregivers, as well as professionals. Studies show there are different outcomes based on the assessor (Lasalvia et al., 2000a , b , c ; Macpherson et al., 2003 ), and that professionals may see the needs differently to the users. Therefore, it is important not only what the tool is being used, but also who can complete it. Therefore, the most useful tools are the ones that can be used by various different people, so that the needs are assessed (also) from the patients’ standpoints (Larson et al., 2001 ).

Although the CAN is the most widely used tool, the research shows that sometimes there is not a very high agreement between staff and patients about needs, as was also found with the Health of the Nation Outcome Scales (HoNOS), which is the reason why some additional scales, such as the Profile of Community Psychiatry Clients, were developed. There are also some tools, such as the HoNOS, that indirectly measure needs for care, so they can be used as either a clinical or needs assessment tool.

Needs assessment tools are generally used by community psychiatry organizations and are also used to support changes to the organizations of countries’ related systems. The tools have already been used in order to assess the needs within clinical procedures, as well as at higher organizational levels in order to supplement services and direct programming (Royse & Drude, 1982 ). Different tools have good potential to evaluate community mental health services through assessing if patients’ needs have been met. Therefore, this study also aims at answering the question of which tool(s) can be most appropriate regarding different goals.

Within this review, we identified three systematic approaches to needs assessment which encompass different tools. The first is the DISC (Developing Individual Services in the Community) Framework (Smith, 1998 ), which includes the CAN and the Avon Self-Assessment Measure. The second is the Cumulative Needs for Care Monitor (Drukker et al., 2010 ), developed in order to choose the best treatment for each person. This one also uses the CAN and other more clinical tools and outcome measures (such as quality of life). The third is the Colorado Client Assessment Record (Ellis et al., 1984 ), which includes different measures of social functioning, such as the Denver Community of Mental Health Questionnaire, the Community Adjustment Profile, the Fort Logan Evaluation Screen, the Personal Role Skills Scale and the Global Assessment Scale.

This study has several strengths. First, we searched for as many tools and articles as possible. Second, we followed the standard rules of systematic and scoping reviews to present the data in a structured and non-biased manner: we thus searched for information extensively; the search was transparent and reproducible; the data were presented in a structured way. Finally, the scoping review was carried out, since the goal was not to compare and assess the quality of the evidence in the studies, but rather to review of all potential tools that can be used within the process of assessing the needs. Third, this study considered different populations, from severe mental disorders to other mental health problems, including addiction, which produced a strong overview of different tools and versions of the same tool used in other contexts. Fourth, the use of such tools also has a different basis depending on the goals of the system, so it can reflect the organization of care for mental health in a given country. The fifth strength of this work is that in addition to the original 242 articles within the review, we have also included all original sources for development of each of the 29 recognized tools.

This study also has some limitations. First, as the keywords are not same for every study, some studies could have been left out and therefore some tools might have been unrecognized. Second, our needs assessment review focuses on all people with mental health problems, even though the group of those with severe mental illness differs from the group with less severe mental health disorders. Therefore, no conclusion can be made on which tool is better for use in different population groups or disease severities. Third, we only included tools that assess the needs of people with mental health problems, although other tools for the general population could also potentially be useful. Fourth, some tools were developed and validated in only one country, so transferability is questionable or requires additional validation.

Since this scoping review provides insight into the evidence about the existence of different tools for needs assessment, it would also be valuable to conduct additional research on the level of each tool to see if it has already been validated and culturally adapted. To the best of our knowledge, the CAN is the most frequently used tool, and has been translated and adapted into more than 33 different languages (Phelan et al., 1995 ). Some of the tools reviewed in this study use items similar to the CAN, such as the Needs Assessment Scale (de Weert-van Oene et al., 2009 ). Some tools use the same items with a few additional ones, such as the Montreal Assessment of Needs Questionnaire (Tremblay et al., 2014 ), which shows even greater use of the CAN. Thus, the concepts in this latter tool are widely applied.

There are different fields in which certain needs must be addressed to deal with the mental health of the general population or the needs of the population with mental health problems, with the latter being our main focus. This review aimed to develop a tool for needs assessment that can be applied clinically and for research purposes. It is also vital to see what kind of tools can be used to assess needs for the purpose of a formative evaluation process, and the possibility of service development following the identification of actual needs (Makivić et al., 2021 ). Therefore, this article is valuable for a variety of final users, as it can be used by service providers at the level of health or social care, researchers, policymakers and other relevant stakeholders.

Moreover, it is also necessary to assess needs in the field of communication, especially targeting anti-stigma and anti-discrimination campaigns, and to assess the needs of educational systems (Kragelj et al., 2022 ) for the representation of mental health topics (Makivić et al., 2022 ). The use of different tools for assessing needs not only gives us the possibility of identifying such needs, but also establishes the possibility of meeting those needs when these tools are used within bio-psycho-socially oriented primary care or interdisciplinary-oriented mental health care. The assessment of needs at the individual level is important for the effective development of person-centred care plans (Martin et al., 2009 ). Patient-centred psychiatric practice is also needed to increase patient empowerment, which can be done with the help of a needs assessment process.

The review of all the tools for assessing different needs for people with mental health problems presented in this work is new, and therefore fills an important gap in the scientific knowledge of the needs assessment process in the field of mental health.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Aoun, S., Pennebaker, D., & Wood, C. (2004). Assessing population need for mental health care: A review of approaches and predictors. Mental Health Services Research, 6 (1), 33–46. https://doi.org/10.1023/b:mhsr.0000011255.10887.59

Article   PubMed   Google Scholar  

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8 (1), 19–32. https://doi.org/10.1080/1364557032000119616

Article   Google Scholar  

Asadi-Lari, M., & Gray, D. (2005). Health needs assessment tools: Progress and potential. International Journal of Technology Assessment in Health Care, 21 (3), 288–297. https://doi.org/10.1017/s0266462305050385

Ashaye, O. A., Livingston, G., & Orrell, M. W. (2003). Does standardized needs assessment improve the outcome of psychiatric day hospital care for older people? A randomized controlled trial. Aging & Mental Health, 7 (3), 195–199. https://doi.org/10.1080/1360786031000101166

Article   CAS   Google Scholar  

Balacki, M. F. (1988). Assessing mental health needs in the rural community: A critique of assessment approaches. Issues in Mental Health Nursing, 9 (3), 299–315. https://doi.org/10.3109/01612848809140931

Article   PubMed   CAS   Google Scholar  

Bebbington, P., Brewin, C. R., Marsden, L., & Lesage, A. (1996). Measuring the need for psychiatric treatment in the general population: the community version of the MRC Needs for Care Assessment. Psychological Medicine, 26 (2), 229–236. https://doi.org/10.1017/s0033291700034620

Carter, M. F., Crosby, C., Geerthuis, S., & Startup, M. (1995). A client-centered assessment of need needs assessment. Journal of Mental Health, 4 (4), 383–394. https://doi.org/10.1080/09638239550037433

Carvacho, R., Carrasco, M., Lorca, M. B. F., & Miranda-Castillo, C. (2021). Met and unmet needs of dependent older people according to the Camberwell Assessment of Need for the Elderly (CANE): A scoping review. Revista Espanola De Geriatria y Gerontologia, 56 (4), 225–235. https://doi.org/10.1016/j.regg.2021.02.004

Davies, E. L., Gordon, A. L., Pelentsov, L. J., Hooper, K. J., & Esterman, A. J. (2018). Needs of individuals recovering from a first episode of mental illness: A scoping review. International Journal of Mental Health Nursing, 27 (5), 1326–1343. https://doi.org/10.1111/inm.12518

Davies, E. L., Gordon, A. L., Pelentsov, L. J., & Esterman, A. J. (2019). The supportive care needs of individuals recovering from first episode psychosis: A scoping review. Perspectives in Psychiatric Care, 55 (1), 6–14. https://doi.org/10.1111/ppc.12259

de Weert-van Oene, G. H., Havenaar, J. M., & Schrijvers, A. J. (2009). Self-assessment of need for help in patients undergoing psychiatric treatment. Psychiatry Research, 167 (3), 221–230. https://doi.org/10.1016/j.psychres.2008.02.001

Dobrzyńska, E., Rymaszewska, J., & Kiejna, A. (2008a). CANSAS-Krótka Ocena Potrzeb Camberwell oraz inne narzedzia oceny potrzeb osób z zaburzeniami psychicznymi [CANSAS-Camberwell Assessment of Need Short Appraisal Schedule and other needs of persons with mental disorders assessment tools]. Psychiatria Polska, 42 (4), 525–532.

PubMed   Google Scholar  

Dobrzyńska, E., Rymaszewska, J., & Kiejna, A. (2008b). Potrzeby osób z zaburzeniami psychicznymi–definicje, przeglad badań [Needs of persons with mental disorders-definitions and literature review]. Psychiatria Polska, 42 (4), 515–524.

Drukker, M., Bak, M., Campo, J.à, Driessen, G., Van Os, J., & Delespaul, P. (2010). The cumulative needs for care monitor: a unique monitoring system in the south of the Netherlands. Social Psychiatry and Psychiatric Epidemiology , 45 (4), 475–485. https://doi.org/10.1007/s00127-009-0088-3

Ellis, R. H., Wilson, N. Z., & Foster, F. M. (1984). Statewide treatment outcome assessment in Colorado: The Colorado Client Assessment Record (CCAR). Community Mental Health Journal, 20 (1), 72–89. https://doi.org/10.1007/BF00754105

Endacott, R. (1997). Clarifying the concept of need: A comparison of two approaches to concept analysis. Journal of Advanced Nursing, 25 (3), 471–476. https://doi.org/10.1046/j.1365-2648.1997.1997025471.x

Hamid, W. A., Lewis-Cole, K., Holloway, F., & Silverman, M. (2009). Older people with enduring mental illness: A needs assessment tool. Psychiatric Bulletin, 33 (3), 91–95. https://doi.org/10.1192/pb.bp.107.017392

Joska, J., & Flisher, A. J. (2005). The assessment of need for mental health services. Social Psychiatry and Psychiatric Epidemiology, 40 (7), 529–539. https://doi.org/10.1007/s00127-005-0920-3

Keulen-de Vos, M., & Schepers, K. (2016). Needs assessment in forensic patients: A review of instrument suites. International Journal of Forensic Mental Health, 15 (3), 283–300. https://doi.org/10.1080/14999013.2016.1152614

Kilbourne, A. M., Beck, K., Spaeth-Rublee, B., Ramanuj, P., O’Brien, R. W., Tomoyasu, N., & Pincus, H. A. (2018). Measuring and improving the quality of mental health care: A global perspective. World Psychiatry: Official Journal of the World Psychiatric Association (WPA), 17 (1), 30–38. https://doi.org/10.1002/wps.20482

Kragelj, A., Pahor, M., Zaletel-Kragelj, L., & Makivic, I. (2022). Learning needs assessment among professional workers in community mental health centres in Slovenia: Study protocol. South Eastern European Journal of Public Health , Special Volume No. 1 . https://doi.org/10.11576/seejph-5478

Larson, A., Frkovic, I., van Kooten-Prasad, M., & Manderson, L. (2001). Mental health needs assessment in Australia’s Culturally Diverse Society. Transcultural Psychiatry, 38 (3), 333–347. https://doi.org/10.1177/136346150103800304

Lasalvia, A., Stefani, B., & Ruggeri, M. (2000a). I bisogni di cura nei pazienti psichiatrici: una revisione sistematica della letteratura. I. Concetti generali e strumenti di valutazione. I bisogni di servizi [Therapeutic needs in psychiatric patients: a systematic review of the literature. I. General concepts and assessment measures. Needs for services]. Epidemiologia e Psichiatria Sociale , 9 (3), 190–213. https://doi.org/10.1017/s1121189x00007879

Lasalvia, A., Stefani, B., & Ruggeri, M. (2000b). I bisogni di cura nei pazienti psichiatrici: una revisione sistematica della letteratura. II. I bisogni di cura individuali [Needs for care in psychiatric patients: a systematic review. II. Needs for care on individual]. Epidemiologia e Psichiatria Sociale , 9 (4), 282–307. https://doi.org/10.1017/s1121189x00008411

Lasalvia, A., Ruggeri, M., Mazzi, M. A., & Dall'Agnola, R. B. (2000). The perception of needs for care in staff and patients in community-based mental health services. The South-Verona Outcome Project 3. Acta Psychiatrica Scandinavica , 102 (5), 366–375. https://doi.org/10.1034/j.1600-0447.2000.102005366.x

Le Grand, D., Kessler, E., & Reeves, B. (1996). The avon mental health measure. Mental Health Review Journal, 1 (4), 31–32. https://doi.org/10.1108/13619322199600042

Lesage, A. (2017). Populational and individual perspective on needs. Epidemiology and Psychiatric Sciences, 26 (6), 609–611. https://doi.org/10.1017/S204579601700049X

Article   PubMed   PubMed Central   CAS   Google Scholar  

Macpherson, R., Varah, M., Summerfield, L., Foy, C., & Slade, M. (2003). Staff and patient assessments of need in an epidemiologically representative sample of patients with psychosis-staff and patient assessments of need. Social Psychiatry and Psychiatric Epidemiology, 38 (11), 662–667. https://doi.org/10.1007/s00127-003-0669-5

Makivić, I., & Klemenc-Ketiš, Z. (2022). Development and validation of the scale for measuring biopsychosocial approach of family physicians to their patients. Family Medicine and Community Health, 10 (2), e001407. https://doi.org/10.1136/fmch-2021-001407

Article   PubMed   PubMed Central   Google Scholar  

Makivić, I., & Kragelj, A. (2023). Needs assessment tools at the field of mental health: Countries presentation. V: Conference abstract book: 11th European Conference on Mental Health [S. l. ]: Evipro, Str. Ljubljana, Slovenia, p. 184.

Makivić, I., Švab, V., & Selak, Š. (2021). Mental health needs assessment during the COVID-19 Pandemic: Consensus based on Delphi Study. Frontiers in Public Health, 9 , 732539. https://doi.org/10.3389/fpubh.2021.732539

Makivić, I., Kragelj, A., & Selak, Š. (2022). Educational needs assessment in the field of mental health: Review of the tools. V: 10th European conference on mental health: Conference abstract boo k. Evipro, Str. Lisbon, Portugal, Helsinki, pp. 183–184.

Marshall, M., Hogg, L. I., Gath, D. H., & Lockwood, A. (1995). The cardinal needs schedule-a modified version of the MRC needs for care assessment schedule. Psychological Medicine, 25 (3), 605–617. https://doi.org/10.1017/s0033291700033511

Martin, L., Hirdes, J. P., Morris, J. N., Montague, P., Rabinowitz, T., & Fries, B. E. (2009). Validating the Mental Health Assessment Protocols (MHAPs) in the Resident Assessment Instrument Mental Health (RAI-MH). Journal of Psychiatric and Mental Health Nursing, 16 (7), 646–653. https://doi.org/10.1111/j.1365-2850.2009.01429.x

McGorry, P. D., & Hamilton, M. P. (2016). Stepwise expansion of evidence-based care is needed for mental health reform. The Medical Journal of Australia, 204 (9), 351–353. https://doi.org/10.5694/mja16.00120

Meadows, G., Harvey, C., Fossey, E., & Burgess, P. (2000). Assessing perceived need for mental health care in a community survey: Development of the Perceived Need for Care Questionnaire (PNCQ). Social Psychiatry and Psychiatric Epidemiology, 35 (9), 427–435. https://doi.org/10.1007/s001270050260

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18 (1), 143. https://doi.org/10.1186/s12874-018-0611-x

Phelan, M., Slade, M., Thornicroft, G., Dunn, G., Holloway, F., Wykes, T., Strathdee, G., Loftus, L., McCrone, P., & Hayward, P. (1995). The Camberwell assessment of need: The validity and reliability of an instrument to assess the needs of people with severe mental illness. The British Journal of Psychiatry: The Journal of Mental Science, 167 (5), 589–595. https://doi.org/10.1192/bjp.167.5.589

Ravaghi, H., Guisset, A. L., Elfeky, S., Nasir, N., Khani, S., Ahmadnezhad, E., & Abdi, Z. (2023). A scoping review of community health needs and assets assessment: Concepts, rationale, tools and uses. BMC Health Services Research, 23 (1), 44. https://doi.org/10.1186/s12913-022-08983-3

Royse, D., & Drude, K. (1982). Mental health needs assessment: Beware of false promises. Community Mental Health Journal, 18 (2), 97–106. https://doi.org/10.1007/BF00754454

Slade M. (1994). Needs assessment. Involvement of staff and users will help to meet needs. The British journal of psychiatry: the journal of mental science, 165 (3), 293–296. https://doi.org/10.1192/bjp.165.3.293

Smith, H. (1998). Needs assessment in mental health services: The DISC framework. Journal of Public Health Medicine, 20 (2), 154–160. https://doi.org/10.1093/oxfordjournals.pubmed.a024736

Stewart, R. (1979). The nature of needs assessment in community mental health. Community Mental Health Journal, 15 , 287–295. https://doi.org/10.1007/BF00778708

Thornicroft, G. (1991). The concept of case management for long-term mental illness. International Review of Psychiatry, 3 (1), 125–132. https://doi.org/10.3109/0954026910906752

Thornicroft, G., & Slade, M. (2002). Comparing needs assessed by staff and by service users: Paternalism or partnership in mental health? Epidemiologia e Psichiatria Sociale, 11 (3), 186–191. https://doi.org/10.1017/s1121189x00005704

Tremblay, J., Bamvita, J. M., Grenier, G., & Fleury, M. J. (2014). Utility of the Montreal assessment of need questionnaire for community mental health planning. The Journal of Nervous and Mental Disease, 202 (9), 677–687. https://doi.org/10.1097/NMD.0000000000000180

Witkin, B. R., & Altschuld, J. W. (1995). Planning and Conducting Needs Assessments: A Practical Guide . Sage Publications. https://doi.org/10.1177/002087289603900316

Wykes, T., Bell, A., Carr, S., Coldham, T., Gilbody, S., Hotopf, M., Johnson, S., Kabir, T., Pinfold, V., Sweeney, A., Jones, P. B., & Creswell, C. (2021). Shared goals for mental health research: what, why and when for the 2020s. Journal of Mental Health (Abingdon, England) , 1–9. Advance online publication. https://doi.org/10.1080/09638237.2021.1898552

Yu, J. Y., Dong, Z. Q., Liu, Y., Liu, Z. H., Chen, L., Wang, J., Huang, M. J., Mo, L. L., Luo, S. X., Wang, Y., Guo, W. J., He, N., Chen, R., & Zhang, L. (2019). Patient-doctor concordance of perceived mental health service needs in Chinese hospitalized patients: a cross-sectional study. Annals of Palliative Medicine , 8 (4), 442–450. https://doi.org/10.21037/apm.2019.08.06

Download references

This work was supported by the Slovenian Research Agency (project No.: Z3-2652).[Display Image Removed]. Project was funded by the Reseach agency. Only IM was paid from this. But the project didn't finance the outcomes and the goal of publishing this work was solely for the research purposes.

Author information

Authors and affiliations.

National Institute of Public Health, Ljubljana, Slovenia

Irena Makivić & Anja Kragelj

Section of Psychiatry, Department of Neuroscience, Biomedicine and Movement Sciences, University of Verona, Verona, Italy

Antonio Lasalvia

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization: [Irena Makivić]; Methodology: [Irena Makivić]; Formal analysis and investigation: [Irena Makivić], [Anja Kragelj]; Writing—original draft preparation: [Irena Makivić]; Writing—review and editing: [Irena Makivić], [Anja Kragelj], [Antonio Lasalvia]; Funding acquisition: [Irena Makivić]; Resources: [Irena Makivić]; Supervision: [Antonio Lasalvia].

Corresponding author

Correspondence to Irena Makivić .

Ethics declarations

There are no financial or non-financial interests that are directly or indirectly related to the work submitted for publication. All authors have no relevant financial or non-financial interests to disclose.

There was no need to apply for ethical committee or for informed consent, since there were no humans directly involved in this scoping review.

Embargoes on data sharing are permitted.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 432 KB)

Supplementary file2 (pdf 556 kb), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Makivić, I., Kragelj, A. & Lasalvia, A. Quantitative needs assessment tools for people with mental health problems: a systematic scoping review. Curr Psychol (2024). https://doi.org/10.1007/s12144-024-05817-9

Download citation

Accepted : 27 February 2024

Published : 12 March 2024

DOI : https://doi.org/10.1007/s12144-024-05817-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Needs assessment
• Mental health
• Patient-centred care
• Find a journal
• Publish with us
• Track your research

Examining the mental health of university students: A quantitative and qualitative approach to identifying prevalence, associations, stressors, and interventions

Affiliations.

• 1 Department of Dental Public Health and Behavioural Sciences, University of Missouri-Kansas City School of Dentistry, Kansas City, MO, USA.
• 2 Office of Research and Graduate Programs, University of Missouri-Kansas City School of Dentistry, Kansas City, MO, USA.
• PMID: 35380931
• DOI: 10.1080/07448481.2022.2057192

Objective To identify the prevalence of anxiety, depression, and suicidal ideation that would place university students at risk for mental health disorders. To explore the source of stressors and possible interventions that may benefit student mental health in a university setting.

Participants: University students (n = 483) who had been learning remotely due to the COVID-19 pandemic.

Methods: A mixed-methods cross-sectional survey was administered in 2020.

Results: Students were at an increased rate of depression, anxiety and suicidal ideation as compared to the general population. Female gender, lack of social support, living alone, being a first-generation college student and COVID-19 were significantly associated with mental health disorders. Stressors were identified and categorized into themes and interventions were recognized that may improve student well-being.

Conclusion: Students enrolled in university programs appear to experience significant amounts of anxiety, depression, and suicidal ideation. Additional mental health education, resources, and support is needed.

Keywords: Anxiety; COVID-19; college students; depression; suicidal ideation.

• Anxiety / epidemiology
• Anxiety / psychology
• Cross-Sectional Studies
• Depression* / epidemiology
• Depression* / psychology
• Mental Health*
• Students / psychology
• Suicidal Ideation
• Universities

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 10, Issue 12
• Impact of the COVID-19 pandemic on mental health and well-being of communities: an exploratory qualitative study protocol
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0003-0180-0213 Anam Shahil Feroz 1 , 2 ,
• Naureen Akber Ali 3 ,
• Noshaba Akber Ali 1 ,
• Ridah Feroz 4 ,
• Salima Nazim Meghani 1 ,
• Sarah Saleem 1
• 1 Community Health Sciences , Aga Khan University , Karachi , Pakistan
• 2 Institute of Health Policy, Management and Evaluation , University of Toronto , Toronto , Ontario , Canada
• 3 School of Nursing and Midwifery , Aga Khan University , Karachi , Pakistan
• 4 Aga Khan University Institute for Educational Development , Karachi , Pakistan
• Correspondence to Ms Anam Shahil Feroz; anam.sahyl{at}gmail.com

Introduction The COVID-19 pandemic has certainly resulted in an increased level of anxiety and fear in communities in terms of disease management and infection spread. Due to fear and social stigma linked with COVID-19, many individuals in the community hide their disease and do not access healthcare facilities in a timely manner. In addition, with the widespread use of social media, rumours, myths and inaccurate information about the virus are spreading rapidly, leading to intensified irritability, fearfulness, insomnia, oppositional behaviours and somatic complaints. Considering the relevance of all these factors, we aim to explore the perceptions and attitudes of community members towards COVID-19 and its impact on their daily lives and mental well-being.

Methods and analysis This formative research will employ an exploratory qualitative research design using semistructured interviews and a purposive sampling approach. The data collection methods for this formative research will include indepth interviews with community members. The study will be conducted in the Karimabad Federal B Area and in the Garden (East and West) community settings in Karachi, Pakistan. The community members of these areas have been selected purposively for the interview. Study data will be analysed thematically using NVivo V.12 Plus software.

Ethics and dissemination Ethical approval for this study has been obtained from the Aga Khan University Ethical Review Committee (2020-4825-10599). The results of the study will be disseminated to the scientific community and to the research subjects participating in the study. The findings will help us explore the perceptions and attitudes of different community members towards the COVID-19 pandemic and its impact on their daily lives and mental well-being.

• mental health
• public health

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2020-041641

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

The mental health impact of the COVID-19 pandemic is likely to last much longer than the physical health impact, and this study is positioned well to explore the perceptions and attitudes of community members towards the pandemic and its impact on their daily lives and mental well-being.

This study will guide the development of context-specific innovative mental health programmes to support communities in the future.

One limitation is that to minimise the risk of infection all study respondents will be interviewed online over Zoom and hence the authors will not have the opportunity to build rapport with the respondents or obtain non-verbal cues during interviews.

The COVID-19 pandemic has affected almost 180 countries since it was first detected in Wuhan, China in December 2019. 1 2 The COVID-19 outbreak has been declared a public health emergency of international concern by the WHO. 3 The WHO estimates the global mortality to be about 3.4% 4 ; however, death rates vary between countries and across age groups. 5 In Pakistan, a total of 10 880 cases and 228 deaths due to COVID-19 infection have been reported to date. 6

The worldwide COVID-19 pandemic has not only incurred massive challenges to the global supply chains and healthcare systems but also has a detrimental effect on the overall health of individuals. 7 The pandemic has led to lockdowns and has created destructive impact on the societies at large. Most company employees, including daily wage workers, have been prohibited from going to their workplaces or have been asked to work from home, which has caused job-related insecurities and financial crises in the communities. 8 Educational institutions and training centres have also been closed, which resulted in children losing their routine of going to schools, studying and socialising with their peers. Delay in examinations is likewise a huge stressor for students. 8 Alongside this, parents have been struggling with creating a structured milieu for their children. 9 COVID-19 has hindered the normal routine life of every individual, be it children, teenagers, adults or the elderly. The crisis is engendering burden throughout populations and communities, particularly in developing countries such as Pakistan which face major challenges due to fragile healthcare systems and poor economic structures. 10

The COVID-19 pandemic has certainly resulted in an increased level of anxiety and fear in communities in terms of disease management and infection spread. 8 Further, the highly contagious nature of COVID-19 has also escalated confusion, fear and panic among community residents. Moreover, social distancing is often an unpleasant experience for community members and for patients as it adds to mental suffering, particularly in the local setting where get-togethers with friends and families are a major source of entertainment. 9 Recent studies also showed that individuals who are following social distancing rules experience loneliness, causing a substantial level of distress in the form of anxiety, stress, anger, misperception and post-traumatic stress symptoms. 8 11 Separation from family members, loss of autonomy, insecurity over disease status, inadequate supplies, inadequate information, financial loss, frustration, stigma and boredom are all major stressors that can create drastic impact on an individual’s life. 11 Due to fear and social stigma linked with COVID-19, many individuals in the community hide their disease and do not access healthcare facilities in a timely manner. 12 With the widespread use of social media, 13 rumours, myths and inaccurate information about COVID-19 are also spreading rapidly, not only among adults but are also carried on to children, leading to intensified irritability, fearfulness, insomnia, oppositional behaviours and somatic complaints. 9 The psychological symptoms associated with COVID-19 at the community level are also manifested as anxiety-driven panic buying, resulting in exhaustion of resources from the market. 14 Some level of panic also dwells in the community due to the unavailability of essential protective equipment, particularly masks and sanitisers. 15 Similarly, mental health issues, including depression, anxiety, panic attacks, psychotic symptoms and even suicide, were reported during the early severe acute respiratory syndrome outbreak. 16 17 COVID-19 is likely posing a similar risk throughout the world. 12

The fear of transmitting the disease or a family member falling ill is a probable mental function of human nature, but at some point the psychological fear of the disease generates more anxiety than the disease itself. Therefore, mental health problems are likely to increase among community residents during an epidemic situation. Considering the relevance of all these factors, we aim to explore the perceptions and attitudes towards COVID-19 among community residents and the impact of these perceptions and attitude on their daily lives and mental well-being.

Methods and analysis

Study design.

This study will employ an exploratory qualitative research design using semistructured interviews and a purposive sampling approach. The data collection methods for this formative research will include indepth interviews (IDIs) with community members. The IDIs aim to explore perceptions of community members towards COVID-19 and its impact on their mental well-being.

Study setting and study participants

The study will be conducted in two communities in Karachi City: Karimabad Federal B Area Block 3 Gulberg Town, and Garden East and Garden West. Karimabad is a neighbourhood in the Karachi Central District of Karachi, Pakistan, situated in the south of Gulberg Town bordering Liaquatabad, Gharibabad and Federal B Area. The population of this neighbourhood is predominantly Ismailis. People living here belong mostly to the middle class to the lower middle class. It is also known for its wholesale market of sports goods and stationery. Garden is an upmarket neighbourhood in the Karachi South District of Karachi, Pakistan, subdivided into two neighbourhoods: Garden East and Garden West. It is the residential area around the Karachi Zoological Gardens; hence, it is popularly known as the ‘Garden’ area. The population of Garden used to be primarily Ismailis and Goan Catholics but has seen an increasing number of Memons, Pashtuns and Baloch. These areas have been selected purposively because the few members of these communities are already known to one of the coinvestigators. The coinvestigator will serve as a gatekeeper for providing entrance to the community for the purpose of this study. Adult community members of different ages and both genders will be interviewed from both sites, as mentioned in table 1 . Interview participants will be selected following the eligibility criteria.

• View inline

Study participants for indepth interviews

IDIs with community members

We will conduct IDIs with community members to explore the perceptions and attitudes of community members towards COVID-19 and its effects on their daily lives and mental well-being. IDI participants will be identified via the community WhatsApp group, and will be invited for an interview via a WhatsApp message or email. Consent will be taken over email or WhatsApp before the interview begins, where they will agree that the interview can be audio-recorded and that written notes can be taken. The interviews will be conducted either in Urdu or in English language, and each interview will last around 40–50 min. Study participants will be assured that their information will remain confidential and that no identifying features will be mentioned on the transcript. The major themes will include a general discussion about participants’ knowledge and perceptions about the COVID-19 pandemic, perceptions on safety measures, and perceived challenges in the current situation and its impact on their mental well-being. We anticipate that 24–30 interviews will be conducted, but we will cease interviews once data saturation has been achieved. Data saturation is the point when no new themes emerge from the additional interviews. Data collection will occur concurrently with data analysis to determine data saturation point. The audio recordings will be transcribed by a transcriptionist within 24 hours of the interviews.

An interview guide for IDIs is shown in online supplemental annex 1 .

Supplemental material

Eligibility criteria.

The following are the criteria for inclusion and exclusion of study participants:

Inclusion criteria

Residents of Garden (East and West) and Karimabad Federal B Area of Karachi who have not contracted the disease.

Exclusion criteria

Those who refuse to participate in the study.

Those who have experienced COVID-19 and are undergoing treatment.

Those who are suspected for COVID-19 and have been isolated/quarantined.

Family members of COVID-19-positive cases.

Data collection procedure

A semistructured interview guide has been developed for community members. The initial questions on the guide will help to explore participants’ perceptions and attitudes towards COVID-19. Additional questions on the guide will assess the impact of these perceptions and attitude on the daily lives and mental health and well-being of community residents. All semistructured interviews will be conducted online via Zoom or WhatsApp. Interviews will be scheduled at the participant’s convenient day and time. Interviews are anticipated to begin on 1 December 2020.

Patient and public involvement

No patients were involved.

Data analysis

We will transcribe and translate collected data into English language by listening to the audio recordings in order to conduct a thematic analysis. NVivo V.12 Plus software will be used to import, organise and explore data for analysis. Two independent researchers will read the transcripts at various times to develop familiarity and clarification with the data. We will employ an iterative process which will help us to label data and generate new categories to identify emergent themes. The recorded text will be divided into shortened units and labelled as a ‘code’ without losing the main essence of the research study. Subsequently, codes will be analysed and merged into comparable categories. Lastly, the same categories will be grouped into subthemes and final themes. To ensure inter-rater reliability, two independent investigators will perform the coding, category creation and thematic analyses. Discrepancies between the two investigators will be resolved through consensus meetings to reduce researcher bias.

Ethics and dissemination

Study participants will be asked to provide informed, written consent prior to participation in the study. The informed consent form can be submitted by the participant via WhatsApp or email. Participants who are unable to write their names will be asked to provide a thumbprint to symbolise their consent to participate. Ethical approval for this study has been obtained from the Aga Khan University Ethical Review Committee (2020-4825-10599). The study results will be disseminated to the scientific community and to the research subjects participating in the study. The findings will help us explore the perceptions and attitudes of different community members towards the COVID-19 pandemic and its impact on their daily lives and mental well-being.

The findings of this study will help us to explore the perceptions and attitudes towards the COVID-19 pandemic and its impact on the daily lives and mental well-being of individuals in the community. Besides, an indepth understanding of the needs of the community will be identified, which will help us develop context-specific innovative mental health programmes to support communities in the future. The study will provide insights into how communities are managing their lives under such a difficult situation.

• World Health Organization
• Nielsen-Saines K , et al
• Worldometer
• Ebrahim SH ,
• Gozzer E , et al
• Snoswell CL ,
• Harding LE , et al
• Nargis Asad
• van Weel C ,
• Qidwai W , et al
• Brooks SK ,
• Webster RK ,
• Smith LE , et al
• Tripathy S ,
• Kar SK , et al
• Schwartz J ,
• Maunder R ,

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1

ASF and NAA are joint first authors.

Contributors ASF and NAA conceived the study. ASF, NAA, RF, NA, SNM and SS contributed to the development of the study design and final protocols for sample selection and interviews. ASF and NAA contributed to writing the manuscript. All authors reviewed and approved the final version of the paper.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

Mental Health Counseling

• Find Articles
• Search Strategies
• Find Journals
• Web Resources
• Qualitative vs. Quantitative Research
• Evidence Based Practice Research
• Citation: APA

Mixed Methods Research

As its name suggests, mixed methods research involves using elements of both quantitative and qualitative research methods. Using mixed methods, a researcher can more fully explore a research question and provide greater insight. 

Need to find quantitative or qualitative research?

The CINAHL and PsycINFO databases both allow for the application of filters that will yield results that are either qualitative or quantitative in nature. 

For detailed information about how to do that in CINAHL  or PsycINFO, visit the Quantitative and Qualitative LibGuide found here.   

What is Qualitative Research?

Quantitative research gathers data that can be measured numerically and analyzed mathematically. Quantitative research attempts to answer research questions through the quantification of data. 

Indicators of quantitative research include:

contains statistical analysis 

large sample size 

objective - little room to argue with the numbers 

types of research: descriptive studies, exploratory studies, experimental studies, explanatory studies, predictive studies, clinical trials 

What is Quantitative Research?

Qualitative research is based upon data that is gathered by observation. Qualitative research articles will attempt to answer questions that cannot be measured by numbers but rather by perceived meaning. Qualitative research will likely include interviews, case studies, ethnography, or focus groups. 

Indicators of qualitative research include:

interviews or focus groups 

small sample size 

subjective - researchers are often interpreting meaning 

methods used: phenomenology, ethnography, grounded theory, historical method, case study 

• << Previous: Web Resources
• Next: Evidence Based Practice Research >>
• Last Updated: Mar 27, 2024 2:05 PM
• URL: https://libguides.hofstra.edu/MentalHealthCounseling

This site is compliant with the W3C-WAI Web Content Accessibility Guidelines HOFSTRA UNIVERSITY Hempstead, NY 11549-1000 (516) 463-6600 © 2000-2009 Hofstra University

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Topic collections
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 13, Issue 2
• Qualitative Research Methods in Mental Health
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Sarah Peters
• Correspondence to : Dr Sarah Peters, School of Psychological Sciences, The University of Manchester, Coupland Building 1, Oxford Road M13 9PL, UK; sarah.peters{at}manchester.ac.uk

As the evidence base for the study of mental health problems develops, there is a need for increasingly rigorous and systematic research methodologies. Complex questions require complex methodological approaches. Recognising this, the MRC guidelines for developing and testing complex interventions place qualitative methods as integral to each stage of intervention development and implementation. However, mental health research has lagged behind many other healthcare specialities in using qualitative methods within its evidence base. Rigour in qualitative research raises many similar issues to quantitative research and also some additional challenges. This article examines the role of qualitative methods within mental heath research, describes key methodological and analytical approaches and offers guidance on how to differentiate between poor and good quality qualitative research.

https://doi.org/10.1136/ebmh.13.2.35

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

The trajectory of qualitative methods in mental health research

Qualitative methodologies have a clear home within the study of mental health research. Early and, arguably, seminal work into the study of mental illnesses and their management was based on detailed observation, moving towards theory using inductive reasoning. Case studies have been long established in psychiatry to present detailed analysis of unusual cases or novel treatments. Participant observation was the principle method used in Goffman's seminal study of psychiatric patients in asylums that informed his ideas about the institutionalising and medicalising of mental illness by medical practice. 1 However, the 20th century saw the ‘behaviourist revolution’, a movement where quantification and experimentation dominated. Researchers sought to identify cause and effects, and reasoning became more deductive – seeking to use data to confirm theory. The study of health and illness was determined by contemporary thinking about disease, taking a biomedical stance. Psychologists and clinical health researchers exploited natural science methodologies, attempting to measure phenomenon in their smallest entities and do so as objectively as possible. This reductionist and positivist philosophy shaped advances in research methods and meant that qualitative exploration failed to develop as a credible scientific approach. Indeed, ‘objectivity’ and the ‘discovery of truth’ have become synonymous with ‘scientific enquiry’ and qualitative methods are easily dismissed as ‘anecdotal’. The underlying epistemology of this approach chimes well with medical practice for which training is predominately in laboratory and basic sciences (such as physics and chemistry) within which the discourse of natural laws dominate. To this end, research in psychiatry still remains overwhelmingly quantitative. 2

Underlying all research paradigms are assumptions. However, most traditional researchers remain unaware of these until they start to use alternative paradigms. Key assumptions of quantitative research are that facts exist that can be quantified and measured and that these should be examined, as far as possible, objectively, partialling out or controlling for the context within which they exist. There are research questions within mental health where this approach can hold: where phenomenon of interest can be reliably and meaningfully quantified and measured, it is feasible to use data to test predictions and examine change. However, for many questions these assumptions prove unsatisfying. It is often not possible or desirable to try and create laboratory conditions for the research; indeed it would be ecologically invalid to do so. For example, to understand the experience of an individual who has been newly diagnosed with schizophrenia, it is clearly important to consider the context within which they live, their family, social grouping and media messages they are exposed to. Table 1 depicts the key differences between the two methodological approaches and core underlying assumptions for each.

• View inline

Comparison of underlying assumptions of quantitative and qualitative research approaches

It should be cautioned that it is easy to fall into the trap of categorising studies as either quantitative or qualitative. The two traditions are often positioned within the literature as opposing and in conflict. This division is unhelpful and likely to impede methodological advancement. Though, undeniably, there are differences in the two approaches to research, there are also many exceptions that expose this dichotomy to be simplistic: some qualitative studies seek to test a priori hypotheses, and some quantitative studies are atheoretical and exploratory. 3 Hence it is more useful to consider research methodologies as lying along a spectrum and that researchers should be familiar with the full range of methodologies, so that a method is chosen according to the research question rather than the researcher's ability.

Rationale for qualitative methods in current mental health research

There are a number of scientific, practical and ethical reasons why mental health is an area that can particularly benefit from qualitative enquiry. Mental health research is complex. Health problems are multifactorial in their aetiology and the consequences they have on the individual, families and societies. Management can involve self-help, pharmacological, educative, social and psychotherapeutic approaches. Services involved are often multidisciplinary and require liaison between a number of individuals including professionals, service-users and relatives. Many problems are exacerbated by poor treatment compliance and lack of access to, or engagement with, appropriate services. 4

Engagement with mental health research can also be challenging. Topics may be highly sensitive or private. Individuals may have impaired capacity or be at high risk. During the research process there may be revelations of suicidal ideation or criminal activity. Hence mental health research can raise additional ethical issues. In other cases scepticism of services makes for reluctant research participants. However, if we accept the case that meaningful research can be based in subjective enquiry then qualitative methods provide a way of giving voice to participants. Qualitative methods offer an effective way of involving service-users in developing interventions for mental health problems 5 ensuring that the questions asked are meaningful to individuals. This may be particularly beneficial if participants are stakeholders, for example potential users of a new service.

Qualitative methods are valuable for individuals who have limited literacy skills who struggle with pencil and paper measures. For example qualitative research has proved fruitful in understanding children's concepts of mental illness and associated services. 6

How qualitative enquiry is used within mental health research

There are a range of types of research question where qualitative methods prove useful – from the development and testing of theory, to the piloting and establishing efficacy of treatment approaches, to understanding issues around translation and implementation into routine practice. Each is discussed in turn.

Development and testing of theory

Qualitative methods are important in exploratory work and in generating understanding of a phenomenon, stimulating new ideas or building new theory. For example, stigma is a concept that is recognised as a barrier to accessing services and also an added burden to mental health. A focus-group study sought to understand the meaning of stigma from the perspectives of individuals with schizophrenia, their relatives and health professionals. 7 From this they developed a four-dimensional theory which has subsequently informed interventions to reduce stigma and discrimination that target not only engagement with psychiatric services but also interactions with the public and work. 7

Development of tools and measures

Qualitative methods access personal accounts, capturing how individuals talk about a lived experience. This can be invaluable for designing new research tools. For example, Mavaddat and colleagues used focus groups with 56 patients with severe or common mental health problems to explore their experiences of primary care management. 8 Nine focus groups were conducted and analysis identified key themes. From these, items were generated to form a Patient Experience Questionnaire, of which the psychometric properties were subsequently examined quantitatively in a larger sample. Not only can dimensions be identified, the rich qualitative data provide terminology that is meaningful to service users that can then be incorporated into question items.

Development and testing of interventions

As we have seen, qualitative methods can inform the development of new interventions. The gold-standard methodology for investigating treatment effectiveness is the randomised controlled trial (RCT), with the principle output being an effect size or demonstration that the primary outcome was significantly improved for participants in the intervention arm compared with those in the control/comparison arm. Nevertheless, what will be familiar for researchers and clinicians involved in trials is that immense research and clinical learning arises from these substantial, often lengthy and expensive research endeavours. Qualitative methods provide a means to empirically capture these lessons, whether they are about recruitment, therapy training/supervision, treatment delivery or content. These data are essential to improve the feasibility and acceptability of further trials and developing the intervention. Conducting qualitative work prior to embarking on an RCT can inform the design, delivery and recruitment, as well as engage relevant stakeholders early in the process; all of these can prevent costly errors. Qualitative research can also be used during a trial to identify reasons for poor recruitment: in one RCT, implementing findings from this type of investigation led to an increased randomisation rate from 40% to 70%. 9

Nesting qualitative research within a trial can be viewed as taking out an insurance policy as data are generated which can later help explain negative or surprising findings. A recent trial of reattribution training for GPs to manage medically unexplained symptoms demonstrated substantial improvements in GP consultation behaviour. 10 However, effects on clinical outcomes were counterintuitive. A series of nested qualitative studies helped shed light as to why this was the case: patients' illness models were complex, and they resisted engaging with GPs (who they perceived as having more simplistic and dualistic understanding) because they were anxious it would lead to non-identification or misdiagnosis of any potential future disease 11 , an issue that can be addressed in future interventions. Even if the insights are unsurprising to those involved in the research, the data collected have been generated systematically and can be subjected to peer review and disseminated. For this reason, there is an increasing expectation from funding bodies that qualitative methodologies are integral to psychosocial intervention research.

Translation and implementation into clinical practice

Trials provide limited information about how treatments can be implemented into clinical practice or applied to another context. Psychological interventions are more effective when delivered within trial settings by experts involved in their development than when they are delivered within clinical settings. 12 Qualitative methods can help us understand how to implement research findings into routine practice. 13

Understanding what stakeholders value about a service and what barriers exist to its uptake is another evidence base to inform clinicians' practice. Relapse prevention is an effective psychoeducation approach that helps individuals with bipolar disorder extend time to relapse. Qualitative methodologies identified which aspects of the intervention service-users and care-coordinators value, and hence, are likely to utilise in routine care. 14 The intervention facilitated better understanding of bipolar disorder (by both parties), demonstrating, in turn, a rationale for medication. Patients discovered new, empowering and less socially isolated ways of managing their symptoms, which had important impacts on interactions with healthcare staff and family members. Furthermore, care-coordinators' reported how they used elements of the intervention when working with clients with other diagnoses. The research also provided insights as to where difficulties may occur when implementing a particular intervention into routine care. For example, for care-coordinators this proved a novel way of working with clients that was more emotionally demanding, thus highlighting the need for supervision and managerial support. 14

Beginners guide to qualitative approaches: one size doesn't fit all

Just as there is a range of quantitative research designs and statistical analyses to choose from, so there are many types of qualitative methods. Choosing a method can be daunting to an inexperienced or beginner-level qualitative researcher, for it requires engaging with new terms and ways of thinking about knowledge. The following summary sets out analytic and data-generation approaches that are used commonly in mental health research. It is not intended to be comprehensive and is provided only as a point of access/familiarisation to researchers less familiar with the literature.

Data generation

Qualitative data are generated in several ways. Most commonly, researchers seek a sample and conduct a series of individual in-depth interviews, seeking participants' views on topics of interest. Typically these last upwards of 45 min and are organised on the basis of a schedule of topics identified from the literature or pilot work. This does not act as a questionnaire, however; rather, it acts as a flexible framework for exploring areas of interest. The researcher combines open questions to elicit free responses, with focused questions for probing and prompting participants to provide effective responses. Usually interviews are audio-recorded and transcribed verbatim for subsequent analysis.

As interviews are held in privately, and on one-to-one basis, they provide scope to develop a trusting relationship so that participants are comfortable disclosing socially undesirable views. For example, in a study of practice nurses views of chronic fatigue syndrome, some nurses described patients as lazy or illegitimate – a view that challenges the stereotype of a nursing professional as a sympathetic and caring person. 15 This gives important information about the education and supervision required to enable or train general nurses to ensure that they are capable of delivering psychological interventions for these types of problems.

Alternatively, groups of participants are brought together for a focus group, which usually lasts for 2 hours. Although it is tempting to consider focus groups as an efficient way of acquiring data from several participants simultaneously, there are disadvantages. They are difficult to organise for geographically dispersed or busy participants, and there are compromises to confidentiality, particularly within ‘captive’ populations (eg, within an organisation individuals may be unwilling to criticise). Group dynamics must be considered; the presence of a dominant or self-professed expert can inhibit the group and, therefore, prevent useful data generation. When the subject mater is sensitive, individuals may be unwilling to discuss experiences in a group, although it often promotes a shared experience that can be empowering. Most of these problems are avoided by careful planning of the group composition and ensuring the group is conducted by a highly skilled facilitator. Lester and colleagues 16 used focus-group sessions with patients and health professionals to understand the experience of dealing with serious mental illness. Though initially participants were observed via focus-group sessions that used patient-only and health professional only groups, subsequently on combined focus groups were used that contained both patients and health professionals. 16 The primary advantage of focus groups is that they enable generation of data about how individuals discuss and interact about a phenomenon; thus, a well-conducted focus group can be an extremely rich source of data.

A different type of data are naturally occurring dialogue and behaviours. These may be recorded through observation and detailed field notes (see ethnography in Table 2 ) or analysed from audio/ video-recordings. Other data sources include texts, for example, diaries, clinical notes, Internet blogs and so on. Qualitative data can even be generated through postal surveys. We thematically analysed responses to an open-ended question set within a survey about medical educators' views of behavioural and social sciences (BSS). 17 From this, key barriers to integrating BSS within medical training were identified, which included an entrenched biomedical mindset. The themes were analysed in relation to existing literature and revealed that despite radical changes in medical training, the power of the hidden curriculum persists. 17

Key features of a range of analytical approaches used within mental health research

Analysing qualitative data

Researchers bring a wide range of analytical approaches to the data. A comprehensive and detailed discussion of the philosophy underlying different methods is beyond the scope of this paper; however, a summary of the key analytical approaches used in mental health research are provided in Table 2 . An illustrative example is provided for each approach to offer some insight into the commonalities and differences between methodologies. The procedure for analysis for all methods involves successive stages of data familiarisation/immersion, followed by seeking and reviewing patterns within the data, which may then be defined and categorized as specific themes. Researchers move back and forth between data generation and analysis, confirming or disconfirming emerging ideas. The relationship of the analysis to theory-testing or theory-building depends on the methodology used.

Some approaches are more common in healthcare than others. Interpretative phenomenological (lPA) analysis and thematic analysis have proved particularly popular. In contrast, ethnographic research requires a high level of researcher investment and reflexivity and can prove challenging for NHS ethic committees. Consequently, it remains under used in healthcare research.

Recruitment and sampling

Quantitative research is interested in identifying the typical, or average. By contrast, qualitative research aims to discover and examine the breadth of views held within a community. This includes extreme or deviant views and views that are absent. Consequently, qualitative researchers do not necessarily (though in some circumstances they may) seek to identify a representative sample. Instead, the aim may be to sample across the range of views. Hence, qualitative research can comment on what views exist and what this means, but it is not possible to infer the proportions of people from the wider population that hold a particular view.

However, sampling for a qualitative study is not any less systematic or considered. In a quantitative study one would take a statistical approach to sampling, for example, selecting a random sample or recruiting consecutive referrals, or every 10th out-patient attendee. Qualitative studies, instead, often elect to use theoretical means to identify a sample. This is often purposive; that is, the researcher uses theoretical principles to choose the attributes of included participants. Healey and colleagues conducted a study to understand the reasons for individuals with bipolar disorder misusing substances. 18 They sought to include participants who were current users of each substance group, and the recruitment strategy evolved to actively target specific cases.

Qualitative studies typically use far smaller samples than quantitative studies. The number varies depending on the richness of the data yielded and the type of analytic approach that can range from a single case to more than 100 participants. As with all research, it is unethical to recruit more participants than needed to address the question at hand; a qualitative sample should be sufficient for thematic saturation to be achieved from the data.

Ensuring that findings are valid and generalisable

A common question from individuals new to qualitative research is how can findings from a study of few participants be generalised to the wider population? In some circumstances, findings from an individual study (quantitative or qualitative) may have limited generalisability; therefore, more studies may need to be conducted, in order to build local knowledge that can then be tested or explored across similar groups. 4 However, all qualitative studies should create new insights that have theoretical or clinical relevance which enables the study to extend understanding beyond the individual participants and to the wider population. In some cases, this can lead to generation of new theory (see grounded theory in Table 2 ).

Reliability and validity are two important ways of ascertaining rigor in quantitative research. Qualitative research seeks to understand individual construction and, by definition, is subjective. It is unlikely, therefore, that a study could ever be repeated with exactly the same circumstances. Instead, qualitative research is concerned with the question of whether the findings are trustworthy; that is, if the same circumstances were to prevail, would the same conclusions would be drawn?

There are a number of ways to maximise trustworthiness. One is triangulation, of which there are three subtypes. Data triangulation involves using data from several sources (eg, interviews, documentation, observation). A research team may include members from different backgrounds (eg, psychology, psychiatry, sociology), enabling a range of perspectives to be used within the discussion and interpretation of the data. This is termed researcher triangulation . The final subtype, theoretical triangulation, requires using more than one theory to examine the research question. Another technique to establish the trustworthiness of the findings is to use respondent validation. Here, the final or interim analysis is presented to members of the population of interest to ascertain whether interpretations made are valid.

An important aspect of all qualitative studies is researcher reflexivity. Here researchers consider their role and how their experience and knowledge might influence the generation, analysis and interpretation of the data. As with all well-conducted research, a clear record of progress should be kept – to enable scrutiny of recruitment, data generation and development of analysis. However, transparency is particularly important in qualitative research as the concepts and views evolve and are refined during the process.

Judging quality in qualitative research

Within all fields of research there are better and worse ways of conducting a study, and range of quality in mental health qualitative research is variable. Many of the principles for judging quality in qualitative research are the same for judging quality in any other type of research. However, several guidelines have been developed to help readers, reviewers and editors who lack methodological expertise to feel more confident in appraising qualitative studies. Guidelines are a prerequisite for the relatively recent advance of methodologies for systematic reviewing of qualitative literature (see meta-synthesis in Table 2 ). Box 1 provides some key questions that should be considered while studying a qualitative report.

Box 1 Guidelines for authors and reviewers of qualitative research (adapted from Malterud 35 )

▶ Is the research question relevant and clearly stated?

Reflexivity

▶ Are the researcher's motives and background presented?

Method, sampling and data collection

▶ Is a qualitative method appropriate and justified?

▶ Is the sampling strategy clearly described and justified?

▶ Is the method for data generation fully described

▶ Are the characteristics of the sample sufficiently described?

Theoretical framework

▶ Was a theoretical framework used and stated?

▶ Are the principles and procedures for data organisation and analysis described and justified?

▶ Are strategies used to test the trustworthiness of the findings?

▶ Are the findings relevant to the aim of the study?

▶ Are data (e.g. quotes) used to support and enrich the findings?

▶ Are the conclusions directly linked to the study? Are you convinced?

▶ Do the findings have clinical or theoretical value?

▶ Are findings compared to appropriate theoretical and empirical literature?

▶ Are questions about the internal and external validity and reflexivity discussed?

▶ Are shortcomings of the design, and the implications these have on findings, examined?

▶ Are clinical/theoretical implications of the findings made?

Presentation

▶ Is the report understandable and clearly contextualised?

▶ Is it possible to distinguish between the voices of informants and researchers?

▶ Are sources from the field used and appropriately referenced?

Conclusions and future directions

Qualitative research has enormous potential within the field of mental health research, yet researchers are only beginning to exploit the range of methods they use at each stage of enquiry. Strengths of qualitative research primarily lie in developing theory and increasing understanding about effective implementation of treatments and how best to support clinicians and service users in managing mental health problems. An important development in the field is how to integrate methodological approaches to address questions. This raises a number of challenges, such as how to integrate textual and numerical data and how to reconcile different epistemologies. A distinction can be made between mixed- method design (eg, quantitative and qualitative data are gathered and findings combined within a single or series of studies) and mixed- model study, a pragmatist approach, whereby aspects of qualitative and quantitative research are combined at different stages during a research process. 19 Qualitative research is still often viewed as only a support function or as secondary to quantitative research; however, this situation is likely to evolve as more researchers gain a broader skill set.

Though it is undeniable that there has been a marked increase in the volume and quality of qualitative research published within the past two decades, mental health research has been surprisingly slow to develop, compared to other disciplines e.g. general practice and nursing, with relatively fewer qualitative research findings reaching mainstream psychiatric journals. 2 This does not appear to reflect overall editorial policy; however, it may be partly due to the lack of confidence on the part of editors and reviewers while identifying rigorous qualitative research data for further publication. 20 However, the skilled researcher should no longer find him or herself forced into a position of defending a single-methodology camp (quantitative vs qualitative), but should be equipped with the necessary methodological and analytical skills to study and interpret data and to appraise and interpret others' findings from a full range of methodological techniques.

• Crawford MJ ,
• Cordingley L
• Dowrick C ,
• Edwards S ,
• ↵ MRC Developing and Evaluating Complex Interventions 2008
• Nelson ML ,
• Quintana SM
• Schulze B ,
• Angermeyer MC
• Mavaddat N ,
• Lester HE ,
• Donovan J ,
• Morriss R ,
• Barkham M ,
• Stiles WB ,
• Connell J ,
• Chew-Graham C ,
• England E ,
• Kinderman P ,
• Tashakkoria A ,
• Ritchie J ,
• Ssebunnya J ,
• Chilvers R ,
• Glasman D ,
• Finlay WM ,
• Strauss A ,
• Hodges BD ,
• Dobransky K
• Dixon-Woods M ,
• Fitzpatrick R ,
• Espíndola CR ,

Read the full text or download the PDF:

Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Data and Statistics

Below are resources that provide up-to-date statistics around mental health. However, the list of data tools is not extensive.

• Behavioral Risk Factor Surveillance System (BRFSS) BRFSS collects information on health risk behaviors, preventative practices, and healthcare access. Questions include recent mentally unhealthy days, anxiety and depressive disorders, mental illness and stigma, and psychological distress.
• Household Pulse Survey CDC has partnered with the U.S. Census Bureau on an experimental data system called the Household Pulse Survey to monitor trends in mental health, health insurance coverage, and problems accessing care.
• National Health and Nutrition Examination Survey (NHANES) NHANES assesses health and nutritional status through interviews and physical examinations. Collected data cover a number of conditions, including depression and anxiety, symptoms of conditions, concerns associated with mental health and substance abuse, and mental health service use and need.
• National Health Interview Survey (NHIS) NHIS collects data on both adult and children’s mental health and mental disorders. For adults, this includes serious psychological distress and feelings of depression and anxiety.  For children, this includes the presence of  attention deficit/hyperactivity disorder and autism spectrum disorder.  The NHIS also examines mental health service use and whether individuals have unmet mental health needs. Questions about recent anxiety or frequent stress have been included in previous years.
• National Ambulatory Medical Care Survey (NAMCS) NAMCS collects data on visits to non-federally employed office-based physicians who are primarily engaged in direct patient care and, starting in 2006, a separate sample of visits to community health centers.  Data are collected on type of provider, medications, primary diagnoses and presence of long-lasting conditions.
• National Hospital Care Survey (NHCS) NHCS allows examination of care provided across treatment settings. Data cover physicians' diagnoses, services and procedures, types of healthcare professionals seen, hospital characteristics, discharge diagnoses, surgical and diagnostic procedures, and prescriptions for ambulatory visits.
• National Post-acute and Long-term Care Study (NPALS) NPALS monitors trends in the supply, provision, and use of the major sectors of paid, regulated long-term care services. Data cover mental illness, depression, and service use.
• National Survey of Children’s Health (NSCH) NSCH examines the health of children including those with special needs with an emphasis on well-being, such as medical homes, family interactions, parental health, school and after-school experiences, and safe neighborhoods. The survey also collects information on the presence of a mental or behavioral problem.
• National Survey of the Diagnosis and Treatment of ADHD and Tourette Syndrome (NS-DATA) NS-DATA collects information about children 2 to 15 years old in 2011-2012 who had ever been diagnosed with attention-deficit/hyperactivity disorder (ADHD) and/or Tourette syndrome (TS) with the goal of better understanding diagnostic practices, level of impairment, and treatments for this group of children.
• National Violent Death Reporting System (NVDRS) NVDRS collects data from medical examiners, coroners, police, crime labs, and death certificates to understand the circumstances surrounding violent deaths, including suicide. NVDRS can also provide details on the circumstances that may have led to violent deaths, including mental illness and mental disorders.
• Pregnancy Risk Assessment Monitoring System (PRAMS) PRAMS collects data on maternal attitudes and experiences before, during, and after pregnancy. Surveillance research includes the prevalence of self-reported postpartum depression and anxiety symptoms.
• School Health Policies and Programs Study (SHPPS) SHPPS is a national survey assessing school health policies and practices at the state, district, school, and classroom levels. Collected data includes mental health and social service policies.
• WISQARS™ CDC’s WISQARS (Web-based Injury Statistics Query and Reporting System) is an interactive database system that provides customized reports of injury-related data, such as intentional self-harm including suicide.
• Youth Risk Behavior Surveillance System (YRBSS) YRBSS monitors health-risk behaviors including tobacco use, substance abuse, unintentional injuries and violence, sexual behaviors that contribute to unintended pregnancy and STDs.

Exit Notification / Disclaimer Policy

• The Centers for Disease Control and Prevention (CDC) cannot attest to the accuracy of a non-federal website.
• Linking to a non-federal website does not constitute an endorsement by CDC or any of its employees of the sponsors or the information and products presented on the website.
• You will be subject to the destination website's privacy policy when you follow the link.
• CDC is not responsible for Section 508 compliance (accessibility) on other federal or private website.

• Systematic review
• Open access
• Published: 10 October 2019

An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance

• Eric Badu   ORCID: orcid.org/0000-0002-0593-3550 1 ,
• Anthony Paul O’Brien 2 &
• Rebecca Mitchell 3  

Archives of Public Health volume  77 , Article number:  37 ( 2019 ) Cite this article

23k Accesses

19 Citations

1 Altmetric

Metrics details

Several typologies and guidelines are available to address the methodological and practical considerations required in mental health research. However, few studies have actually attempted to systematically identify and synthesise these considerations. This paper provides an integrative review that identifies and synthesises the available research evidence on mental health research methodological considerations.

A search of the published literature was conducted using EMBASE, Medline, PsycINFO, CINAHL, Web of Science, and Scopus. The search was limited to papers published in English for the timeframe 2000–2018. Using pre-defined inclusion and exclusion criteria, three reviewers independently screened the retrieved papers. A data extraction form was used to extract data from the included papers.

Of 27 papers meeting the inclusion criteria, 13 focused on qualitative research, 8 mixed methods and 6 papers focused on quantitative methodology. A total of 14 papers targeted global mental health research, with 2 papers each describing studies in Germany, Sweden and China. The review identified several methodological considerations relating to study design, methods, data collection, and quality assurance. Methodological issues regarding the study design included assembling team members, familiarisation and sharing information on the topic, and seeking the contribution of team members. Methodological considerations to facilitate data collection involved adequate preparation prior to fieldwork, appropriateness and adequacy of the sampling and data collection approach, selection of consumers, the social or cultural context, practical and organisational skills; and ethical and sensitivity issues.

The evidence confirms that studies on methodological considerations in conducting mental health research largely focus on qualitative studies in a transcultural setting, as well as recommendations derived from multi-site surveys. Mental health research should adequately consider the methodological issues around study design, sampling, data collection procedures and quality assurance in order to maintain the quality of data collection.

Peer Review reports

In the past decades there has been considerable attention on research methods to facilitate studies in various academic fields, such as public health, education, humanities, behavioural and social sciences [ 1 , 2 , 3 , 4 ]. These research methodologies have generally focused on the two major research pillars known as quantitative or qualitative research. In recent years, researchers conducting mental health research appear to be either employing both qualitative and quantitative research methods separately, or mixed methods approaches to triangulate and validate findings [ 5 , 6 ].

A combination of study designs has been utilised to answer research questions associated with mental health services and consumer outcomes [ 7 , 8 ]. Study designs in the public health and clinical domains, for example, have largely focused on observational studies (non-interventional) and experimental research (interventional) [ 1 , 3 , 9 ]. Observational design in non-interventional research requires the investigator to simply observe, record, classify, count and analyse the data [ 1 , 2 , 10 ]. This design is different from the observational approaches used in social science research, which may involve observing (participant and non- participant) phenomena in the fieldwork [ 1 ]. Furthermore, the observational study has been categorized into five types, namely cross-sectional design, case-control studies, cohort studies, case report and case series studies [ 1 , 2 , 3 , 9 , 10 , 11 ]. The cross-sectional design is used to measure the occurrence of a condition at a one-time point, sometimes referred to as a prevalence study. This approach of conducting research is relatively quick and easy but does not permit a distinction between cause and effect [ 1 ]. Conversely, the case-control is a design that examines the relationship between an attribute and a disease by comparing those with and without the disease [ 1 , 2 , 12 ]. In addition, the case-control design is usually retrospective and aims to identify predictors of a particular outcome. This type of design is relevant when investigating rare or chronic diseases which may result from long-term exposure to particular risk factors [ 10 ]. Cohort studies measure the relationship between exposure to a factor and the probability of the occurrence of a disease [ 1 , 10 ]. In a case series design, medical records are reviewed for exposure to determinants of disease and outcomes. More importantly, case series and case reports are often used as preliminary research to provide information on key clinical issues [ 12 ].

The interventional study design describes a research approach that applies clinical care to evaluate treatment effects on outcomes [ 13 ]. Several previous studies have explained the various forms of experimental study design used in public health and clinical research [ 14 , 15 ]. In particular, experimental studies have been categorized into randomized controlled trials (RCTs), non-randomized controlled trials, and quasi-experimental designs [ 14 ]. The randomized trial is a comparative study where participants are randomly assigned to one of two groups. This research examines a comparison between a group receiving treatment and a control group receiving treatment as usual or receiving a placebo. Herein, the exposure to the intervention is determined by random allocation [ 16 , 17 ].

Recently, research methodologists have given considerable attention to the development of methodologies to conduct research in vulnerable populations. Vulnerable population research, such as with mental health consumers often involves considering the challenges associated with sampling (selecting marginalized participants), collecting data and analysing it, as well as research engagement. Consequently, several empirical studies have been undertaken to document the methodological issues and challenges in research involving marginalized populations. In particular, these studies largely addresses the typologies and practical guidelines for conducting empirical studies in mental health. Despite the increasing evidence, however, only a few studies have yet attempted to systematically identify and synthesise the methodological considerations in conducting mental health research from the perspective of consumers.

A preliminary search using the search engines Medline, Web of Science, Google Scholar, and Scopus Index and EMBASE identified only two reviews of mental health based research. Among these two papers, one focused on the various types of mixed methods used in mental health research [ 18 ], whilst the other paper, focused on the role of qualitative studies in mental health research involving mixed methods [ 19 ]. Even though the latter two studies attempted to systematically review mixed methods mental health research, this integrative review is unique, as it collectively synthesises the design, data collection, sampling, and quality assurance issues together, which has not been previously attempted.

This paper provides an integrative review addressing the available evidence on mental health research methodological considerations. The paper also synthesises evidence on the methods, study designs, data collection procedures, analyses and quality assurance measures. Identifying and synthesising evidence on the conduct of mental health research has relevance to clinicians and academic researchers where the evidence provides a guide regarding the methodological issues involved when conducting research in the mental health domain. Additionally, the synthesis can inform clinicians and academia about the gaps in the literature related to methodological considerations.

Methodology

An integrative review was conducted to synthesise the available evidence on mental health research methodological considerations. To guide the review, the World Health Organization (WHO) definition of mental health has been utilised. The WHO defines mental health as: “a state of well-being, in which the individual realises his or her own potentials, ability to cope with the normal stresses of life, functionality and work productivity, as well as the ability to contribute effectively in community life” [ 20 ]. The integrative review enabled the simultaneous inclusion of diverse methodologies (i.e., experimental and non-experimental research) and varied perspectives to fully understand a phenomenon of concern [ 21 , 22 ]. The review also uses diverse data sources to develop a holistic understanding of methodological considerations in mental health research. The methodology employed involves five stages: 1) problem identification (ensuring that the research question and purpose are clearly defined); 2) literature search (incorporating a comprehensive search strategy); 3) data evaluation; 4) data analysis (data reduction, display, comparison and conclusions) and; 5) presentation (synthesising findings in a model or theory and describing the implications for practice, policy and further research) [ 21 ].

Inclusion criteria

The integrative review focused on methodological issues in mental health research. This included core areas such as study design and methods, particularly qualitative, quantitative or both. The review targeted papers that addressed study design, sampling, data collection procedures, quality assurance and the data analysis process. More specifically, the included papers addressed methodological issues on empirical studies in mental health research. The methodological issues in this context are not limited to a particular mental illness. Studies that met the inclusion criteria were peer-reviewed articles published in the English Language, from January 2000 to July 2018.

Exclusion criteria

Articles that were excluded were based purely on general health services or clinical effectiveness of a particular intervention with no connection to mental health research. Articles were also excluded when it addresses non-methodological issues. Other general exclusion criteria were book chapters, conference abstracts, papers that present opinion, editorials, commentaries and clinical case reviews.

Search strategy and selection procedure

The search of published articles was conducted from six electronic databases, namely EMBASE, CINAHL (EBSCO), Web of Science, Scopus, PsycINFO and Medline. We developed a search strategy based on the recommended guidelines by the Joanna Briggs Institute (JBI) [ 23 ]. Specifically, a three-step search strategy was utilised to conduct the search for information (see Table  1 ). An initial limited search was conducted in Medline and Embase (see Table 1 ). We analysed the text words contained in the title and abstract and of the index terms from the initial search results [ 23 ]. A second search using all identified keywords and index terms was then repeated across all remaining five databases (see Table 1 ). Finally, the reference lists of all eligible studies were manually hand searched [ 23 ].

The selection of eligible articles adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [ 24 ] (see Fig.  1 ). Firstly, three authors independently screened the titles of articles that were retrieved and then approved those meeting the selection criteria. The authors reviewed all the titles and abstracts and agreed on those needing full-text screening. E.B (Eric Badu) conducted the initial screening of titles and abstracts. A.P.O’B (Anthony Paul O’Brien) and R.M (Rebecca Mitchell) conducted the second screening of titles and abstracts of all the identified papers. The authors (E.B, A.P.O’B and R.M) conducted full-text screening according to the inclusion and exclusion criteria.

Flow Chart of studies included in the review

Data management and extraction

The integrative review used Endnote ×8 to screen and handle duplicate references. A predefined data extraction form was developed to extract data from all included articles (see Additional file 1 ). The data extraction form was developed according to Joanna Briggs Institute (JBI) [ 23 ] and Cochrane [ 24 ] manuals, as well as the literature associated with concepts and methods in mental health research. The data extraction form was categorised into sub-sections, such as study details (citation, year of publication, author, contact details of lead author, and funder/sponsoring organisation, publication type), objective of the paper, primary subject area of the paper (study design, methods, sampling, data collection, data analysis, quality assurance). The data extraction form also had a section on additional information on methodological consideration, recommendations and other potential references. The authors extracted results of the included papers in numerical and textual format [ 23 ]. EB (Eric Badu) conducted the data extraction, A.P.O’B (Anthony Paul O’Brien) and R.M (Rebecca Mitchell), conducted the second review of the extracted data.

Data synthesis

Content analysis was used to synthesise the extracted data. The content analysis process involved several stages which involved noting patterns and themes, seeing plausibility, clustering, counting, making contrasts and comparisons, discerning common and unusual patterns, subsuming particulars into general, noting relations between variability, finding intervening factors and building a logical chain of evidence [ 21 ] (see Table  2 ).

Study characteristics

The integrative review identified a total of 491 records from all databases, after which 19 duplicates were removed. Out of this, 472 titles and abstracts were assessed for eligibility, after which 439 articles were excluded. Articles not meeting the inclusion criteria were excluded. Specifically, papers excluded were those that did not address methodological issues as well as papers addressing methodological consideration in other disciplines. A total of 33 full-text articles were assessed – 9 articles were further excluded, whilst an additional 3 articles were identified from reference lists. Overall, 27 articles were included in the final synthesis (see Fig. 1 ). Of the total included papers, 12 contained qualitative research, 9 were mixed methods (both qualitative and quantitative) and 6 papers focused on quantitative data. Conversely, a total of 14 papers targeted global mental health research and 2 papers each describing studies in Germany, Sweden and China. The papers addressed different methodological issues, such as study design, methods, data collection, and analysis as well as quality assurance (see Table  3 ).

Mixed methods design in mental health research

Mixed methods research is defined as a research process where the elements of qualitative and quantitative research are combined in the design, data collection, and its triangulation and validation [ 48 ]. The integrative review identified four sub-themes that describe mixed methods design in the context of mental health research. The sub-themes include the categories of mixed methods, their function, structure, process and further methodological considerations for mixed methods design. These sub-themes are explained as follows:

Categorizing mixed methods in mental health research

Four studies highlighted the categories of mixed methods design applicable to mental health research [ 18 , 19 , 43 , 48 ]. Generally, there are differences in the categories of mixed methods design, however, three distinct categories predominantly appear to cross cut in all studies. These categories are function, structure and process. Some studies further categorised mixed method design to include rationale, objectives, or purpose. For instance, Schoonenboom and Johnson [ 48 ] categorised mixed methods design into primary and secondary dimensions.

The function of mixed methods in mental health research

Six studies explain the function of conducting mixed methods design in mental health research. Two studies specifically recommended that mixed methods have the ability to provide a more robust understanding of services by expanding and strengthening the conclusions from the study [ 42 , 45 ]. More importantly, the use of both qualitative and quantitative methods have the ability to provide innovative solutions to important and complex problems, especially by addressing diversity and divergence [ 48 ]. The review identified five underlying functions of a mixed method design in mental health research which include achieving convergence, complementarity, expansion, development and sampling [ 18 , 19 , 43 ].

The use of mixed methods to achieve convergence aims to employ both qualitative and quantitative data to answer the same question, either through triangulation (to confirm the conclusions from each of the methods) or transformation (using qualitative techniques to transform quantitative data). Similarly, complementarity in mixed methods integrates both qualitative and quantitative methods to answer questions for the purpose of evaluation or elaboration [ 18 , 19 , 43 ]. Two papers recommend that qualitative methods are used to provide the depth of understanding, whilst the quantitative methods provide a breadth of understanding [ 18 , 43 ]. In mental health research, the qualitative data is often used to examine treatment processes, whilst the quantitative methods are used to examine treatment outcomes against quality care key performance targets.

Additionally, three papers indicated that expansion as a function of mixed methods uses one type of method to answer questions raised by the other type of method [ 18 , 19 , 43 ]. For instance, qualitative data is used to explain findings from quantitative analysis. Also, some studies highlight that development as a function of mixed methods aims to use one method to answer research questions, and use the findings to inform other methods to answer different research questions. A qualitative method, for example, is used to identify the content of items to be used in a quantitative study. This approach aims to use qualitative methods to create a conceptual framework for generating hypotheses to be tested by using a quantitative method [ 18 , 19 , 43 ]. Three papers suggested that using mixed methods for the purpose of sampling utilize one method (eg. quantitative) to identify a sample of participants to conduct research using other methods (eg. qualitative) [ 18 , 19 , 43 ]. For instance, quantitative data is sequentially utilized to identify potential participants to participate in a qualitative study and the vice versa.

Structure of mixed methods in mental health research

Five studies categorised the structure of conducting mixed methods in mental health research, into two broader concepts including simultaneous (concurrent) and sequential (see Table 3 ). In both categories, one method is regarded as primary and the other as secondary, although equal weight can be given to both methods [ 18 , 19 , 42 , 43 , 48 ]. Two studies suggested that the sequential design is a process where the data collection and analysis of one component (eg. quantitative) takes place after the data collection and analysis of the other component (eg qualitative). Herein, the data collection and analysis of one component (e.g. qualitative) may depend on the outcomes of the other component (e.g. quantitative) [ 43 , 48 ]. An earlier review suggested that the majority of contemporary studies in mental health research use a sequential design, with qualitative methods, more often preceding quantitative methods [ 18 ].

Alternatively, the concurrent design collects and analyses data of both components (e.g. quantitative and qualitative) simultaneously and independently. Palinkas, Horwitz [ 42 ] recommend that one component is used as secondary to the other component, or that both components are assigned equal priority. Such a mixed methods approach aims to provide a depth of understanding afforded by qualitative methods, with the breadth of understanding offered by the quantitative data to elaborate on the findings of one component or seek convergence through triangulation of the results. Schoonenboom and Johnson [ 48 ] recommended the use of capital letters for one component and lower case letters for another component in the same design to indicate that one component is primary and the other is secondary or supplemental.

Process of mixed methods in mental health research

Five papers highlighted the process for the use of mixed methods in mental health research [ 18 , 19 , 42 , 43 , 48 ]. The papers suggested three distinct processes or strategies for combining qualitative and quantitative data. These include merging or converging the two data sets, connecting the two datasets by having one build upon the other; and embedding one data set within the other [ 19 , 43 ]. The process of connecting occurs when the analysis of one dataset leads to the need for the other data set. For instance, in the situation where quantitative results lead to the subsequent collection and analysis of qualitative data [ 18 , 43 ]. A previous study suggested that most studies in mental health sought to connect the data sets. Similarly, the process of merging the datasets brings together two sets of data during the interpretation, or transforms one type of data into the other type, by combining the data into new variables [ 18 ]. The process of embedding data into mixed method designs in mental health uses one dataset to provide a supportive role to the other dataset [ 43 ].

Consideration for using mixed methods in mental health research

Three studies highlighted several factors that need to be considered when conducting mixed methods design in mental health research [ 18 , 19 , 45 ]. Accordingly, these factors include developing familiarity with the topic under investigation based on experience, willingness to share information on the topic [ 19 ], establishing early collaboration, willingness to negotiate emerging problems, seeking the contribution of team members, and soliciting third-party assistance to resolve any emerging problems [ 45 ]. Additionally, Palinkas, Horwitz [ 18 ] recommended that mixed methods in the context of mental health research are mostly applied in studies that assess needs of services, examine existing services, developing new or adapting existing services, evaluating services in randomised control trials, and examining service implementation.

Qualitative study in mental health research

This theme describes the various qualitative methods used in mental health research. The theme also addresses methodological considerations for using qualitative methods in mental health research. The key emerging issues are discussed below:

Considering qualitative components in conducting mental health research

Six studies recommended the use of qualitative methods in mental health research [ 19 , 26 , 28 , 32 , 36 , 44 ]. Two qualitative research paradigms were identified, including the interpretive and critical approach [ 32 ]. The interpretive methodologies predominantly explore the meaning of human experiences and actions, whilst the critical approach emphasises the social and historical origins and contexts of meaning [ 32 ]. Two studies suggested that the interpretive qualitative methods used in mental health research are ethnography, phenomenology and narrative approaches [ 32 , 36 ].

The ethnographic approach describes the everyday meaning of the phenomena within a societal and cultural context, for instance, the way phenomena or experience is contrasted within a community, or by collective members over time [ 32 ]. Alternatively, the phenomenological approach explores the claims and concerns of a subject with a speculative development of an interpretative account within their cultural and physical environments focusing on the lived experience [ 32 , 36 ].

Moreover, the critical qualitative approaches used in mental health research are predominantly emancipatory (for instance, socio-political traditions) and participatory action-based research. The emancipatory traditions recognise that knowledge is acquired through critical discourse and debate but are not seen as discovered by objective inquiry [ 32 ]. Alternatively, the participatory action based approach uses critical perspectives to engage key stakeholders as participants in the design and conduct of the research [ 32 ].

Some studies highlighted several reasons why qualitative methods are relevant to mental health research. In particular, qualitative methods are significant as they emphasise naturalistic inquiry and have a discovery-oriented approach [ 19 , 26 ]. Two studies suggested that qualitative methods are often relevant in the initial stages of research studies to understand specific issues such as behaviour, or symptoms of consumers of mental services [ 19 ]. Specifically, Palinkas [ 19 ] suggests that qualitative methods help to obtain initial pilot data, or when there is too little previous research or in the absence of a theory, such as provided in exploratory studies, or previously under-researched phenomena.

Three studies stressed that qualitative methods can help to better understand socially sensitive issues, such as exploring the solutions to overcome challenges in mental health clinical policies [ 19 , 28 , 44 ]. Consequently, Razafsha, Behforuzi [ 44 ] recommended that the natural holistic view of qualitative methods can help to understand the more recovery-oriented policy of mental health, rather than simply the treatment of symptoms. Similarly, the subjective experiences of consumers using qualitative approaches have been found useful to inform clinical policy development [ 28 ].

Sampling in mental health research

The theme explains the sampling approaches used in mental health research. The section also describes the methodological considerations when sampling participants for mental health research. The sub-themes emerging are explained in the following sections:

Sampling approaches (quantitative)

Some studies reviewed highlighted the sampling approaches previously used in mental health research [ 25 , 34 , 35 ]. Generally, all quantitative studies tend to use several probability sampling approaches, whilst qualitative studies used non-probability techniques. The quantitative mental health studies conducted at community and population level employ multi-stage sampling techniques usually involving systematic sampling, stratified and random sampling [ 25 , 34 ]. Similarly, quantitative studies that recruit consumers in the hospital setting employ consecutive sampling [ 35 ]. Two studies reviewed highlighted that the identification of consumers of mental health services for research is usually conducted by service providers. For instance, Korver, Quee [ 35 ] research used a consecutive sampling approach by identifying consumers through clinicians working in regional psychosis departments, or academic centres.

Sampling approaches (qualitative)

Seven studies suggested that the sampling procedures widely used in mental health research involving qualitative methods are non-probability techniques, which include purposive [ 19 , 28 , 32 , 42 , 46 ], snowballing [ 30 , 32 , 46 ] and theoretical sampling [ 31 , 32 ]. The purposive sampling identifies participants that possess relevant characteristics to answer a research question [ 28 ]. Purposive sampling can be used in a single case study, or for multiple cases. The purposive sampling used in mental health research is usually extreme, or deviant case sampling, criterion sampling, and maximum variation sampling [ 19 ]. Furthermore, it is advised when using purposive sampling in a multistage level study, that it should aim to begin with the broader picture to achieve variation, or dispersion, before moving to the more focused view that considers similarity, or central tendencies [ 42 ].

Two studies added that theoretical sampling involved sampling participants, situations and processes based on concepts on theoretical grounds and then using the findings to build theory, such as in a Grounded Theory study [ 31 , 32 ]. Some studies highlighted that snowball sampling is another strategy widely used in mental health research [ 30 , 32 , 46 ]. This is ascribed to the fact that people with mental illness are perceived as marginalised in research and practically hard-to-reach using conventional sampling [ 30 , 32 ]. Snowballing sampling involves asking the marginalised participants to recommend individuals who might have direct knowledge relevant to the study [ 30 , 32 , 46 ]. Although this approach is relevant, some studies advise the limited possibility of generalising the sample, because of the likelihood of selection bias [ 30 ].

Sampling consideration

Four studies in this section highlighted some of the sampling considerations in mental health research [ 30 , 31 , 32 , 46 ]. Generally, mental health research should consider the appropriateness and adequacy of sampling approach by applying attributes such as shared social, or cultural experiences, or shared concern related to the study [ 32 ], diversity and variety of participants [ 31 ], practical and organisational skills, as well as ethical and sensitivity issues [ 46 ]. Robinson [ 46 ] further suggested that sampling can be homogenous or heterogeneous depending on the research questions for the study. Achieving homogeneity in sampling should employ a variety of parameters, which include demographic, graphical, physical, psychological, or life history homogeneity [ 46 ]. Additionally, applying homogeneity in sampling can be influenced by theoretical and practical factors. Alternatively, some samples are intentionally selected based on heterogeneous factors [ 46 ].

Data collection in mental health research

This theme highlights the data collection methods used in mental health research. The theme is explained according to three sub-themes, which include approaches for collecting qualitative data, methodological considerations, as well as preparations for data collection. The sub-themes are as follows:

Approaches for collecting qualitative data

The studies reviewed recommended the approaches that are widely applied in collecting data in mental health research. The widely used qualitative data collection approaches in mental health research are focus group discussions (FGDs) [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ], extended in-depth interviews [ 19 , 30 , 34 ], participant and non-participant observation [ 19 ], Delphi data collection, quasi-statistical techniques [ 19 ] and field notes [ 31 , 40 ]. Seven studies suggest that FGDs are widely used data collection approaches [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ] because they are valuable in gathering information on consumers’ perspectives of services, especially regarding satisfaction, unmet/met service needs and the perceived impact of services [ 47 ]. Conversely, Ekblad and Baarnhielm [ 31 ] recommended that this approach is relevant to improve clinical understanding of the thoughts, emotions, meanings and attitudes towards mental health services.

Such data collection approaches are particularly relevant to consumers of mental health services, due to their low self-confidence and self-esteem [ 41 ]. The approach can help to understand specific terms, vocabulary, opinions and attitudes of consumers of mental health services, as well as their reasoning about personal distress and healing [ 31 ]. Similarly, the reliance on verbal rather than written communication helps to promote the participation of participants with serious and enduring mental health problems [ 31 , 41 ]. Although FGD has several important outcomes, there are some limitations that need critical consideration. Ekblad and Baarnhielm [ 31 ] for example suggest, that marginalised participants may not always feel free to talk about private issues regarding their condition at the group level mostly due to perceived stigma and group confidentiality.

Some studies reviewed recommended that attempting to capture comprehensive information and analysing group interactions in mental health research requires the research method to use field notes as a supplementary data source to help validate the FGDs [ 31 , 40 , 41 ]. The use of field notes in addition to FGDs essentially provides greater detail in the accounts of consumers’ subjective experiences. Furthermore, Montgomery and Bailey [ 40 ] suggest that field notes require observational sensitivity, and also require having specific content such as descriptive and interpretive data.

Three studies in this section suggested that in-depth interviews are used to collect data from consumers of mental health services [ 19 , 30 , 34 ]. This approach is particularly important to explore the behaviour, subjective experiences and psychological processes; opinions, and perceptions of mental health services. de Jong and Van Ommeren [ 30 ] recommend that in-depth interviews help to collect data on culturally marked disorders, their personal and interpersonal significance, patient and family explanatory models, individual and family coping styles, symptom symbols and protective mediators. Palinkas [ 19 ] also highlights that the structured narrative form of extended interviewing is the type of in-depth interview used in mental health research. This approach provides participants with the opportunity to describe the experience of living with an illness and seeking services that assist them.

Consideration for data collection

Six studies recommended consideration required in the data collection process [ 31 , 32 , 37 , 41 , 47 , 49 ]. Some studies highlighted that consumers of mental health services might refuse to participate in research due to several factors [ 37 ] like the severity of their illness, stigma and discrimination [ 41 ]. Subsequently, such issues are recommended to be addressed by building confidence and trust between the researcher and consumers [ 31 , 37 ]. This is a significant prerequisite, as it can sensitise and normalise the research process and aims with the participants prior to discussing their personal mental health issues. Similarly, some studies added that the researcher can gain the confidence of service providers who manage consumers of mental health services [ 41 , 47 ], seek ethical approval from the relevant committee(s) [ 41 , 47 ], meet and greet the consumers of mental health services before data collection, and arrange a mutually acceptable venue for the groups and possibly supply transport [ 41 ].

Two studies further suggested that the cultural and social differences of the participants need consideration [ 26 , 31 ]. These factors could influence the perception and interpretation of ethical issues in the research situation.

Additionally, two studies recommended the use of standardised assessment instruments for mental health research that involve quantitative data collection [ 33 , 49 ]. A recent survey suggested that measures to standardise the data collection approach can convert self-completion instruments to interviewer-completion instruments [ 49 ]. The interviewer can then read the items of the instruments to respondents and record their responses. The study further suggested the need to collect demographic and behavioural information about the participant(s).

Preparing for data collection

Eight studies highlighted the procedures involved in preparing for data collection in mental health research [ 25 , 30 , 33 , 34 , 35 , 39 , 41 , 49 ]. These studies suggest that the preparation process involve organising meetings of researchers, colleagues and representatives of the research population. The meeting of researchers generally involves training of interviewers about the overall design, objectives and research questions associated with the study. de Jong and Van Ommeren [ 30 ] recommended that preparation for the use of quantitative data encompasses translating and adapting instruments with the aim of achieving content, semantic, concept, criterion and technical equivalence.

Quality assurance procedures in mental health research

This section describes the quality assurance procedures used in mental health research. Quality assurance is explained according to three sub-themes: 1) seeking informed consent, 2) the procedure for ensuring quality assurance in a quantitative study and 3) the procedure for ensuring quality control in a qualitative study. The sub-themes are explained in the following content.

Seeking informed consent

The papers analysed for the integrative review suggested that the rights of participants to safeguard their integrity must always be respected, and so each potential subject must be adequately informed of the aims, methods, anticipated benefits and potential hazards of the study and any potential discomforts (see Table 3 ). Seven studies highlight that potential participants of mental health research must be consented to the study prior to data collection [ 25 , 26 , 33 , 35 , 37 , 39 , 47 ]. The consent process helps to assure participants of anonymity and confidentiality and further explain the research procedure to them. Baarnhielm and Ekblad [ 26 ] argue that the research should be guided by four basic moral values for medical ethics, autonomy, non-maleficence, beneficence, and justice. In particular, potential consumers of mental health services who may have severe conditions and unable to consent themselves are expected to have their consent signed by a respective family caregiver [ 37 ]. Latvala, Vuokila-Oikkonen [ 37 ] further suggested that researchers are responsible to agree on the criteria to determine the competency of potential participants in mental health research. The criteria are particularly relevant when potential participants have difficulties in understanding information due to their mental illness.

Procedure for ensuring quality control (quantitative)

Several studies highlighted procedures for ensuring quality control in mental health research (see Table 3 ). The quality control measures are used to achieve the highest reliability, validity and timeliness. Some studies demonstrate that ensuring quality control should consider factors such as pre-testing tools [ 25 , 49 ], minimising non-response rates [ 25 , 39 ] and monitoring of data collection processes [ 25 , 33 , 49 ].

Accordingly, two studies suggested that efforts should be made to re-approach participants who initially refuse to participate in the study. For instance, Liu, Huang [ 39 ] recommended that when a consumer of mental health services refuse to participate in a study (due to low self-esteem) when approached for the first time, a different interviewer can re-approach the same participant to see if they are more comfortable to participate after the first invitation. Three studies further recommend that monitoring data quality can be accomplished through “checks across individuals, completion status and checks across variables” [ 25 , 33 , 49 ]. For example, Alonso, Angermeyer [ 25 ] advocate that various checks are used to verify completion of the interview, and consistency across instruments against the standard procedure.

Procedure for ensuring quality control (qualitative)

Four studies highlighted the procedures for ensuring quality control of qualitative data in mental health research [ 19 , 32 , 37 , 46 ]. A further two studies suggested that the quality of qualitative research is governed by the principles of credibility, dependability, transferability, reflexivity, confirmability [ 19 , 32 ]. Some studies explain that the credibility or trustworthiness of qualitative research in mental health is determined by methodological and interpretive rigour of the phenomenon being investigated [ 32 , 37 ]. Consequently, Fossey, Harvey [ 32 ] propose that the methodological rigour for assessing the credibility of qualitative research are congruence, responsiveness or sensitivity to social context, appropriateness (importance and impact), adequacy and transparency. Similarly, interpretive rigour is classified as authenticity, coherence, reciprocity, typicality and permeability of the researcher’s intentions; including engagement and interpretation [ 32 ].

Robinson [ 46 ] explained that transparency (openness and honesty) is achieved if the research report explicitly addresses how the sampling, data collection, analysis, and presentation are met. In particular, efforts to address these methodological issues highlight the extent to which the criteria for quality profoundly interacts with standards for ethics. Similarly, responsiveness, or sensitivity, helps to situate or locate the study within a place, a time and a meaningful group [ 46 ]. The study should also consider the researcher’s background, location and connection to the study setting, particularly in the recruitment process. This is often described as role conflict or research bias.

In the interpretive phenomenon, coherence highlights the ability to select an appropriate sampling procedure that mutually matches the research aims, questions, data collection, analysis, as well as any theoretical concepts or frameworks [ 32 , 46 ]. Similarly, authenticity explains the appropriate representation of participants’ perspectives in the research process and the interpretation of results. Authenticity is maximised by providing evidence that participants are adequately represented in the interpretive process, or provided an opportunity to give feedback on the researcher’s interpretation [ 32 ]. Again, the contribution of the researcher’s perspective to the interpretation enhances permeability. Fossey, Harvey [ 32 ] further suggest that reflexive reporting, which distinguishes the participants’ voices from that of the researcher in the report, enhances the permeability of the researcher’s role and perspective.

One study highlighted the approaches used to ensure validity in qualitative research, which includes saturation, identification of deviant or non-confirmatory cases, member checking and coding by consensus. Saturation involves completeness in the research process, where all relevant data collection, codes and themes required to answer the phenomenon of inquiry are achieved; and no new data emerges [ 19 ]. Similarly, member checking is the process whereby participants or others who share similar characteristics review study findings to elaborate on confirming them [ 19 ]. The coding by consensus involves a collaborative approach to analysing the data. Ensuring regular meetings among coders to discuss procedures for assigning codes to segments of data and resolve differences in coding procedures, and by comparison of codes assigned on selected transcripts to calculate a percentage agreement or kappa measure of interrater reliability, are commonly applied [ 19 ].

Two studies recommend the need to acknowledge the importance of generalisability (transferability). This concept aims to provide sufficient information about the research setting, findings and interpretations for readers to appropriately determine the replicability of the findings from one context, or population to another, otherwise known as reliability in quantitative research [ 19 , 32 ]. Similarly, the researchers should employ reflexivity as a means of identifying and addressing potential biases in data collection and interpretation. Palinkas [ 19 ] suggests that such bias is associated with theoretical orientations; pre-conceived beliefs, assumptions, and demographic characteristics; and familiarity and experience with the methods and phenomenon. Another approach to enhance the rigour of analysis involves peer debriefing and support meetings held among team members which facilitate detailed auditing during data analysis [ 19 ].

The integrative review was conducted to synthesise evidence into recommended methodological considerations when conducting mental health research. The evidence from the review has been discussed according to five major themes: 1) mixed methods study in mental health research; 2) qualitative study in mental health research; 3) sampling in mental health research; 4) data collection in mental health research; and 5) quality assurance procedures in mental health research.

Mixed methods study in mental health research

The evidence suggests that mixed methods approach in mental health are generally categorised according to their function (rationale, objectives or purpose), structure and process [ 18 , 19 , 43 , 48 ]. The mixed methods study can be conducted for the purpose of achieving convergence, complementarity, expansion, development and sampling [ 18 , 19 , 43 ]. Researchers conducting mental health studies should understand the underlying functions or purpose of mixed methods. Similarly, mixed methods in mental health studies can be structured simultaneously (concurrent) and sequential [ 18 , 19 , 42 , 43 , 48 ]. More importantly, the process of combining qualitative and quantitative data can be achieved through merging or converging, connecting and embedding one data set within the other [ 18 , 19 , 42 , 43 , 48 ]. The evidence further recommends that researchers need to understand the stage of integrating the two sets of data and the rationale for doing so. This can inform researchers regarding the best stage and appropriate ways of combining the two components of data to adequately address the research question(s).

The evidence recommended some methodological consideration in the design of mixed methods projects in mental health [ 18 , 19 , 45 ]. These issues include establishing early collaboration, becoming familiar with the topic, sharing information on the topic, negotiating any emerging problems and seeking contributions from team members. The involvement of various expertise could ensure that methodological issues are clearly identified. However, addressing such issues midway, or late through the design can negatively affect the implementation [ 45 ]. Any robust discoveries can rarely be accommodated under the existing design. Therefore, the inclusion of various methodological expertise during inception can lead to a more robust mixed-methods design which maximises the contributions of team members. Whilst fundamental and philosophical differences in qualitative and quantitative methods may not be resolved, some workable solutions can be employed, particularly if challenges are viewed as philosophical rather than personal [ 45 ]. The cultural issues can be alleviated by understanding the concepts, norms and values of the setting, further to respecting and including perspectives of the various stakeholders.

The review findings suggest that qualitative methods are relevant when conducting mental health research. The qualitative methods are mostly used where there has been limited previous research and an absence of theoretical perspectives. The approach is also used to gather initial pilot data. More importantly, the qualitative methods are relevant when we want to understand sensitive issues, especially from consumers of mental health services, where the ‘lived experience is paramount [ 19 , 28 , 44 ]. Qualitative methods can help understand the experiences of consumers in the process of treatment, as well as their therapeutic relationship with mental health professionals. The experiences of consumers from qualitative data are particularly important in developing clinical policy [ 28 ]. The review findings find two paradigms of qualitative methods are used in mental health research. These paradigms are the interpretive and critical approach [ 32 ]. The interpretive qualitative method(s) include phenomenology, ethnography and narrative approaches [ 32 , 36 ]. Conversely, critical qualitative approaches are participatory action research and emancipatory approach. The review findings suggest that these approaches to qualitative methods need critical considerations, particularly when dealing with consumers of mental health services.

The review findings identified several sampling techniques used in mental health research. Quantitative studies, usually employ probability sampling, whilst qualitative studies use non-probability sampling [ 25 , 34 ]. The most common sampling techniques for quantitative studies are multi-stage sampling, which involves systematic, stratified, random sampling and consecutive sampling. In contrast, the predominant sampling approaches for qualitative studies are purposive [ 19 , 28 , 32 , 42 , 46 ], snowballing [ 30 , 32 , 46 ] and theoretical sampling [ 31 , 32 ].

The sampling of consumers of mental health services requires some important considerations. The sampling should consider the appropriateness and adequacy of the sampling approach, diversity and variety of consumers of services, attributes such as social, or cultural experiences, shared concerns related to the study, practical and organisational skills, as well as ethical and sensitivity issues are all relevant [ 31 , 32 , 46 ]. Sampling consumers of mental health services should also consider the homogeneity and heterogeneity of consumers. However, failure to address these considerations can present difficulty in sampling and subsequently result in selection and reporting bias in mental health research.

The evidence recommends several data collection approaches in collecting data in mental health research, including focus group discussion, extended in-depth interviews, observations, field notes, Delphi data collection and quasi-statistical techniques. The focus group discussions appear as an approach widely used to collect data from consumers of mental health services [ 19 , 28 , 30 , 31 , 41 , 44 , 47 ]. The focus group discussion appears to be a significant source of obtaining information. This approach promotes the participation of consumers with severe conditions, particularly at the group level interaction. Mental health researchers are encouraged to use this approach to collect data from consumers, in order to promote group level interaction. Additionally, field notes can be used to supplement information and to more deeply analyse the interactions of consumers of mental health services. Field notes are significant when wanting to gather detailed accounts about the subjective experiences of consumers of mental health services [ 40 ]. Field notes can help researchers to capture the gestures and opinions of consumers of mental health services which cannot be covered in the audio-tape recording. Particularly, the field note is relevant to complement the richness of information collected through focus group discussion from consumers of mental health services.

Furthermore, it was found that in-depth interviews can be used to explore specific mental health issues, particularly culturally marked disorders, their personal and interpersonal significance, patient and family explanatory models, individual and family coping styles, as well as symptom symbols and protective mediators [ 19 , 30 , 34 ]. The in-depth interviews are particularly relevant if the study is interested in the lived experiences of consumers without the contamination of others in a group situation. The in-depth interviews are relevant when consumers of mental health services are uncomfortable in disclosing their confidential information in front of others [ 31 ]. The lived experience in a phenomenological context preferably allows the consumer the opportunity to express themselves anonymously without any tacit coercion created by a group context.

The review findings recommend significant factors requiring consideration when collecting data in mental health research. These considerations include building confidence and trust between the researcher and consumers [ 31 , 37 ], gaining confidence of mental health professionals who manage consumers of mental health services, seeking ethical approval from the relevant committees, meeting consumers of services before data collection as well as arranging a mutually acceptable venue for the groups and providing transport services [ 41 , 47 ]. The evidence confirms that the identification of consumers of mental health services to participate in research can be facilitated by mental health professionals. Similarly, the cultural and social differences of the consumers of mental health services need consideration when collecting data from them [ 26 , 31 ].

Moreover, our review advocates that standardised assessment instruments can be used to collect data from consumers of mental health services, particularly in quantitative data. The self-completion instruments for collecting such information can be converted to interviewer-completion instruments [ 33 , 49 ]. The interviewer can read the questions to consumers of mental health services and record their responses. It is recommended that collecting data from consumers of mental health services requires significant preparation, such as training with co-investigators and representatives from consumers of mental health services [ 25 , 30 , 33 , 34 , 35 , 39 , 49 ]. The training helps interviewers and other investigators to understand the research project, particularly translating and adapting an instrument for the study setting with the aim to achieve content, semantic, concept, criteria and technical equivalence [ 30 ]. The evidence indicates that there is a need to adequately train interviewers when preparing for fieldwork to collect data from consumers of mental health services.

The evidence provides several approaches that can be employed to ensure quality assurance in mental health research involving quantitative methods. The quality assurance approach encompasses seeking informed consent from consumers of mental health services [ 26 , 37 ], pre-testing of tools [ 25 , 49 ], minimising non-response rates and monitoring of the data collection process [ 25 , 33 , 49 ]. The quality assurance process in mental health research primarily aims to achieve the highest reliability, validity and timeliness, to improve the quality of care provided. For instance, the informed consent exposes consumers of mental health services to the aim(s), methods, anticipated benefits and potential hazards and discomforts of participating in the study. Herein, consumers of mental health services who cannot respond to the inform consent process because of the severity of their illness can have it signed by their family caregivers. The implication is that researchers should determine which category of consumers of mental health services need family caregivers involved in the consent process [ 37 ].

The review findings advises that researchers should use pre-testing to evaluate the data collection procedure on a small scale and then to subsequently make any necessary changes [ 25 ]. The pre-testing aims to help the interviewers get acquainted with the procedures and to detect any potential problems [ 49 ]. The researchers can discuss the findings of the pre-testing and then further resolve any challenges that may arise prior to the actual field work being commenced. The non-response rates in mental health research can be minimised by re-approaching consumers of mental health services who initially refuse to participate in the study.

In addition, quality assurance for qualitative data can be ensured by applying the principles of credibility, dependability, transferability, reflexivity, confirmability [ 19 , 32 ]. It was found that the credibility of qualitative research in mental health is achieved through methodological and interpretive rigour [ 32 , 37 ]. The methodological rigour for assessing credibility relates to congruence, responsiveness or sensitivity to a social context, appropriateness, adequacy and transparency. By contrast, ensuring interpretive rigour is achieved through authenticity, coherence, reciprocity, typicality and permeability of researchers’ intentions, engagement and interpretation [ 32 , 46 ].

Strengths and limitations

The evidence has several strengths and limitations that require interpretation and explanation. Firstly, we employed a systematic approach involving five stages of problem identification, literature search, data evaluation, data synthesis and presentation of results [ 21 ]. Similarly, we searched six databases and developed a data extraction form to extract information. The rigorous process employed in this study, for instance, searching databases and data extraction forms, helped to capture comprehensive information on the subject.

The integrative review has several limitations largely related to the search words, language limitations, time period and appraisal of methodological quality of included papers. In particular, the differences in key terms and words concerning methodological issues in the context of mental health research across cultures and organisational contexts may possibly have missed some relevant articles pertaining to the study. Similarly, limiting included studies to only English language articles and those published from January 2000 to July 2018 could have missed useful articles published in other languages and those published prior to 2000. The review did not assess the methodological quality of included papers using a critical appraisal tool, however, the combination of clearly articulated search methods, consultation with the research librarian, and reviewing articles with methodological experts in mental health research helped to address the limitations.

The review identified several methodological issues that need critical attention when conducting mental health research. The evidence confirms that studies that addressed methodological considerations in conducting mental health research largely focuses on qualitative studies in a transcultural setting, in addition to lessons from multi-site surveys in mental health research. Specifically, the methodological issues related to the study design, sampling, data collection processes and quality assurance are critical to the research design chosen for any particular study. The review highlighted that researchers conducting mental health research can establish early collaboration, familiarise themselves with the topic, share information on the topic, negotiate to resolve any emerging problems and seek the contribution of clinical (or researcher) team members on the ground. In addition, the recruitment of consumers of mental health services should consider the appropriateness and adequacy of sampling approaches, diversity and variety of consumers of services, their social or cultural experiences, practical and organisational skills, as well as ethical and sensitivity issues.

The evidence confirms that in an attempt to effectively recruit and collect data from consumers of mental health services, there is the need to build confidence and trust between the researcher and consumers; and to gain the confidence of mental health service providers. Furthermore, seeking ethical approval from the relevant committee, meeting with consumers of services before data collection, arranging a mutually acceptable venue for the groups, and providing transport services, are all further important considerations. The review findings establish that researchers conducting mental health research should consider several quality assurance issues. Issues such as adequate training prior to data collection, seeking informed consent from consumers of mental health services, pre-testing of tools, minimising non-response rates and monitoring of the data collection process. More specifically, quality assurance for qualitative data can be achieved by applying the principles of credibility, dependability, transferability, reflexivity, confirmability.

Based on the findings from this review, it is recommended that mental health research should adequately consider the methodological issues regarding study design, sampling, data collection procedures and quality assurance issues to effectively conduct meaningful research.

Availability of data and materials

Not applicable

Abbreviations

focus group discussions

Joanna Briggs Institute

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

National Ethics Advisory Committee. Ethical guidelines for intervention studies: revised edition. Wellington (New Zealand): Ministry of Health. 2012.

Mann C. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J. 2003;20(1):54–60.

Article   CAS   Google Scholar  

DiPietro NA. Methods in epidemiology: observational study designs. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. 2010;30(10):973–84.

Article   Google Scholar  

Hong NQ, Pluyr P, Fabregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed Methods Appraisal Tool (MMAT). Canada.: Intellectual Property Office, Canada; 2018.

Creswell JW, Creswell JD. Research design: qualitative, quantitative, and mixed methods approaches: sage publications; 2017.

Google Scholar  

Wisdom J, Creswell JW. Mixed methods: integrating quantitative and qualitative data collection and analysis while studying patient-centered medical home models. Rockville: Agency for Healthcare Research and Quality; 2013.

Bonita R, Beaglehole R, Kjellström T. Basic epidemiology: World Health Organization; 2006.

Centers for Disease Control Prevention [CDC]. Principles of epidemiology in public health practice: an introduction to applied epidemiology and biostatistics. Atlanta, GA: US Dept. of Health and Human Services, Centers for Disease Control and Prevention (CDC), Office of Workforce and Career Development; 2012.

Parab S, Bhalerao S. Study designs. International journal of Ayurveda research. 2010;1(2):128.

Yang W, Zilov A, Soewondo P, Bech OM, Sekkal F, Home PD. Observational studies: going beyond the boundaries of randomized controlled trials. Diabetes Res Clin Pract. 2010;88:S3–9.

Department of Family Medicine (McGill University). Mixed Methods Appraisal Tool (MMAT) – Version 2011 Canada: McGill University; 2011 [Available from: http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/84371689/MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.08.21.pdf .

Besen J, Gan DS. A critical evaluation of clinical research study designs. J Investig Dermatol. 2014;134.

Axelrod DA, Hayward R. Nonrandomized interventional study designs (quasi-experimental designs). Clinical research methods for surgeons: Springer; 2006. p. 63–76.

Thiese MS. Observational and interventional study design types; an overview. Biochemia medica: Biochemia medica. 2014;24(2):199–210.

Velengtas P, Mohr P, Messner DA. Making informed decisions: assessing the strengths and weaknesses of study designs and analytic methods for comparative effectiveness research. National Pharmaceutical Council 2012.

Guerrera F, Renaud S, Tabbò F, Filosso PL. How to design a randomized clinical trial: tips and tricks for conduct a successful study in thoracic disease domain. Journal of thoracic disease. 2017;9(8):2692.

Bhide A, Shah PS, Acharya G. A simplified guide to randomized controlled trials. Acta Obstet Gynecol Scand. 2018;97(4):380–7.

Palinkas L, Horwitz SM, Chamberlain P, Hurlburt MS, Landsverk J. Mixed-methods designs in mental health services research: a review. Psychiatr Serv. 2011;62(3):255–63.

Palinkas L. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol. 2014;43(6):851–61.

World Health Organization [WHO]. Mental health: a state of well-being 2014 [Available from: http://www.who.int/features/factfiles/mental_health/en/ .

Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546–53.

Hopia H, Latvala E, Liimatainen L. Reviewing the methodology of an integrative review. Scand J Caring Sci. 2016;30(4):662–9.

Pearson A, White H, Bath-Hextall F, Apostolo J, Salmond S, Kirkpatrick P. Methodology for JBI mixed methods systematic reviews. The Joanna Briggs Institute Reviewers Manual. 2014;1:5–34.

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6(7):e1000097.

Alonso J, Angermeyer MC, Bernert S, Bruffaerts R, Brugha TS, Bryson H, et al. Sampling and methods of the European study of the epidemiology of mental disorders (ESEMeD) project. Acta Psychiatr Scand Suppl. 2004;109(420):8–20.

Baarnhielm S, Ekblad S. Qualitative research, culture and ethics: a case discussion. Transcultural Psychiatry. 2002;39(4):469–83.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Brown C, Lloyd K. Qualitative methods in psychiatric research. Adv Psychiatr Treat. 2001;7(5):350–6.

Davidsen AS. Phenomenological approaches in psychology and health sciences. Qual Res Psychol. 2013;10(3):318–39.

de Jong JT, Van Ommeren M. Toward a culture-informed epidemiology: combining qualitative and quantitative research in transcultural contexts. Transcultural Psychiatry. 2002;39(4):422–33.

Ekblad S, Baarnhielm S. Focus group interview research in transcultural psychiatry: reflections on research experiences. Transcultural Psychiatry. 2002;39(4):484–500.

Fossey E, Harvey C, McDermott F, Davidson L. Understanding and evaluating qualitative research. Aust N Z J Psychiatry. 2002;36(6):717–32.

Jacobi F, Wittchen H-U, Holting C, Sommer S, Lieb R, Hofler M, et al. Estimating the prevalence of mental and somatic disorders in the community: aims and methods of the German National Health Interview and examination survey. Int J Methods Psychiatr Res 2002;11(1):1–18.

Koch A, Vogel A, Holzmann M, Pfennig A, Salize HJ, Puschner B, et al. MEMENTA-‘Mental healthcare provision for adults with intellectual disability and a mental disorder’. A cross-sectional epidemiological multisite study assessing prevalence of psychiatric symptomatology, needs for care and quality of healthcare provision for adults with intellectual disability in Germany: a study protocol. BMJ Open. 2014;4(5):e004878.

Korver N, Quee PJ, Boos HB, Simons CJ, de Haan L, Investigators G. Genetic risk and outcome of psychosis (GROUP), a multi site longitudinal cohort study focused on gene–environment interaction: objectives, sample characteristics, recruitment and assessment methods. Int J Methods Psychiatr Res. 2012;21(3):205–21.

Larkin M, Watts S, Clifton E. Giving voice and making sense in interpretative phenomenological analysis. Qual Res Psychol. 2006;3(2):102–20.

Latvala E, Vuokila-Oikkonen P, Janhonen S. Videotaped recording as a method of participant observation in psychiatric nursing research. J Adv Nurs. 2000;31(5):1252–7.

Leese MN, White IR, Schene AH, Koeter MW, Ruggeri M, Gaite L. Reliability in multi-site psychiatric studies. Int J Methods Psychiatr Res. 2001;10(1):29–42.

Liu Z, Huang Y, Lv P, Zhang T, Wang H, Li Q, et al. The China mental health survey: II. Design and field procedures. Soc Psychiatry Psychiatr Epidemiol. 2016;51(11):1547–57.

Montgomery P, Bailey PH. Field notes and theoretical memos in grounded theory. West J Nurs Res. 2007;29(1):65–79.

Owen S. The practical, methodological and ethical dilemmas of conducting focus groups with vulnerable clients. J Adv Nurs. 2001;36(5):652–8.

Palinkas L, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health Ment Health Serv Res. 2015;42(5):533–44.

Palinkas L, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):44–53.

Razafsha M, Behforuzi H, Azari H, Zhang Z, Wang KK, Kobeissy FH, et al. Qualitative versus quantitative methods in psychiatric research. Methods Mol Biol. 2012;829:49–62.

Robins CS, Ware NC, Dosreis S, Willging CE, Chung JY, Lewis-Fernández R. Dialogues on mixed-methods and mental health services research: anticipating challenges, building solutions. Psychiatr Serv. 2008;59(7):727–31.

Robinson OC. Sampling in interview-based qualitative research: a theoretical and practical guide. Qual Res Psychol. 2014;11(1):25–41.

Schilder K, Tomov T, Mladenova M, Mayeya J, Jenkins R, Gulbinat W, et al. The appropriateness and use of focus group methodology across international mental health communities. International Review of Psychiatry. 2004;16(1–2):24–30.

Schoonenboom J, Johnson RB. How to construct a mixed methods research DesignWie man ein mixed methods-Forschungs-design konstruiert. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie. 2017;69(2):107–31.

Yin H, Phillips MR, Wardenaar KJ, Xu G, Ormel J, Tian H, et al. The Tianjin mental health survey (TJMHS): study rationale, design and methods. Int J Methods Psychiatr Res. 2017;26(3):09.

Download references

Acknowledgements

The authors wish to thank the University of Newcastle Graduate Research and the School of Nursing and Midwifery, for the Doctoral Scholarship offered to the lead author. The authors are also grateful for the support received from Ms. Debbie Booth, the Librarian for supporting the literature search.

Author information

Authors and affiliations.

School of Nursing and Midwifery, The University of Newcastle, Callaghan, Australia

Faculty of Health and Medicine, School Nursing and Midwifery, University of Newcastle, Callaghan, Australia

Anthony Paul O’Brien

Faculty of Business and Economics, Macquarie University, North Ryde, Australia

Rebecca Mitchell

You can also search for this author in PubMed   Google Scholar

Contributions

EB, APO’B, and RM conceptualized the study. EB conducted the data extraction, APO’B, and RM, conducted the second review of the extracted data. EB, working closely with APO’B and RM performed the content analysis and drafted the manuscript. EB, APO’B, and RM, reviewed and made inputs into the intellectual content and agreed on its submission for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Eric Badu .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Data extraction form. (DOCX 18 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Badu, E., O’Brien, A.P. & Mitchell, R. An integrative review on methodological considerations in mental health research – design, sampling, data collection procedure and quality assurance. Arch Public Health 77 , 37 (2019). https://doi.org/10.1186/s13690-019-0363-z

Download citation

Received : 13 November 2018

Accepted : 22 July 2019

Published : 10 October 2019

DOI : https://doi.org/10.1186/s13690-019-0363-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mental health
• Methodological approach
• Mixed methods
• Data collection

Archives of Public Health

ISSN: 2049-3258

• General enquiries: [email protected]

• Search This Site All UCSD Sites Faculty/Staff Search Term
• At A Glance
• Dean's Message
• Annual Report
• Bachelor of Science in Public Health
• Master of Public Health
• Master of Science in Biostatistics
• Master of Advanced Studies, Leadership of Healthcare Organizations
• Joint Doctoral Program in Public Health
• Doctoral Program in Biostatistics
• NIH T32 Programs
• UC San Diego - San Diego State University General Preventive Medicine Residency
• Faculty Mentor Training Program
• Biostatistics & Bioinformatics
• Climate and Environmental Sciences
• Community Health Services and Preventive Medicine
• Epidemiology
• Global Health
• Health Behavior
• Health Policy
• Public Mental Health and Substance Use
• Technology and Precision Health

Climate and Environmental Health

Health equity and global health justice, health services research and health policy, healthy aging and human longevity science, mental health and substance use, quantitative methods in public health.

• Women's Health and Reproductive Justice
• Research Partners
• Research Recharge Cores
• Faculty Research Labs
• Administration
• Academics and MSP Physicians
• Benefactors
• Recruitment
• Available Research Talent
• Commencement 2024
• COVID-19 In the News
• Our Research
• Research Programs

​​

To best serve the public good, the Herbert Wertheim School of Public Health emphasizes research in seven high priority areas: climate and environmental health, health equity and global health justice, healthy births and aging and longevity science, health services research, mental health and substance use, women's health and reproductive justice, and quantitative methods in public health. 

To remain nimble to a shifting environment and health concerns over time, the school has established seven research programs that focus on these areas, which reflect collective strengths within the school and key areas of public health need.

Women’s Health and Reproductive Justice

The mission of Women's Health and Reproductive Justice is broad and includes mandates to improve women’s health across a lifespan ranging from reproductive and gynecologic issues to chronic diseases of aging. Of particular interest is to study conditions where there are significant biological differences between the sexes in rates of susceptibility, symptoms and response to treatment.

• Open access
• Published: 29 May 2024

Exploring the use of body worn cameras in acute mental health wards: a mixed-method evaluation of a pilot intervention

• Una Foye 1 , 2 ,
• Keiran Wilson 1 , 2 ,
• Jessica Jepps 1 , 2 ,
• James Blease 1 ,
• Ellen Thomas 3 ,
• Leroy McAnuff 3 ,
• Sharon McKenzie 3 ,
• Katherine Barrett 3 ,
• Lilli Underwood 3 ,
• Geoff Brennan 1 , 2 &
• Alan Simpson 1 , 2  

BMC Health Services Research volume  24 , Article number:  681 ( 2024 ) Cite this article

Metrics details

Body worn cameras (BWC) are mobile audio and video capture devices that can be secured to clothing allowing the wearer to record some of what they see and hear. This technology is being introduced in a range of healthcare settings as part of larger violence reduction strategies aimed at reducing incidents of aggression and violence on inpatient wards, however limited evidence exists to understand if this technology achieves such goals.

This study aimed to evaluate the implementation of BWCs on two inpatient mental health wards, including the impact on incidents, the acceptability to staff and patients, the sustainability of the resource use and ability to manage the use of BWCs on these wards.

The study used a mixed-methods design comparing quantitative measures including ward activity and routinely collected incident data at three time-points before during and after the pilot implementation of BWCs on one acute ward and one psychiatric intensive care unit, alongside pre and post pilot qualitative interviews with patients and staff, analysed using a framework based on the Consolidated Framework for Implementation Research.

Results showed no clear relationship between the use of BWCs and rates or severity of incidents on either ward, with limited impact of using BWCs on levels of incidents. Qualitative findings noted mixed perceptions about the use of BWCs and highlighted the complexity of implementing such technology as a violence reduction method within a busy healthcare setting Furthermore, the qualitative data collected during this pilot period highlighted the potential systemic and contextual factors such as low staffing that may impact on the incident data presented.

This study sheds light on the complexities of using such BWCs as a tool for ‘maximising safety’ on mental health settings. The findings suggest that BWCs have a limited impact on levels of incidents on wards, something that is likely to be largely influenced by the process of implementation as well as a range of contextual factors. As a result, it is likely that while BWCs may see successes in one hospital site this is not guaranteed for another site as such factors will have a considerable impact on efficacy, acceptability, and feasibility.

Peer Review reports

Body worn cameras (BWC) are mobile audio and video capture devices that can be secured to clothing allowing the wearer to record some of what they see and hear. In England, these have been introduced in the National Health Service (NHS) as part of a violence reduction strategy [ 1 ] which emphasises the reduction of aggression and violence against staff. The NHS Staff Survey 2022 found that 14.7% of NHS staff had experienced at least one incident of physical violence from patients, relatives or other members of the public in the previous 12 months. Violent attacks on staff were found to contribute to almost half of staff illness [ 2 ]. Levels of violence against staff working in mental health trusts remain much higher than other types of healthcare providers [ 3 ]. Numerous reports internationally highlight the increased risks faced by staff working in psychiatric care [ 4 ], though studies have reported that both ward staff and mental health patients experience violence and feeling unsafe on inpatient wards [ 5 , 6 ].

Body worn cameras have been in use for over a decade within law enforcement, where they hoped to provide transparency and accountability within use-of-force incidents and in the event of citizen complaints against police [ 7 ]. It was believed that video surveillance would help identify integral problems within the organisation, improve documentation of evidence, reduce use-of-force incidents, improve police-community relations, and provide training opportunities for officers [ 8 ]. However, a recent extensive international systematic review by Lum et al. [ 9 ], found that despite the successes noted in early evaluations, the way BWCs are currently used by police may not substantially affect most officer or citizen behaviours. Irrespective of these findings, other public services such as train operators have been implementing BWCs for security purposes, with reductions reported in the number of assaults on railway staff [ 10 ].

A recent systematic review of BWC use in public sector services established that there is a poor evidence base supporting the use of BWCs in the reduction of violence and aggression [ 11 ]. Yet, we are seeing a swift increase in the use of BWCs in mental health settings with that aim, with few studies conducted on the use of BWC technology in inpatient mental health wards, and even fewer studies exploring staff or patients’ views. Two evaluations conducted in England reported mixed results with both increases and decreases in violence and aggression found, and variation between types of wards. There is some suggestion of a reduction in more serious incidents and the use of restraint, but quality of evidence is low [ 12 , 13 ].

The use of BWCs in mental healthcare settings for safety and security remains a contentious topic due to the lack of evidence regarding the influence that such technology has on preventing violence and aggression and the complex philosophical and ethical issues raised, particularly where many patients may lack capacity and/or are detained under mental health legislation [ 14 ]. Additionally, there are concerns that BWCs may be used as a ‘quick fix’ for staff shortages rather than addressing the wider systemic and resourcing issues facing services [ 15 ]. With little independent evaluation of body-worn cameras in mental health settings, many of these concerns remain unanswered. There is also limited understanding of this technology from an implementation perspective. Therefore, in this study we aimed to conduct an independent evaluation of the introduction of BWCs as a violence reduction intervention on two inpatient mental health wards during a six-month pilot period to explore the impact of using the technology, alongside an exploration of the facilitators and barriers to implementation.

Research aim(s)

To evaluate the implementation of BWCs on two inpatient mental health wards, including the impact on incidents, the acceptability to staff and patients, the sustainability of the resource use and ability to manage the use of BWCs on these wards.

Patient and public involvement

The research team included a researcher and independent consultant, each with lived experience of mental health inpatient care. In addition, we recruited and facilitated a six member Lived Experience Advisory Panel (LEAP). This group was made up of patients and carers, some of whom had experienced the use of BWCs. Members were of diverse ethnic backgrounds and included four women and two men. The LEAP provided guidance and support for the research team in developing an understanding of the various potential impacts of the use of BWCs on inpatient mental health wards. Members contributed to the design of the study, development of the interview schedule, practice interviews prior to data collection on the wards, and supported the analysis and interpretation of the data, taking part in coding sessions to identify themes in the interview transcripts. The LEAP met once a month for two hours and was chaired by the Lived Experience Research Assistant and Lived Experience Consultant. Participants in the LEAP were provided with training and paid for their time.

The pilot introduction of the body worn cameras was conducted within a London mental health Trust consisting of four hospital sites with 17 acute wards. The research team were made aware of extensive preparatory work and planning that was conducted at a directorate and senior management level prior to camera implementation, including lived experience involvement and consultation, and the development of relevant policies and protocols inclusive of a human rights assessment and legal consultation.

The pilot period ran from 25th April to 25th October 2022. Reveal (a company who supply BWCs nationally across the UK) provided the Trust with 12 Calla BWCs for a flat fee that covered use of the cameras, cloud-based storage of footage, management software, and any support/maintenance required during the pilot period. Cameras were introduced to two wards based on two hospital sites, with six cameras provided to each of the wards on the same date. Training on using the BWCs was provided by the BWC company to staff working on both wards prior to starting the pilot period. Ward one was a 20-bed male acute inpatient ward, representing the most common ward setting where cameras have been introduced. Ward two was a ten-bed male Psychiatric Intensive Care Unit (PICU), representing smaller and more secure wards in which patients are likely to present as more unwell and where there are higher staff to patient ratios.

To answer our research questions, we used a mixed-methods design [ 16 ]. Using this design allowed us to investigate the impact of implementing BWCs in mental health settings on a range of quantitative and qualitative outcomes. This mixed methods design allows the study to statistically evaluate the effectiveness of using BWCs in these settings on key dependent variables (i.e., rates of violence and aggression, and incidents of conflict and containment) alongside qualitatively exploring the impact that the implementation of such technology has on patients and staff.

To ensure that the study was able to capture the impact and effect of implementation of the cameras, a repeated measures design was utilised to capture data at three phases on these wards:

Pre-pilot data: data prior of the implementation of the BWCs (quantitative and qualitative data).

Pilot period data: data collected during the six-month pilot period when BWCs were implemented on the wards (quantitative and qualitative data).

Post-pilot: data collected after the pilot period ended and cameras had been removed from the wards (quantitative data only).

Quantitative methods

Quantitative data was collected at all three data collection periods:

Pre-period: Data spanning six months prior to the implementation of BWCs (Nov 21 to May 22).

Pilot period: Data spanning the six months of the Trusts pilot period of using BWCs on the wards (June 22 to Nov 22).

Post-pilot: Data spanning the six months following the pilot period, when BWCs had been removed (Dec 22 to May 23).

Quantitative measures

To analyse the impact of BWC implementation, we collected two types of incident data related to violence and aggression and use of containment measures, including BWCs. Combined, these data provide a view of a wide range of incidents and events happening across the wards prior to, during, and after the implementation and removal of the BWCs.

The patient-staff conflict checklist

The Patient-staff Conflict Checklist (PCC-SR) [ 17 ] is an end of shift report that is completed by nurses to collate the frequency of conflict and containment events. This measure has been used successfully in several studies on inpatient wards [ 18 , 19 , 20 ].The checklist consists of 21 conflict behaviour items, including physical and verbal aggression, general rule breaking (e.g., smoking, refusing to attend to personal hygiene), eight containment measures (e.g., special observation, seclusion, physical restraint, time out), and staffing levels. In tests based on use with case note material, the PCC-SR has demonstrated an interrater reliability of 0.69 [ 21 ] and has shown a significant association with rates of officially reported incidents [ 22 ].

The checklist was revised for this study to include questions related to the use of BWCs ( e.g., how many uses of BWCs happened during the shift when a warning was given and the BWC was not used; when a warning was given and the BWC was used; when the BWC was switched on with no warning given ) in order to provide insight into how the cameras were being used on each ward (see appendix 1). Ward staff were asked to complete the checklist online at the end of each shift.

Routinely collected incident data (via datix system)

To supplement the PCC-SR-R, we also used routinely collected incident data from both wards for all three data collection phases. This data is gathered as part of routine practice by ward staff members via the Datix system Datix [ 23 ] is a risk management system used widely across mental health wards and Trusts in the UK to gather information on processes and errors. Previous studies have utilised routinely collect data via this system [ 24 , 25 ]. Incidents recorded in various Datix categories were included in this study (see Table  1 ). Incidents were anonymised before being provided to the research team to ensure confidentiality.

Routinely collected data included:

Recorded incidents of violence and aggression.

Recorded use of restrictive practices including seclusion, restraint, and intra-muscular medication/rapid tranquilisations.

Patient numbers.

Staffing levels.

Numbers of staff attending BWC training.

Quantitative data analysis

Incident reports.

Incident reports retrieved from Datix were binary coded into aggregate variables to examine violence and aggression, self-harm, and other conflict as outlined in Table  1 . Multivariate analyses of variance (MANOVA) were used to identify differences in type of incident (violence against person, violence against object, verbal aggression, self-harm, conflict) for each ward. MANOVA was also used to examine differences in incident outcomes (severity, use of restrictive practice, police involvement) across pre-trial, trial, and post-trial periods for each ward. Incident severity was scored by ward staff on a four-point scale (1 = No adverse outcome, 2 = Low severity, 3 = Moderate severity, 4 = Severe). Use of restrictive practice and police involvement were binary coded for presence or absence. Analyses were conducted using SPSS [ 26 ].

Patient-staff conflict checklist shift-report – revised (PCC-SR-R; )

Data were condensed into weeks for analysis rather than shifts to account for variability in PCC-SR-R submission by shift. Linear regressions assessed the relationship between BWC use and incident outcome (severity, use of restrictive practice, police involvement).

Qualitative methods

We used semi-structured qualitative interviews to explore participants’ experiences of BWCs on the ward to understand the impact of their use as well as to identify any salient issues for patients, staff and visitors that align with the measures utilised within the quantitative aspect of this study. These interviews were conducted at two time points: pre-pilot and at the end of the six-month pilot period.

Sample selection, eligibility, and recruitment

Convenience sampling was used to recruit staff and patients on wards. Researchers approached ward managers to distribute information sheets to staff, who shared that information with patients. Staff self-selected to participate in the study by liaising directly with the research team. Patients that were identified as close to discharge and having capacity to consent were approached by a clinical member of the team who was briefed on the study inclusion criteria (see Table  2 ). The staff member spoke with the patient about the study and provided them with a copy of the information sheet to consider. If patients consented, a member of the research team approached the participant to provide more information on the study and answer questions. After initial contact with the research team, participants were given a 24-hour period to consider whether they wanted to participate before being invited for an interview.

Participants were invited to take part in an interview within a private space on the ward. Interviews were scheduled for one hour with an additional 15 min before and after to obtain informed consent and answer any questions. Participation was voluntary and participants were free to withdraw at any time. To thank patients for their time, we offered a £10 voucher following the interview. Interviews were audio-recorded and saved to an encrypted server. Interview recordings were transcribed by an external company, and the research team checked the transcripts for accuracy and pseudonymised all participants. All transcripts were allocated a unique ID number and imported to MicroSoft Excel [ 27 ] for analysis.

Qualitative data analysis

Qualitative data were analysed using a framework analysis [ 28 ] informed by implementation science frameworks. Our coding framework used the Consolidated Framework for Implementation Research (CFIR) [ 29 ], which is comprised of five major domains including: Intervention Characteristics, Implementation Processes, Outer Setting, Inner Setting, and Characteristics of the Individual. Each domain consists of several constructs that reflect the evidence base of the types of factors that are most likely to influence implementation of interventions. The CFIR is frequently used to design and conduct implementation evaluations and is commonly used for complex health care delivery interventions to understand barriers and facilitators to implementation. Based on its description, the CFIR is an effective model to address our research question, particularly given the complexity of the implementation of surveillance technology such as BWCs in this acute care setting.

The initial analytic stage was undertaken by eight members of the study team with each researcher charting data summaries onto the framework for each of the interviews they had conducted on MicroSoft Excel [ 27 ]. Sub-themes within each broad deductive theme from our initial framework were then derived inductively through further coding and collaborative discussion within the research team, inclusive of Lived Experience Researcher colleagues. Pseudonyms were assigned to each participant during the anonymisation of transcripts along with key identifiers to provide context for illustrative quotes (e.g., P = patient, S = staff, A = acute ward, I = Intensive Care, Pre = pre-BWC implementation interview, Post = Post BWC implementation interview).

All participants gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the Health Research Authority: London - Camden & Kings Cross Research Ethics Committee (IRAS Project ID 322,268, REC Reference 23/LO/0337).

Quantitative results

Exploring how body worn cameras were used during the pilot period.

Analysis of the PCC-SR-R provides information about how the BWCs were used on a day-to-day basis during the pilot period. Out of 543 total shift reports completed, BWC use was reported 50 times, indicating that BWCs were used on less than 10% of shifts overall; 78% of those deployments were on the Acute ward (see Figure 1 ). Overall, the majority of deployments happened as activations without a warning being given ( n  = 30, 60% of activations), 19 times the BWC was deployed with a warning but the camera was not activated (38%), and only one was the camera activated without a warning being given (2%).

BWC use by ward per week of pilot (no data available before week 6 on Ward 1)

According to the PCC-SR-R, a total of 227 incidents of aggression occurred during the pilot period across both wards (see Table  3 ). Overall, there were small statistically significant correlations between BWC usage and certain types of conflict, aggression, and restrictive practice. Results found that BWC use was positively correlated with verbal aggression and use of physical restraint. BWC use was moderately positively correlated with verbal aggression ( r  = .37, p  < .001). This indicates that BWCs were more likely to be used in incidents involving verbal aggression, which do not tend to be documented in Datix. Similarly, BWC use was moderately positively correlated with physical restraint ( r  = .31, p  < .001) indicating that they were also more likely to be used alongside physical restraint.

Exploring the impact of BWCs utilising routinely collected ward data

Acute ward results.

Routine data collected via Datix records were used to examine differences in frequency of conflict and aggression, incident severity, and use of containment measures before, during, and after introduction of BWCs on each trial ward (see Table  4 ).

There was no effect of trial period on incident type ( F (10, 592) = 1.703, p  = .077, Wilk’s Λ = 0.945), meaning there was no discernible difference in the type of incidents that occurred (E.g., verbal aggression, physical aggression) before, during, and after the pilot phase.

Incident outcomes

There was an effect of trial period on incident outcomes ( F (6, 596) = 10.900, p  < .001, Wilk’s Λ = 0.812). Incident severity was statistically significantly higher in the trial and post-trial periods compared to the pre-trial period. Use of restrictive practice was significantly lower in the post-trial period compared to the pre-trial and trial period. Police involvement was also lower in the post-trial period compared to the pre-trial and trial periods (see Table  5 ).

Results for the psychiatric intensive care unit

There was an effect of trial period on incident type ( F (10, 490) = 4.252, p  < .001, Wilk’s Λ = 0.847). Verbal aggression was statistically significantly higher in the post-trial period compared to the pre and trial periods. Self-harm was statistically significantly higher in the trial period compared to the pre-trial and post-trial periods. There were no differences in violence against a person ( p  = .162), violence against an object or conflict behaviour (see Table  4 ).

There was a statistically significant difference in incident outcome across the trial periods ( F (6, 494) = 12.907, p  < .001, Wilk’s Λ = 0.747). There was no difference in incident severity or police involvement. However, use of restrictive practice was statistically significantly higher in the pre-trial period, reducing in the test period, and reducing further in the post-trial period (see Table  5 ).

Qualitative findings

A total of 22 participants took part in interviews: five patients and 16 staff members. During the pre-pilot interviews a total of nine staff took part (five in the acute ward, four in the PICU ward) and two patients (both from the acute ward). After the pilot period, a total of eight staff took part (four from each ward) and three patients (all from the acute ward). Table  6 includes a full description of participants.

Below we have presented the key themes aligning to the five core CFIR categories of Intervention Characteristics, Characteristics of Individuals, The Process of Implementation, the Inner Setting, and The Outer Setting (see Table  7 ).

Intervention characteristics

Design and usability of wearing a bwc on the ward.

When discussing the use of the BWCs, staff noted a range of design issues related to the cameras that they said impacted on their use and acceptance of the cameras. This included the nature of the camera pulling on clothing necklines (a particular issue for female staff working on male wards), and overheating causing discomfort and irritation to skin, challenges with infection control, as well as the issue of cameras in a mental health setting where they can be easily grabbed, thrown and broken during an incident. Staff often cited these design issues as related to the lack of proactive use of the cameras on the wards.

There were issues around the devices getting overheated or about it going on your clothing, it pulls down the top… we had one person who was leading on it, whenever he was around, of course, the camera was being used, but if he wasn’t there, people weren’t as proactive in using the camera. Petra (f), Staff, A, Post.

There were also issues with staff forgetting to wear the cameras, forgetting to switch them on during incidents, and forgetting to charge them at the end of the shift, reducing the potential use of the cameras by other staff. These were perceived as key logistical issues prior to the pilot and were reported as issues at the end of the pilot by several staff on the wards.

The practicalities of will they actually turn it on in those sorts of incidents, I don’t know. Just little stuff as well, like if they don’t put it back on the docking station, so you think you’re charging it for next shift but then it’s not charged and the battery is dead, that’s one less camera to use, so little stuff. Jamal (m), Staff, A, Pre.

In relation to usability, staff noted that the cameras were small and easy to use given their simple single switch interface. It was felt that not having to upload and manage the data themselves made cameras more user friendly and usable by staff members. Protocols put into place such as signing the cameras in and out, and allocation for use during shifts were likened to procedures in place for other security measures therefore the implementation of this for the BWCs was viewed as easy for many staff.

It’s just like the ASCOM alarms that we wear. There’s a system to sign in and sign out, and that’s it. Alice (f), Staff, A, Pre.

While staff were generally positive about the usability of the cameras, some were cautious of with concerns for those less confident with technology.

… you have to be conscious that there’s some people – it’s quite easy to use, but I can say that because I’m alright using devices and all that but there’s some that are older age or not that familiar with using devices that may struggle with using it… they’re feeling a bit anxious and a bit scared, if they’re not familiar with it then they won’t use it. Jamal (m), Staff, A, Pre.

Evidence strength and quality: do BWCs change anything?

There were conflicting reports regarding the potential benefits of using BWCs on the wards, with both staff and patients reporting mixed perceptions as to whether the cameras might reduce violence and aggression. In the pre-pilot interviews, some staff reported feeling that the BWCs may have a positive impact on reducing physical violence.

I think it’s going to reduce violence and aggression on the ward…I don’t think they’ll want to punch you…they might be verbally abusive but in terms of physical that might reduce. Sarah (f), Staff, I, Pre.

Patients however noted that the cameras might hold staff to account of their own behaviours and therefore may improve care, however they felt that this impact would wear off after the first few months after which people might forget about the cameras being there.

Now they’ve got the body cams, it’s going to be a lot of changes. They’ll think, ‘Ooh well he’s on tape’. So, it might do something to their conscience, they actually start to listen to patients… until the novelty wears off and it might go back to square one again. Ian (m), Patient, A, Pre.

One staff member suggested that incident rates had reduced following introduction of the BWCs, but they remained unsure as to whether this was due to the cameras, reflecting that violence and aggression on wards can be related to many factors.

I know our violence and aggression has reduced significantly since the start of the cameras pilot… I don’t know, because obviously wearing the camera’s one thing, but if they weren’t in use, I don’t know maybe just the presence of the camera made a difference. But yeah, it’s hard to tell. Petra (f), Staff, A, Post.

In contrast, several staff reported that they had seen limited evidence for such changes.

I used it yesterday. He was aggressive and I used it, but he even when I was using [it] he doesn’t care about the camera… it didn’t make any difference… It doesn’t stop them to do anything, this camera does not stop them to do anything. Abraham (m), Staff, I, Post.

Some staff suggested that in some circumstances the cameras increased patient agitation and created incidents, so there was a need to consider whether the BWCs were going to instigate aggression in some circumstances.

There has been with a few patients because they will threaten you. They will tell you, ‘if you turn it on, I’m gonna smash your head in’. So incidents like that, I will not turn it on… Yeah, or some of them will just tell you, ‘if you come close by, I’m going to pull that off your chest’. So things like that, I just stay back. Ada (f), Staff, A, Post.

One rationale for a potential lack of effectiveness was noted by both staff and patients and was related to the levels of acute illness being experienced by patients which meant that for many they were too unwell to have insight into their own actions or those of staff switching on the cameras.

We’ve had instances where patients are so unwell that they just don’t care. You switch on the camera, whether you switch it on or not, it doesn’t really change the behaviour. ‘All right, okay, whatever switch it on’. They’re so unwell, they’re not really understanding. Petra (f), Staff, A, Post. It might make [staff] feel safer as a placebo effect, but I don’t think it would necessarily make them safer… I think the people that are likely to attack a member of staff are crazy enough that they’re not gonna even consider the camera as a factor. Harry (m), Patient, A, Pre.

This lack of evidence that the cameras were necessarily effective in reducing incident rates or severity of incidents may have had an impact on staff buy-in and the use of the cameras as a result. One staff member reflected that having feedback from senior management about the impact and evidence would have been useful during the pilot period to inform ward staff whether the cameras were influencing things or not.

Staff want feedback. I don’t think we’ve had any since we’ve had the cameras… it would be nice to get feedback from, I don’t know, whoever is watching it, and stuff like that. Ada (f), Staff, A, Post.

Relative advantage: are BWCs effective and efficient for the ward?

Due to a combination of personal beliefs related to BWCs, the lack of evidence of their impact on violence and aggression, and other elements of care and culture on the wards, a number of staff and patients explored alternative interventions and approaches that may be more beneficial than BWCs. Both staff and patients suggested that Closed Circuit Television (CCTV) as an intervention that provided the transparency of using cameras and video footage but with an independent perspective. This was felt by many to remove the bias that could be introduced in BWC use as the video capture didn’t require staff control of the filming.

I feel like [BWCs] puts all the power and trust into the hands of the staff and I feel that it would be better to have CCTV on the ward because CCTV is neutral. Harry (m), Patient, A, Pre. I have control over that [BWC recording] … It kind of gives that split as well between staff and patients. You can tell me or I can tell you when to switch it on. Whereas I feel like a CCTV camera is there all the time. Nobody’s asking to switch it on. It’s there. If you wanted to review the footage you can request it, anyone can request to view the footage for a legitimate reason. Whereas the camera can come across as if you’re threatening. Petra (f), Staff, A, Post.

In addition, some participants reflected that the nature and design of BWCs meant that unless staff were present for an incident it wouldn’t be captured, whereas CCTV has the advantage of being always present.

If there’s CCTV, then it’s the same thing, you get me. Like, if its body worn cameras that people can always do things away from staff. They can always go down to that corridor to have their fight or go to the side where staff ain’t gonna see them to have their fight, but with CCTV you can’t do that. Elijah (m), Patient, A, Post.

In addition to exploring technological and video-based interventions, many staff noted that the key tool to violence reduction had to be the use of de-escalation skills, noting that the use of communication and positive relationships had to be the primary tool before other interventions such as BWCs or CCTV.

We do a lot of verbal de-escalation. So we got our destress room now still open. That has a punch bag, and it’s got sensory tiles, and the aim and hope is that when people do get frustrated, because we’re all human. We all get annoyed at anything or many little things in life. There is the aim that they go into that room and start punching the bag instead of property and damaging furniture. But we also are working really hard on verbal de-escalation and actually trying to listen to patients and talk to them before anything else. And that’s helped a lot. And between this kind of shared, or role modelling, where while we’re showing staff, actually even spending an extra 20 min is okay. If it means you’re not going to end up having to restrain a patient. Petra (f), Staff, A, Post.

By using communication skills and de-escalation techniques skilfully, some staff felt there was no need to utilise the BWCs. One concern with the introduction of the BWCs for staff was that the use of this technology may negatively impact on trust and relationships and the use of de-escalation.

Some situations I feel like it can make a situation worse sometimes… I think a lot of situations can be avoided if you just talk with people…. Trying to find out why they’re angry, trying to just kind of see it from their point of view, understand them… I think maybe additional training for verbal de-escalation is needed first. Patrick (m), Staff, A, Post.

Characteristics of individuals

Staff and patients’ knowledge and beliefs about the intervention.

Overall, there were mixed views among both staff and patients as to whether cameras would reduce incidents, prior to and after the pilot period. When considering the possible impact on violence and aggressive incidents there was a view among staff that there was the need for a nuanced and person-centred view.

All the patients that come in, they’re different you know. They have different perceptions; they like different things… everyone is different. So, it just depends. We might go live, and then we have good feedback because the patients they are open and the understand why we have it, and then as they get discharged and new patients come in it might not go as well. It just depends. Serene (f), Staff, A, Pre.

As a result of the desire to be person-centred in the use of such interventions, one staff member noted that they weighed-up such consequences for the patient before using the BWC and would make decisions not to use the camera where they thought it may have a negative impact.

Actually, with this body worn camera, as I did mention, if a patient is unwell, that doesn’t, the patient will not have the capacity to I mean, say yes, you cannot just put it on like that. Yeah, I know it’s for evidence, but when something happens, you first have to attend to the patient. You first have to attend to the patient before this camera is, for me. Ruby (f), Staff, I, Post.

Some staff questioned the existing evidence and theories as to why BWCs work to reduce incidents, and instead noted that for some people it will instigate an incident, while others may be triggered by a camera.

I’m on the fence of how that is going to work because I know the evidence is that by telling a patient ‘look if you keep escalating I’m gonna have to turn this on’, but I know several of our patients would kind of take that as a dare and escalate just to spite so that you would turn it on. Diana (f), Staff, A, Pre.

In contrast, some staff felt the cameras helped them feel safer on wards due to transparency of footage as evidence for both staff and patients.

They [staff] need to use it for protection, for recording evidence, that type of thing… They can record instances for later evidence. Yeah, for them as well. Safer for them and for patients because you can also have the right to get them to record, because a patient might be in the wrong but sometimes it may be the staff is in the wrong position. And that’s achieving safety for patients as well. Yeah, I think it works both ways. Dylan (m), Patient, A, Post.

Positive buy-in was also related to the potential use of the intervention as a training, learning or reflective tool for staff to improve practice and care and promote positive staff behaviour.

If you know that your actions might be filmed one way or the other, that would make me to step up your behaviour to patients… if you know that your actions can be viewed, if the authority wants to, then you behave properly with patients so I think that will improve the quality of the care to patient. Davide (m), Staff, I, Pre.

While there were some positive attitudes towards the cameras, there remained considerable concerns among participants regarding the transparency of camera use to collate evidence in relation to incidents as it was widely noted that the cameras remain in staff control therefore there is an issue in relation to bias and power.

I do think my gut would say that it wouldn’t necessarily be well received. Because also I think people feel like prisoners in here, that’s how some of the patients have described their experience, so in terms of the power dynamic and also just – I think that can make one feel a bit, even worse, basically, you know? Leslie (m), Staff, A, Pre.

These issues lead to staff reporting they didn’t want to wear the camera.

I’d feel quite uncomfortable wearing one to be honest. Leslie (m), Staff, A, Pre.

The staff control of the cameras had a particular impact on patient acceptability of the intervention as it led to some patients viewing BWCs as being an intervention for staff advantage and staff safety, thus increasing a ‘them and us’ culture and leading to patient resistance to the cameras. This was particularly salient for those with prior negative experiences of police use of cameras or mistrust in staff.

I feel like the fact that the body worn cameras is gonna be similar to how the police use them, if a staff member has negative intent toward a patient, they would be able to instigate an incident and then turn the camera on and use the consequences of what they’ve instigated to expect restraint or injection or whatever else might happen. So, I feel like it would be putting all the power and trust into the hands of the staff and I feel that it would be better to have CCTV on the ward because CCTV is neutral. Whereas, the body worn camera, especially with some of the personality conflicts/bad attitudes, impressions I’ve had from certain members of staff since I’ve been here, I feel like body worn cameras might be abused in that way possible. Harry (m), Patient, A, Pre.

Perceived unintended consequences and impact on care

Prior to the implementation there were concerns from staff that the introduction of BWCs could have consequences beyond the intended use of reducing violence and aggression, unintentionally affecting a range of factors that may impact on the overall delivery of care. There was a key concern regarding the potential negative impact that cameras may have for patients who have paranoia or psychosis as well as for those who may have prior traumatic experiences of being filmed.

It might have negative impacts on these patients because I’m thinking about kind of patients with schizophrenia and things like that who already have paranoid delusions, thinking that people are after them, thinking that people are spying on them, people are watching them, and then seeing kind of cameras around. It might have negative impacts on them. Tayla (f), Staff, I, Pre. When I was admitted I was going through psychosis… I don’t want to be filmed and things like that. So you just see a camera, a guy with a camera on, you are like, are you filming me? Elijah (m), Patient, A, Post.

There was also a considerable concern among both staff and patients that the use of cameras would have a negative impact on the therapeutic relationship between staff and patients. This was felt to be related to the implication that the cameras enhanced a ‘them and us’ dynamic due to the power differential that staff controlling the cameras can create, likened to policing and criminalisation of patients. With the potential of a negative impact on relationships between staff and patients, staff suggested they may be disinclined to use BWCs if it would stop patients speaking to them or approaching them if they needed support.

Yeah, I think it would probably damage [the therapeutic relationship] because I think what’s probably quite helpful is things that maybe create less of a power difference. I think to some extent, [the BWC] might hinder that ability. Like for example imagine going to a therapist and them just like ‘I’ve got this camera in the corner of the room and it’s gonna be filming our session and just in case – or like, just in case I feel that you might get aggressive with me’. Um, I don’t think that’s going to help the therapeutic relationship! Jamal (m), Staff, A, Pre. When you get body worn cameras on there, the relationship as well between staff and patients, is just gonna instantly change because you’re looking like police! Elijah (m), Patient, A, Post.

In contrast, a minority of staff felt that the presence of cameras may improve relationships as they provide transparency of staff behaviour and would encourage staff to behave well and provide high quality care for patients.

It will also help how, improve the way we look at the patients… because if you know that your actions might be filmed one way or the other, that would make me to step up your behaviour you know… you behave properly with patients so I think that will improve the quality of the care to patient. More efficiently, more caring to patient. Davide (m), Staff, I, Pre.

The process of implementation

Planning: top-down implementation.

Staff perceived that BWC implementation directives had been given by senior management or policy stakeholders whom they felt viewed the process from a position of limited understanding due to a lack of ‘frontline’ mental health service experience. This led to a lack of faith amongst staff, and a perception that funds were being misspent.

They sit up there, they just roll it out and see how it works, how it goes. They waste a whole lot of money, millions or whatever, thousands of pounds in it, and then they see that ‘Oh, it’s not gonna work’. They take it back and all of that. Before coming out with it, you need to come speak to us… they just sit up there drinking tea and coffee, and then they’re just like, Oh, yeah, well, let’s do it this way…come stay with these people, work with them, for just I give you a 12 h shift, stay with them. Richard (m), Staff, I, Post.

This was exacerbated when staff felt there was a lack of consultation or explanation.

we don’t always get the ins and outs of certain things…We know that the cameras are coming in and stuff like that, but you know, and obviously it’s gone through every avenue to make sure that it’s fine. But then sometimes we don’t always know the ins and outs to then explain to people why we have the cameras. Patrick (m), Staff, A, Post.

It was also highlighted that due to multiple initiatives being implemented and directives handed down in parallel, staff felt negative towards interventions more widely, with the BWCs being ‘ just another thing to do’ , adding to workload for staff and reducing enthusiasm to use the cameras.

it’s not just to do with the camera, I just think there’s lots of changes happening at once, and there’s loads of new things being constantly introduced that people are just thinking oh it’s another thing. I think that’s what it is more than the camera itself. Alice (f), Staff, A, Pre.

Execution: training, Use and Ward Visibility

Overall, there was a lack of consistency amongst staff in their understanding of the purpose and processes of using the BWCs on the wards.

What do you do, do you record every single thing or, I don’t know. Do you record like, if a patient said, I want to talk to you, confidential, you go sit in a room, do you record things like those or is it just violence and aggression? Ada (f), Staff, A, Post.

The lack of clarity regarding the purpose of the intervention and the appropriate use of the cameras was felt to impact staffs’ attitudes and acceptance of using them and contributed to a lack of transparency or perhaps trust regarding the use of any subsequent video footage.

I think if the importance of the recording was explained a bit more…and how it would improve things, I think people would use it more… that’s why I don’t think it’s always used sometimes… if you’re not sure why some of it’s important, then you’re not going to see the value…I think if you’re gonna keep with them, it’s about updating the training, teaching staff when to use it, then where does that information go? How does that look in terms of improving? Just a bit of transparency, I think. But when you don’t know certain things it’s a bit hard to get behind something or back it, you know? Patrick (m), Staff, A, Post.

The lack of information about the purpose and processes related to the intervention was also seen among patients, with most patients noting that they hadn’t received information about the cameras during their admissions.

No information at all. I don’t think any of the patients know about it. Toby (m), Patient, A, Post.

While training was provided it was widely felt that it was insufficient to provide understanding about the purpose of the cameras or the more in-depth processes beyond operational aspects such as charging and docking. Several staff interviewed were unaware of the training, while others noted that they had an informal run-through by colleagues rather than anything formal.

What training are you talking about?… I wasn’t here, so I was taught by my colleague. I mean, from what I was taught, to operate the camera, and to give a warning to the patient that you’re going to use the camera. Nevis (f), Staff, A, Post.

Longer training with further details beyond operational use was felt to be needed by staff.

I think the training should have to be longer, even if it’s like an hour or something… Like what situations deem the camera to be… more information on the cameras, when to use it, why it’s used, and I think if the importance of the recording was explained a bit more and what it was doing and how that recording would go and how it would improve things. Patrick (m), Staff, A, Post.

Furthermore, there was a need for training to be on a rolling basis given the use of bank staff who were not trained to use the cameras or to understand the proper processes or purpose of using the BWCs, which could leave them vulnerable to misuse or abuse.

We have bank staff [who aren’t trained] so they say ‘I don’t know how to use that camera you are giving me’. Nevis (f), Staff, A, Post.

The inner setting

Ward context: acceptance of violence and aggression is part of the job.

It was widely believed by staff that the nature of working on a mental health ward included accepting that violence and aggression was part of the job. This was not seen as an acceptance of violence but more that the job was providing care for individuals who are mentally unwell, and confusion, fear, frustration and aggression can be part of that. As a result, there was an ambivalence among some staff that the introduction of cameras would change this.

I think like in this line of work, there’s always that potential for like risky behaviours to happen. I’m not sure if putting the camera on will make much difference. Patrick (m), Staff, A, Post.

Staff noted that because of the nature of the job, staff are used to managing these situations and they understood that it was part of the job; therefore, it was unlikely that they would record everything that on paper might be considered an incident.

There’s also enough things that happen here, so I don’t think they would record [the incidents] because it’s just another day here. You know what I’m saying… [staff] can just say, ‘Stop, go back to your room and leave it at that and that kind of be the end of it’. Dylan (m), Patient, A, Post. We are trained for it. Eveline (f), Staff, I, Pre.

This acceptance that incidents are a hazard of mental healthcare was linked to staff’s acknowledgment that many factors make up the complexity of violence and aggression including the nature of individual patients, acuity levels, ward atmosphere, staffing levels, access to activities, leave and outside space. The interplay of multiple factors creates a context in which frustrations and incidents are likely, thus become part of the everyday and ‘normal’ life on the ward for staff and patients alike.

I feel like, you know, how in GP services you say, zero tolerance to abusive language, or any kind of harassment. I don’t think there is that on a psychiatric ward you are kind of expected to take all the abuse and just get on with it. Petra (f), Staff, A, Post.

With staff reported having a higher threshold for these behaviours it was perceived that this was likely to impact on the efficiency of the intervention as staff would be less likely to consider a situation as violent but more ‘ part of the job’ .

Reactive nature of the ward and incidents

Most participants noted that the ward context is always changing with people being admitted and discharged, with daily staff changes and wider turnover of staff, so things are never static and can change at any point. This reflects the dynamic nature of the ward which creates a complex moving picture that staff need to consider and react to.

[the atmosphere] it’s very good at the moment. If you had asked me this two weeks ago, I would say, ‘Oh, my gosh’. But it changes… The type of patient can make your whole ward change… it depends on the client group we have at the time. Nevis (f), Staff, A, Post.

Staff noted that a key limitation of using the cameras to reduce incidents was the reactive nature of the environment and care being provided. This was felt to impact on the feasibility and use of the cameras as staff noted that they often react to what is happening rather than thinking to ‘ put the camera on first ’. It was felt by staff with experience of reacting to incidents that the failure to use BWCs during these processes were linked to staff’s instincts and training to focus on patients as a priority.

Say for instance, you’re in the office, and two patients start fighting, or a patient attacks someone and, all you’re thinking about is to go there to stop the person. You’re not thinking about putting on any camera. You understand? So sometimes it’s halfway through it, somebody might say, ‘Has anybody switched the camera on’? And that’s the time you start recording… If something happens immediately, you’re not thinking about the camera at that time, you’re just thinking to just go, so yeah. Nevis (f), Staff, A, Post.

Incidents happen quickly and often surprise staff, therefore staff react instantly so are not thinking about new processes such as recording on the cameras as this would slow things down or is not in the reactive nature needed by staff during such incidents.

When you’re in the middle of an incident and your adrenaline’s high, you’re focusing on the incident itself. It’s very difficult for you to now remember, remind yourself to switch on the camera because you’re thinking, patient safety, staff safety, who’s coming to relieve you? What’s going on? Who’s at the door? Petra (f), Staff, A, Post.

In addition, the need for an immediate response meant that it was felt that by the time staff remember to, or have the chance to, switch the camera on it was often too late.

Sometimes in the heat of moments and stuff like that, or if the situation’s happening, sometimes you don’t always think to, you know, put your camera on. Patrick (m), Staff, A, Post.

Outer setting

Resources: staffing.

Issues related to staffing were highlighted by several participants as a key problem facing mental health wards thus leading to staff having higher workloads, and higher rates of bank and agency staff being used on shift and feeling burnt-out.

Out of all the wards I’ve been on I’d say this is the worst. It’s primarily because the staff are overworked…it seems like they spend more time doing paperwork than they do interacting with the patients. Harry (m), Patient, A, Pre. We’re in a bit of a crisis at the minute, we’re really, really understaffed. We’re struggling to cover shifts, so the staff are generally quite burnt out. We’ve had a number of people that have just left all at once, so that had an impact… Staff do get frustrated if they’re burnt out from lack of staff and what have you. Alice (f), Staff, A, Pre.

It was noted by one participant that the link of a new intervention with extra workload was likely to have a negative impact on its acceptability due to these increasing demands.

People automatically link the camera to then the additional paperwork that goes alongside it. It’s like, ‘Oh god, if we do this, we’ve got to do that’, and that could play a part. Petra (f), Staff, A, Post.

One staff member noted that the staffing issue meant there were more likely to be bank staff on wards so the care of patients may be affected as temporary staff may be less able to build meaningful therapeutic relationships.

So obviously there is the basic impact on safety of not having adequate staffing, but then there’s the impact of having a lot of bank staff. So obviously when you have permanent staff they get to know the patients more, we’re able to give them the more individualised care that we ideally should be giving them, but we can’t do that with bank staff. Diana (f), Staff, A, Pre.

It was also suggested that staffing levels and mix often made it more difficult to provide activities or facilitate escorted leave which can lead to patients feeling frustrated and becoming more aggressive.

So you know there is enough staff to facilitate the actual shift, so you know when there’s less staff like you say you’ve got people knocking at the door, but then you don’t have staff to take people out on leave straight away, that all has a rippling effect! Serene (f), Staff, A, Pre.

Wider systemic issues

Overall, there was a concern that the introduction of BWCs would not impact on wider, underlying factors that may contribute to frustration, aggression and incidents on wards. Providing a more enhanced level of care and better addressing the needs of patients was felt to be central to helping people but also reducing the frustration that patients feel when on the ward.

… for violence and aggression, [focus on] the mental health side of things like therapy and psychology should be compulsory. It shouldn’t be something you apply for and have to wait three or four weeks for. I think every person should, more than three or four weeks even, months even… we need psychology and therapists. That’s what will stop most violence, because psychologists and a therapist can edit the way that they speak to people because they’ve been given that skill depending on the way the person behaves. So that’s what we need regularly… not like all this dancing therapy, yoga therapy. That’s a person, that you come and you actually sit down and talk through your shit with them. That will help! Elijah (m), Patient, A, Post. There’s a lack of routine and I think there’s a lack of positive interaction between the patient and the staff as well. The only time you interact with a member of staff is if you’re hassling them for something, you have to hassle for every little thing, and it becomes a sort of, frustration inducing and like I’m a very calm person, but I found myself getting very fucking angry, to be honest, on this ward just because out of pure frustration… there’s bigger problems than body worn cameras going on. Harry (m), Patient, A, Pre.

Staff agreed that there was a need to invest in staff and training rather than new technologies or innovations as it is staff and their skills behind the camera.

It’s not the camera that will do all of that. It’s not making the difference. It’s a very good, very beautiful device, probably doing its job in its own way. But it’s more about investing in the staff, giving them that training and making them reflect on every day-to-day shift. Richard (m), Staff, I, Post.

There was felt to be a need to support staff more in delivering care within wards that can be challenging and where patients are unwell to ensure that staff feel safe. While in some circumstances the cameras made some staff feel safer, greater support from management would be more beneficial in making staff feel valued.

In this study exploring the implementation and use of body-worn cameras on mental health wards, we employed two methods for collecting and comparing data on incidents and use of containment measures, including BWCs, on one acute ward and one psychiatric intensive care unit. We found no clear relationship between the use of BWCs and rates or severity of incidents on either ward. While BWCs may be used when there are incidents of both physical and verbal aggression, results indicate that they may also provoke verbal aggression, as was suggested during some interviews within this study. This should be a concern, as strong evidence that being repeatedly subject to verbal aggression and abuse can lead to burnout and withdrawal of care by staff [ 30 ]. These mixed findings reflect results that were reported in two earlier studies of BWCs on mental health wards [ 12 , 13 ]. However, the very low use of the cameras, on just 10 per cent of the shifts where data was obtained, makes it even more difficult to draw any conclusions.

While the data shows limited impact of using BWCs on levels of incidents, we did find that during the pilot period BWC use tended to occur alongside physical restraint, but the direction of relationship is unclear as staff were asked to use BWCs when planning an intervention such as restraint. This relationship with restraint reflected the findings on several wards in a previous study [ 13 ], while contrasting with those reported in a second study that found reductions in incidents involving restraint during the evaluation period [ 12 ]. Such a mix of findings highlights the complexity of using BWCs as a violence reduction method within a busy healthcare setting in which several interacting components and contextual factors, and behaviours by staff and patients can affect outcomes [ 31 ]. The qualitative data collected during this pilot period highlighted the potential systemic and contextual factors such as low staffing that may have a confounding impact on the incident data presented in this simple form.

The findings presented within this evaluation provide some insights into the process of implementing BWCs as a safety intervention in mental health services and highlight some of the challenges and barriers faced. The use of implementation science to evaluate the piloting of BWCs on wards helps to demonstrate how multiple elements including a variety of contextual and systemic factors can have a considerable impact and thus change how a technology may vary not only between hospitals, but even across wards in the same hospital. By understanding the elements that may and do occur during the process of implementing such interventions, we can better understand if and how BWCs might be used in the future.

Within this pilot, extensive preparatory work conducted at a directorate and senior management level did not translate during the process of implementation at a ward level, which appeared to impact on the use of BWCs by individuals on the wards. This highlights that there is a need to utilise implementation science approaches in planning the implementation of new technologies or interventions and to investigate elements related to behavioural change and context rather than just the desired and actual effects of the intervention itself.

While ward staff and patients identified the potential for BWCs to enhance safety on the wards, participants distrusted their deployment and expressed concerns about ethical issues and possible harmful consequences of their use on therapeutic relationships, care provided and patient wellbeing. These themes reflect previous findings from a national interview study of patient and staff perspectives and experiences of BWCs in inpatient mental health wards [ 14 ]. Given these issues, alternatives such as increasing de-escalation skills were identified by staff as possible routes that may be more beneficial in these settings. Furthermore, other approaches such as safety huddles have also been highlighted within the literature as potential means to improve patient safety by looking ahead at what can be attended to or averted [ 32 ].

Furthermore, it is important to consider that the presence of power imbalances and the pre-existing culture on the ward have considerable implications for safety approaches and must be considered, as exemplified by the preferences by both staff and patients in this evaluation for more perceived ‘impartial’ interventions such as CCTV. As identified within previous studies [ 14 ], BWCs can have different implications for psychological safety, particularly for vulnerable patients who already feel criminalised in an environment with asymmetrical power imbalances between staff and patients. This is particularly salient when considering aspects of identity such as race, ethnicity, and gender both in terms of the identities of the patient group but also in terms of the staff/patient relationship.

While preferences in this study note CCTV as more ‘impartial’, work by Desai [ 33 ] draws on the literature about the use of surveillance cameras in other settings (such as public streets) as well as on psychiatric wards and concludes that CCTV monitoring is fraught with difficulties and challenges, and that ‘watching’ patients and staff through the lens of a camera can distort the reality of what is happening within a ward environment. In her recently published book, Desai [ 34 ] develops this theme to explore the impacts of being watched on both patients and staff through her ethnographic research in psychiatric intensive care units. She highlights concerns over the criminalisation of patient behaviour, safeguarding concerns in relation to the way women’s bodies and behaviours are viewed and judged, and the undermining by CCTV of ethical mental health practice by staff who attempt to engage in thoughtful, constructive, therapeutic interactions with patients in face-to-face encounters. Appenzeller et al.’s [ 35 ] review found that whilst the presence of CCTV appeared to increase subjective feelings of safety amongst patients and visitors, there was no objective evidence that video surveillance increases security, and that staff may develop an over-reliance on the technology.

In addition, our findings add to the existing literature which notes that alternative interventions and approaches that address underlying contextual and systemic issues related to improving care on inpatient wards require attention to address the underlying factors related to incidents, e.g., flashpoints [ 36 ]. Evidence suggests that factors leading to incidents can be predicted; therefore, there is a need to enable staff to work in a proactive way to anticipate and prevent incidents rather than view incidents as purely reactive [ 37 , 38 , 39 ]. Such skills-based and relational approaches are likely to impact more on improving safety and reducing incidents by addressing the complex and multi-faceted issue of incidents on inpatient mental health wards [ 40 ].

These findings highlight that interventions such as BWCs are not used within a vacuum, and that hospitals are complex contexts in which there are a range of unique populations, processes, and microsystems that are multi-faceted [ 41 ]. As a result, interventions will encounter both universal, specific, and local barriers that will impact on its functioning in the real world. This is salient because research suggests that camera use inside mental health wards is based on a perception of the violent nature of the mental health patient, a perception that not only influences practice but also impacts how patients experience the ward [ 33 ]. As a result, there needs to be careful consideration of the use of any new and innovative intervention aimed at improving safety within mental health settings that have limited research supporting their efficacy.

Limitations

While the study provides important insights into the efficacy and acceptability of introducing BWCs onto inpatient mental health wards, there were several limitations. Firstly, the analysis of incident data is limited in its nature as it only presents surface level information about incidents without wider contextual information. Results using such data should be cautiously interpreted as they do not account for confounding factors, such as staffing, acuity, ward culture or ward atmosphere, that are likely to contribute to incidents of violence and aggression. For example, while there was a statistically significant decrease in restrictive practice on the PICU across the study period, we know that BWCs were not widely used on that ward, so this is likely due to a confounding variable that was not accounted for in the study design.

Secondly, the study faced limitations in relation to recruitment, particularly with patients. Researchers’ access to wards was challenging due to high staff turnover and high rates of acuity, meaning many patients were not deemed well enough to be able to consent to take part in the study. In addition, the low use of the cameras on wards meant that many patients, and some staff, had not seen the BWCs in use. Similarly, patients had been provided limited information about the pilot, so their ability to engage in the research and describe their own experiences with BWCs was restricted.

Thirdly, analysis captures the active use of the BWC, however it does not fully capture the impact of staff wearing the cameras even where they do not actively use them. While our qualitative analysis provides insight into the limitation of such passive use, it is likely that the presence of the cameras being worn by staff, even when turned off, may have an impact on both staff and patient behaviours. This may explain trends in the data that did not reach significance but warrant further investigation in relation to the presence of BWCs, nonetheless.

Finally, researchers had planned to collect quantitative surveys from staff and patients in relation to their experiences of the ward atmosphere and climate, views related to therapeutic relationships on the ward, levels of burnout among staff, views on care, and attitudes to containment measures. Due to issues related to staff time, patient acuity, and poor engagement from staff leading to challenges accessing the wards, the collection of such survey data was unfeasible, and this element of the study was discontinued. As a result, we have not reported this aspect in our paper. This limitation reflects the busy nature of inpatient mental health wards with pressures on staff and high levels of ill health among patients. As such, traditional methodologies for evaluation are unlikely to elicit data that is comprehensive and meaningful. Alternative approaches may need to be considered.

Future directions

With BWCs being increasingly used across inpatient mental health services [ 14 ], it is important that further research and evaluation is conducted. To date, there is limited data regarding the effectiveness of this technology in relation to violence reduction; however, there may be other beneficial uses in relation to safeguarding and training [ 13 ]. Future research should consider alternative methods that ensure contextual factors are accounted for and that patient voices can be maximised. For example, focus groups with patients currently admitted to a mental health ward or interviews with those who have recently been on a ward that has used the cameras, would bypass problems encountered with capacity to consent in the present study. Furthermore, ethnographic approaches may provide a deeper understanding of the implementation, deployment and impact that BWCs have on wards.

Overall, this research sheds light on the complexities of using BWCs as a tool for ‘maximising safety’ in mental health settings. The findings suggest that BWCs have a limited impact on levels of incidents on wards, something that is likely to be largely influenced by the process of implementation as well as a range of contextual factors, including the staff and patient populations on the wards. As a result, it is likely that while BWCs may see successes in one hospital site this is not guaranteed for another site as such factors will have a considerable impact on efficacy, acceptability, and feasibility. Furthermore, the findings point towards the need for more consideration to be placed on processes of implementation and the complex ethical discussions regarding BWC use from both a patient and a staff perspective.

In conclusion, while there have been advances in digital applications and immersive technologies showing promise of therapeutic benefits for patients and staff more widely, whether BWCs and other surveillance approaches are to be part of that picture remains to be seen and needs to be informed by high-quality, co-produced research that focuses on wider therapeutic aspects of mental healthcare.

Data availability

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Department of Health and Social Care. The NHS Long Term Plan. 2019. https://www.longtermplan.nhs.uk/about/ . [Accessed on 01/11/2021].

NHS England. Violence prevention and safety. 2023. https://www.england.nhs.uk/supporting-our-nhs-people/health-and-wellbeing-programmes/violence-prevention-and-safety/#:~:text=The%20impact%20on%20staff%20is,thinking%20about%20leaving%20the%20organisation . [Accessed on 06/07/23].

Royal College of Nursing. Violence and aggression in the NHS: estimating the size and the impact of the problem. London: Royal College of Nursing; 2018.

Google Scholar  

Iozzino L, Ferrari C, Large M, Nielssen O, de Girolamo G. Prevalence and risk factors of violence by psychiatric acute inpatients: a systematic review and meta-analysis. PLoS ONE. 2015;10(6):e0128536. https://doi.org/10.1371/journal.pone.0128536 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cranage K, Foster K. Mental health nurses’ experience of challenging workplace situations: a qualitative descriptive study. Int J Ment Health Nurs. 2020;31(3):665–76. https://doi.org/10.1111/inm.12986 .

Article   Google Scholar  

Jenkin G, Quigg S, Paap H, Cooney E, Peterson D, Every-Palmer S. Places of safety? Fear and violence in acute mental health facilities: a large qualitative study of staff and service user perspectives. PLoS ONE. 2022;17(5):e0266935. https://doi.org/10.1371/journal.pone.0266935 .

Gaub JE, Choate DE, Todak N, Katz CM, White MD. Officer perceptions of body-worn cameras before and after deployment: a study of three departments. Police Q. 2016;19(3):275–302.

Cubitt TI, Lesic R, Myers GL, Corry R. Body-worn video: a systematic review of literature. Australian New Z J Criminol. 2017;50(3):379–96. https://doi.org/10.1177/0004865816638909 .

Lum C, Koper CS, Wilson DB, Stoltz M, Goodier M, Eggins E et al. Body-worn cameras’ effects on police officers and citizen behavior: A systemitic review. Campbell Syst Reviews. 2020;16(3), e1112.

Ariel B, Newton M, McEwan L, Ashbridge GA, Weinborn C, Brants HS. Reducing assaults against Staff using body-worn cameras (BWCs) in Railway stations. Criminal Justice Rev. 2019;44(1):76–93.

Wilson K, Eaton J, Foye U, Ellis M, Thomas E, Simpson A. What evidence supports the use of body worn Cameras in mental health inpatient wards? A systematic review and narrative synthesis of the effects of body worn Cameras in public sector services. Int J Ment Health Nurs. 2021. https://doi.org/10.1111/inm.12954 .

Article   PubMed   PubMed Central   Google Scholar  

Ellis T, Shurmer DL, Badham-May S, Ellis-Nee C. The use of body worn video cameras on mental health wards: results and implications from a pilot study. Mental Health Family Med. 2019;15:859–68.

Hardy S, Bennett L, Rosen P, Carroll S, White P, Palmer-Hill S. The feasibility of using body-worn cameras in an inpatient mental health setting. Mental Health Family Med. 2017;13:393–400.

Wilson K, Foye U, Thomas E, Chadwick M, Dodhia S, Allen J, Lynn J, Brennan G, Simpson A. Exploring the use of body-worn cameras in acute mental health wards: a qualitative interview study with mental health patients and staff. Int J Nurs Stud. 2023;140:104456. https://doi.org/10.1016/j.ijnurstu.2023.104456 .

Article   PubMed   Google Scholar  

Baker J, Pryjmachuk S. Will safe staffing in Mental Health nursing become a reality? J Psychiatry Mental Health Nurs. 2016;23(2):75–6. https://doi.org/10.1111/jpm.12282 .

Article   CAS   Google Scholar  

Piano VL, Creswell JW. Understanding Research: A Consumer Guide. 2nd edition. 2015. Boston: Pearson Education, INC.

Bowers L, Simpson A, Alexander J. Patient-staff conflict: results of a survey on acute psychiatric wards. Soc Psychiatry Psychiatr Epidemiol. 2013;38(7):402–8.

Bowers L. The City-128 study of observation and outcomes. BMC Psychiatry. 2007;7(Suppl 1). https://doi.org/10.1186/1471-244X-7-S1-S122 . S122.

Bowers L, Flood C, Brennan G, Allan T. A replication study of the City nurse intervention: reducing conflict and containment on three acute psychiatric wards. J Psychiatr Ment Health Nurs. 2008;15:737–42. https://doi.org/10.1111/j.1365-2850.2008.01294.x .

Article   CAS   PubMed   Google Scholar  

Bowers L, James K, Quirk A, Simpson A, Stewart D, Hodsoll J, SUGAR. Reducing conflict and containment rates on acute psychiatric wards: the safewards cluster randomised controlled trial. Int J Nurs Stud. 2015;52(9):1412–22. https://doi.org/10.1016/j.ijnurstu.2015.05.001 . Erratum in: Int J Nurs Stud. 2016;58:102. PMID: 26166187; PMCID: PMC4518134.

Bowers L, Douzenis A, Galeazzi G et al. Disruptive and dangerous behaviour by patients on acute psychiatric wards in three European centres. Social Psychiatry and Psychiatric Epidemiology. 2005;40, 822–828. https://link.springer.com/article/10.1007/s00127-005-0967-1 .

Bowers L, Flood C, Brennan G, LiPang M, Oladapo P. Preliminary outcomes of a trial to reduce conflict and containment on acute psychiatric wards: City nurses. J Psychiatr Ment Health Nurs. 2006;13:165–72. https://doi.org/10.1111/j.1365-2850.2006.00931.x .

Datix. (2019). Products. Retrieved from https://www.rldatix.com/en-uk/products .

Fazel S, Toynbee M, Ryland H, et al. Modifiable risk factors for inpatient violence in psychiatric hospital: prospective study and prediction model. Psychol Med. 2023;53(2):590–6. https://doi.org/10.1017/S0033291721002063 .

Mushcab H, Bunting D, Yami S, Abandi A, Hunt C. An evaluation of Datix implementation for incident reporting at Johns Hopkins Aramco Healthcare. J Patient Saf Risk Manage. 2020;25(2):67–74. https://doi.org/10.1177/2516043520905481 .

SPSS. IBM SPSS statistics for Windows. Armonk, NY: IBM Corp; 2023.

Microsoft Corporation. (2018). Microsoft Excel . Retrieved from https://office.microsoft.com/excel .

Ritchie J, Lewis J. Qualitative. Research Practice. London: Sage; 2003.

Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. https://doi.org/10.1186/1748-5908-4-50 .

Porath CL, Pearson CM. Emotional and behavioral responses to workplace incivility and the impact of hierarchical status. J Appl Soc Psychol. 2009;42(Suppl 1):E326–57. https://doi.org/10.1111/j.1559-1816.2012.01020.x .

Craig P, Dieppe P, Macintyre S, et al. Developing and evaluating complex interventions: the new medical Research Council guidance. BMJ. 2008;337. https://doi.org/10.1136/bmj.a1655 .

Taylor-Watt J, Cruickshank A, Innes J, Brome B, Shah A. Reducing physical violence and developing a safety culture across wards in East London. Br J Mental Health Nurs. 2017. https://doi.org/10.12968/bjmh.2017.6.1.35 .

Desai S. The new stars of CCTV: what is the purpose of monitoring patients in communal areas of psychiatric hospital wards, bedrooms and seclusion rooms? Divers Equality Health Care. 2009;6(1):12. [Google Scholar].

Desai S. Surveillance Practices and Mental Health: the impact of CCTV inside mental health wards. Routledge; 2022.

Appenzeller YE, Appelbaum PS, Trachsel M. Ethical and practical issues in Video Surveillance of Psychiatric Units. Psychiatric Serv. 2020;71(5):480–6. https://doi.org/10.1176/appi.ps.201900397 .

Bowers L. Safewards: a new model of conflict and containment on psychiatric wards. J Psychiatr Ment Health Nurs. 2014;21(6):499–508. https://doi.org/10.1111/jpm.12129 . PMID: 24548312; PMCID: PMC4237187.

Goodman H, Papastavrou Brooks C, Price O, Barley EA. Barriers and facilitators to the effective de-escalation of conflict behaviours in forensic high-secure settings: a qualitative study. Int J Mental Health Syst. 2020;14:59. https://doi.org/10.1186/s13033-020-00392-5 .

Johnson ME, Delaney KR. Keeping the unit safe: the anatomy of escalation. J Am Psychiatr Nurses Assoc. 2007;13:42–52.

Johnson ME, Hauser PM. The practice of expert nurses: accompanying the patient to a calmer space. Issues Ment Health Nurs. 2001;22:651–68.

Hamrin V, Iennaco J, Olsen D. A review of ecological factors affecting inpatient psychiatric unit violence: implications for relational and unit cultural improvements. Issues Ment Health Nurs. 2009;30(4):214 – 26. doi: 10.1080/01612840802701083. PMID: 19363726.

Squires JE, Estabrooks CA, Scott SD, Cummings GG, Hayduk L. The influence of organizational context on the use of research by nurses in Canadian pediatric hospitals. BMC Health Serv Res. 2013;13. https://doi.org/10.1186/1472-6963-13-351 .

Download references

Acknowledgements

We would like to thank The Burdett Trust for Nursing for funding this work. We would also like to acknowledge our wider Lived Experience Advisory Panel and Project Advisory Panel for their contributions and support and would like to thank the staff and service users on the wards we attended for their warmth and participation.

Funding was provided by The Burdett Trust of Nursing. Funders were independent of the research and did not impact findings.

Author information

Authors and affiliations.

Mental Health Nursing, Health Service and Population Research Department, Institute of Psychiatry, Psychology & Neuroscience, King’s College London, Denmark Hill, London, SE5 8AF, UK

Una Foye, Keiran Wilson, Jessica Jepps, James Blease, Geoff Brennan & Alan Simpson

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, Mental Health Nursing, King’s College London, London, UK

Una Foye, Keiran Wilson, Jessica Jepps, Geoff Brennan & Alan Simpson

Lived Experience Advisor, London, UK

Ellen Thomas, Leroy McAnuff, Sharon McKenzie, Katherine Barrett & Lilli Underwood

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have read and approved the manuscript. Authors AS, UF, KW, GB created the protocol for the study. KW, JJ, UF conducted the recruitment for the study, and conducted the interviews. UF, JJ, JB, LMA, LU, SMK, KB, ET coded data, and contributed to the analysis. All authors supported drafting and development of the manuscript.

Corresponding author

Correspondence to Una Foye .

Ethics declarations

Ethics approval and consent to participate.

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical. Ethical approval was granted by the Health Research Authority: London - Camden & Kings Cross Research Ethics Committee (IRAS PROJECT ID 322268, REC Reference 23/LO/0337). All participants provided informed consent prior to enrolment in the study, including consent for publication of anonymised quotes.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, supplementary material 6, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Foye, U., Wilson, K., Jepps, J. et al. Exploring the use of body worn cameras in acute mental health wards: a mixed-method evaluation of a pilot intervention. BMC Health Serv Res 24 , 681 (2024). https://doi.org/10.1186/s12913-024-11085-x

Download citation

Received : 03 January 2024

Accepted : 07 May 2024

Published : 29 May 2024

DOI : https://doi.org/10.1186/s12913-024-11085-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation
• Body worn cameras
• Qualitative
• Mental health

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Transforming the understanding and treatment of mental illnesses.

Información en español

Celebrating 75 Years! Learn More >>

• Health Topics
• Brochures and Fact Sheets
• Help for Mental Illnesses
• Clinical Trials

COVID-19 and Mental Health

What is covid-19.

COVID-19 is a disease caused by a virus named SARS-CoV-2. COVID-19 most often affects the lungs and respiratory system, but it can also affect other parts of the body. Some people develop post-COVID conditions, also called  Long COVID  . These symptoms can include neurological symptoms such as difficulty thinking or concentrating, sleep problems, and depression or anxiety.

Why is NIMH studying COVID-19 and mental health?

Both SARS-CoV-2 and the COVID-19 pandemic have significantly affected the mental health of adults and children. Many people experienced symptoms of  anxiety ,  depression , and substance use disorder during the pandemic. Data also suggest that people are more likely to develop mental illnesses or disorders in the months following COVID-19 infection. People with Long COVID may experience many symptoms related to brain function and mental health  .

While the COVID-19 pandemic has had widespread mental health impacts, some people are more likely to be affected than others. This includes people from racial and ethnic minority groups, mothers and pregnant people, people with financial and housing insecurity, children, people with disabilities, people with preexisting mental illnesses or substance use problems, and health care workers. 

How is NIMH research addressing this critical topic?

NIMH is supporting research to understand and address the impacts of the pandemic on mental health. This includes research to understand how COVID-19 affects people with existing mental illnesses across their entire lifespan. NIMH also supports research to help meet people’s mental health needs during the pandemic and beyond. This includes research focused on making mental health services more accessible through telehealth, digital tools, and community-based interventions.

NIMH is also working to understand the unique impacts of the pandemic on specific groups of people, including people in underserved communities and children. For example, NIMH supports research investigating how pandemic-related factors, such as school disruptions, may influence children’s brain, cognitive, social, and emotional development.

Where can I learn more about COVID-19 and mental health?

• NIMH video: Mental Illnesses and COVID-19 Risks
• NIMH Director’s Messages about COVID-19
• NIMH events about COVID-19
• NIMH news about COVID-19

Where can I learn more about Long COVID and COVID-19?

• NIH page on Long COVID 
• NIH RECOVER Initiative  
• CDC COVID-19 resources 

How can I find help for mental health concerns?

If you have concerns about your mental health, talk to a primary care provider. They can refer you to a qualified mental health professional, such as a psychologist, psychiatrist, or clinical social worker, who can help you figure out the next steps. Find tips for talking with a health care provider about your mental health.

You can learn more about getting help on the NIMH website. You can also learn about finding support  and locating mental health services  in your area on the Substance Abuse and Mental Health Services Administration (SAMHSA) website.

Last Reviewed:  May 2024

Unless otherwise specified, the information on our website and in our publications is in the public domain and may be reused or copied without permission. However, you may not reuse or copy images. Please cite the National Institute of Mental Health as the source. Read our copyright policy to learn more about our guidelines for reusing NIMH content.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Elsevier - PMC COVID-19 Collection

Comparing efficacy of telehealth to in-person mental health care in intensive-treatment-seeking adults

The heightened acuity in anxiety and depressive symptoms catalyzed by the COVID-19 pandemic presents an urgent need for effective, feasible alternatives to in-person mental health treatment. While tele-mental healthcare has been investigated for practicability and accessibility, its efficacy as a successful mode for delivering high-quality, high-intensity treatment remains unclear. This study compares the clinical outcomes of a matched sample of patients in a private, nation-wide behavioral health treatment system who received in-person, intensive psychological treatment prior to the COVID-19 pandemic (N = 1,192) to the outcomes of a distinctive group of patients who received telehealth treatment during the pandemic (N = 1,192). Outcomes are measured with respect to depressive symptoms (Quick Inventory of Depressive Symptomatology-Self-Report; QIDS-SR) and quality of life (Quality of Life Enjoyment and Satisfaction Questionnaire; Q-LES-Q). There were no significant differences in admission score on either assessment comparing in-person and telehealth groups. Patients in the partial hospitalization level of care stayed longer when treatment was remote. Results suggest telehealth as a viable care alternative with no significant differences between in-person and telehealth groups in depressive symptom reduction, and significant increases in self-reported quality of life across both groups. Future research is needed to replicate these findings in other healthcare organizations in other geographical locations and diverse patient populations.

1. Introduction

In response to the coronavirus disease 2019 (COVID-19) pandemic, healthcare organizations around the world adapted to remote behavioral health treatment to meet an increased need for high-quality mental health care. With lockdowns and stay-at-home orders, the pandemic presented an acute need for mental health treatment to be virtually accessible—regardless of patient or provider setting—as well as near-equivalent in quality to what a person would have received had they attended in person. This paper examines the current landscape of remote mental health care as a viable treatment delivery method and uses clinical outcomes data from patients undergoing in-person and telehealth mental health treatment at a large, multistate behavioral health system to illustrate telehealth's comparability to traditional, in-person care.

The US Centers for Disease Control and Prevention (CDC) reported that symptoms of anxiety and depressive disorders among adults “increased considerably” when comparing reports from April through June of 2019 to the same time period in 2020 ( Czeisler et al., 2020 ). In the survey with 5,412 adult respondents, the CDC measured anxiety and depressive symptoms related to the pandemic using the four-item patient health questionnaire (PHQ-4) and COVID-19 trauma-related symptoms using the six-item Impact of Event Scale. Overall, 40.9% of survey respondents reported at least one adverse mental health condition, including symptoms of anxiety disorder or depressive disorder (30.9%), and 26.3% reported symptoms of a trauma- and stressor-related disorder because of the pandemic. Suicidal ideation in the last 30 days was reported by 10.7% of respondents. In 2021, the CDC published results of the Household Pulse Survey, a self-report online survey, which showed that the percentage of adults who reported experiencing symptoms related to depression and anxiety disorders increased from 36.4% to 41.5% between August 2020 and February 2021 ( Vahratian et al., 2021 ). Additionally, respondents who reported unmet mental health needs increased from 9.2% to 11.7% in the same time period. Historically there is evidence that mental health needs increase during a pandemic ( Soklaridis et al., 2020 ). Past outbreaks, such as severe acute respiratory syndrome (SARS) in 2003, directly resulted in heightened anxiety related to the virus ( Cheng and Cheung 2005 ; Chong et al., 2004 ), as well as increased stigmatization and discrimination of minority groups (Ren et al., 2020). Feelings of uncertainty, isolation, and confusion over rapidly changing health guidelines have exacerbated feelings of anxiety and mental distress ( Rajkumar, 2020 ). What sets the COVID-19 pandemic apart is the already widespread use of telecommunication platforms for remote work, learning, and basic healthcare. Although prior studies have investigated the impact of telehealth compared to traditional in-person treatment, there has been limited research into these differences in treatment settings beyond the outpatient level ( Hilty et al., 2013 ). This is a novel offering of the current analysis.

Mental healthcare providers have been exploring remote treatment options for over six decades ( Hilty et al., 2013 ). In this domain, providers use videoconferencing for therapy, evaluations, and medication management. In some cases, clinicians use telehealth to supplement in-person treatment. Previous research has shown that telehealth reduces patient-incurred costs and time associated with travel ( Rabinowitz et al., 2010 ) and improves accessibility of mental healthcare for those living in rural areas ( Manfredi et al., 2005 ; Morland et al., 2010 ; Weiner et al., 2011 ). Some even argue that remote treatment is superior to in-person consultations for some patients ( Pakyurek et al., 2010 ; Storch et al., 2011 ). In the context of public health, others have highlighted telehealth as a way for patients to maintain human connection during social distancing and lockdowns ( Whaibeh et al., 2020 ). Findings are mixed with respect to patient satisfaction of psychiatric telehealth services ( Frueh et al., 2005 ; Jacob et al., 2012 ; Nelson et al., 2003 ; Ruskin et al., 2004 ; Zimmerman et al., 2021a ), yet remote care has been shown to improve access to mental health treatment in some cases, especially for individuals experiencing linguistic or financial obstacles ( Chong and Moreno, 2012 ; Moreno et al., 2012 ; Mucic 2010 ; Ye et al., 2012 ). For example, Mucic (2010) highlights that the availability of bilingual clinicians through remote care can help overcome travel requirements for both patients and providers, as well as lead to higher patient satisfaction. In the case of the pandemic, telehealth treatment may also help minority groups navigate greater exposure risk to COVID-19 by allowing them to remain remote and socially distanced distanced during care ( Price-Haywood et al., 2020 ). Prior to the pandemic, telehealth was largely reported as comparable to in-person treatment in terms of its feasibility in obtaining favorable treatment outcomes in a variety of symptom profiles (De Las Cuevas et al., 2006 ; Lopez et al., 2004 ; Nelson et al., 2003 ; Richardson et al., 2009 ; Rojas et al., 2020 ; Ruskin et al., 2004 ; Shore et al., 2008 ; Urness et al., 2006 ; Yellowlees et al., 2010 ). Zimmerman and colleagues describe how patients in a partial hospitalization program reported significant reduction in symptoms whether they received in-person or telehealth treatment ( Zimmerman et al., 2021b ). However, to date, there are few other complete comparisons of patient outcomes prior to and following the pandemic, specifically as they relate to the efficacy of remote versus in-person treatment (although see Hom et al., 2020 for preliminary data). Further, the efficacy of telehealth as a treatment alternative has not previously been examined in a comparison of partial hospitalization and intensive outpatient programming, namely how level of care and acuity might affect outcomes.

The present study analyzes the clinical outcomes of patients who underwent mental health treatment in this telehealth format during the COVID-19 pandemic and compares their outcomes to patients from the same healthcare system who completed in-person care prior to the pandemic. The research questions were twofold: (1) How did treatment delivery (in-person vs. telehealth) affect patient outcomes across Rogers as a nation-wide provider; and (2) does treatment delivery modulate therapeutic response for patients in different levels of care (PHP or IOP)? Considering some service lines utilize therapies traditionally conducted in-person—for example, exposure therapies or behavioral activation exercises—it was hypothesized that any between-group differences may be due to the difficulty associated with implementing these methods virtually in such a short timeframe. If this is the case, patients who received in-person treatment are expected to show greater symptom reduction than patients who received telehealth treatment. However, consistent with the Zimmerman et al. (2021b) study, if the therapies employed translate as planned to the virtual format, we do not expect significant differences in scores at discharge comparing in-person and telehealth groups.

2.1. Participants and procedure

The Rogers Behavioral Health (Rogers) Institutional Review Board is committed to the ethical principles for the protection of human subjects in research which guide the IRB's deliberations and decision-making, including respect for persons, beneficence, and justice. This study was approved with exempt determination by the Rogers Institutional Review Board (IRB). Because this study contained retrospective data analysis only and no prospective data collection, there was no consent process. Only de-identified data was used for analyses. All adult patients between 18 and 65 were eligible for inclusion. If patients were missing either an admission or discharge score for either of the two measures, they were excluded from analyses.

In March of 2020, following the onset of the COVID-19 pandemic, Rogers converted partial hospitalization (PHP) and intensive outpatient (IOP) treatment programs to telehealth. Prior to the pandemic, telehealth services were occasionally provided on an as-needed basis. The organization-wide transition to telehealth leveraged existing technological solutions for videoconferencing and scheduling sessions. If patients did not have a device to access their therapy sessions, laptops were provided. To ensure high-quality treatment, fidelity checks of treatment delivery were conducted by clinical supervisors and directors of clinical services. As pre-existing in-person treatment was protocolized and manualized, an existing audit process was leveraged to ensure compliance with treatment groups and to identify gaps in telehealth treatment delivery as compared to in-person care. Specific aspects of treatment varied by services line (e.g., OCD PHP versus Depression Recovery PHP), however there was overlap in main treatment functions. Specifically, individual meeting invites were sent to each patient for every treatment element daily. Individual sessions included those with their assigned therapist and with their psychiatrists. Group sessions may have included cognitive behavioral therapy skills or possibly dialectal behavior therapy skills depending on the program the patient was enrolled in. Couples or family sessions were also delivered via telehealth. All treatment manuals and materials were sent electronically for patients to use during telehealth programming. Completed forms were emailed back to treatment team members to review in future sessions. Patients completed self-reported outcome measures in a protocolized fashion, at home, and all scores received from these assessments were integrated into the patient's treatment. Close attention was paid to patient attendance for each treatment element and local police phone numbers and emergency contact information was collected and made accessible for all treatment team members in case someone did not appear for treatment in a timely manner. In addition to emergency contact information, safety protocols were developed to ensure response to safety concerns was timely and consistent.

The research study included in this submission aligns with the IRB-approved protocol. Participants include 2,384 adult patients who completed either PHP or IOP programs at Rogers, 1,192 patients in each of the in-person and telehealth groups. Patients in the telehealth group received treatment between July 2020 and March 2021. The group of patients who received telehealth was matched to an equally sized subsample of 1,192 patients who received in-person care between May 2013 and December 2019. Groups were matched using optimal pairwise matching computed with the MatchIt package in R ( Ho et al., 2011 ), where covariates unrelated to treatment were balanced between in-person and telehealth groups. Covariates that were balanced in this way include age, sex, race, diagnosis, and level of care, ensuring that comparable demographic groups could be compared in addition to having an equal proportion of PHP and IOP patients included in each of the groups. Patients whose sex and/or race were unknown were excluded from the matching process and were not included in the analyzed dataset ( Table 1 ). There were no significant differences with respect to diagnosis across the dataset either comparing in-person to telehealth groups or with level of care ( Table 2 ).

Comparison of age, sex assigned at birth, and race of in-person and telehealth groups.

Note. Age is denoted as mean ± standard deviation. Demographic percentages are denoted with counts in parentheses. T-tests were used to test for significant differences when age was the dependent variable. Chi-square tests were used to detect differences in sex and race.

Comparison of diagnosis category across in-person and telehealth groups.

Note. Data are shown as proportions with counts in parentheses. Chi-square test results and p-values are reported.

Regardless of treatment delivery mode, prior to admitting to treatment, patients' psychological symptoms are assessed over the phone. Licensed psychiatrists and psychologists with expertise in the patient's behavioral health area of concern review the phone screen interviews to determine treatment program appropriateness and to recommend level of care. PHP and IOP are multidisciplinary programs that involve individual, group, and family therapies along with medication management. Patients in PHP attend treatment 6 h per day, five days a week, and patients in IOP complete 3 h of treatment five days a week. After they are admitted, patients work with a psychologist to complete a diagnostic assessment where diagnoses are confirmed using the Diagnostic and Statistical Manual-5th Edition (DSM-5; American Psychological Association, 2013 ). Patient-reported outcome measures (PROMs) are collected from each patient at the beginning of treatment, either weekly or biweekly during treatment, and at the end of treatment. Patients received the same treatment content and frequency of treatment sessions regardless of when they completed treatment. In-person and telehealth providers participated in the same internal training program, ensuring a high level of similarity in treatment received by in-person and telehealth patient groups. Due to staff turnover, some providers were consistent across in-person and telehealth timeframes, whereas others were unique to one timeframe.

Two PROMs were used to index treatment efficacy: the Quick Inventory of Depressive Symptomology-Self Report (QIDS-SR; Rush et al., 2003 ) and the Quality of Life Enjoyment and Satisfaction Questionnaire - Short Form (Q-LES-Q; Schechter et al., 2007 ). Both assessments are administered upon admission, discharge, and either weekly or biweekly across all adult programs, service lines, and levels of treatment at Rogers. The Quick Inventory of Depressive Symptomatology Self-Report (QIDS-SR; Rush et al., 2003 ) is a 16-item self-report measure that asks respondents to indicate on a scale of 0–3 (0 = low, 3 = high) how well each item describes them for the past week. The QIDS-SR assesses nine domains of depression symptomatology: sleep, mood, weight, concentration, guilt, suicidal ideation, interest, fatigue, and psychomotor changes. Responses are summed to provide a single outcome metric, which can range from 0 to 27, where a higher score indicates a more severe level of depression symptomatology. The Quality-of-Life Enjoyment and Satisfaction Questionnaire (Q-LES-Q; Schechter et al., 2007 ) is a 16-item self-report measure that asks respondents to indicate on a scale of 1–5 (“very poor” = 1, “very good” = 5) how satisfied they have been in the past week with aspects of their general health, well-being, and feelings about their life. Total scores are reported as percentages ranging from 0 to 100%, with higher percentages indicating greater enjoyment and satisfaction.

Statistical analyses were performed using R ( R Core Team, 2021 ). To measure differences between in-person and telehealth groups, we conducted 2 × 2 between-subjects analyses of variance (ANOVAs) with treatment modality (in-person, telehealth) and level of care (PHP, IOP) as independent variables. One ANOVA was conducted for each of total QIDS-SR score at admission, score at discharge, and percent change over time between admission and discharge. The same three tests were conducted for Q-LES-Q scores. Planned pairwise comparisons within each significant factor were conducted using t-tests (e.g. PHP compared to IOP). For all analyses, a two-tailed level of significance of 5% was adopted.

There were no significant differences in admission score on either the QIDS-SR ( F (2380) = 1.06, p  = .30) or the Q-LES-Q ( t (2380) = 0.26, p  = .61) between in-person and telehealth groups ( Table 3 ). For QIDS-SR scores, there was a significant main effect of level of care only at both admission ( F (1, 2380) = 88.00, p  < .001) and at discharge ( F (1, 2380) = 48.07, p  < .001). Planned pairwise comparisons showed that PHP scores at both time points, on average, were higher than IOP scores, which is clinically expected. The main effect of treatment delivery was trending but not significant at discharge only ( F (1, 2380) = 3.60, p  = .06) such that patients who received telehealth treatment had a higher average QIDS discharge score compared to in-person. The interaction between treatment delivery and level of care was not significant.

Comparison of clinical assessment outcomes at admission and discharge for in-person and telehealth groups. Data are shown as mean ± standard deviation with analysis of variance results for the group factor.

Note: Significance threshold used is 0.05. F-statistics and partial eta-squared (η2) reflect the ANOVA results at the group level. T-tests reflect significant pairwise comparisons. Cohen's D effect size reflects the change between admission and discharge score and is presented alongside discharge data only.

The same pattern of results seen for QIDS scores was also seen for Q-LES-Q scores, namely only a significant main effect of level of care at both admission ( F (2380) = 78.00, p  < .001) and at discharge ( F (2380) = 39.14, p  < .001). There was no main effect of treatment delivery or interaction between treatment delivery and level of care. Planned pairwise comparisons showed that Q-LES-Q admission and discharge scores were higher for patients in IOP relative to PHP, which is an expected result given the difference in degree of symptom severity at these different levels of care.

Pairwise Cohen's d effect sizes were calculated, as well as descriptive statistics for the change in QIDS-SR and Q-LES-Q scores from admission to discharge for all. Pairwise comparisons show the degree of change between admission and discharge scores comparing in-person and telehealth groups was not significant at either IOP or PHP levels of care for either QIDS-SR or Q-LES-Q. Effect sizes, overall, were moderate to high at all levels of care for both assessments. Length of stay was not significantly different comparing in-person and telehealth IOP groups ( t (481) = −1.66, p  = .10), but it was significantly different comparing PHP groups, such that patients receiving telehealth stayed 2.8 days longer in treatment than patients seen in person ( t (1877) = −6.62, p  < .001).

4. Discussion

To demonstrate the efficacy of telehealth treatment relative to in-person treatment, we compared clinical outcomes scores at admission and discharge for adult patients (1,192 in-person and 1,192 telehealth) across PHP and IOP programs. Patients who received PHP telehealth stayed an average of 2.8 days longer in treatment than the PHP in-person group; there was no difference in length of stay in IOP programs. This significant finding of increased length of stay in the PHP telehealth group is consistent with prior work ( Zimmerman et al., 2021b ). There were no significant differences in QIDS or Q-LES-Q discharge scores for either IOP or PHP groups, and there were no significant differences in the degree of change between admission and discharge scores for any group. Effect sizes were moderate to large across treatment modalities and levels of care. These data support remote treatment as a viable alternative to in-person mental health services, specifically as both in-person and remote patients experienced symptom reduction, and both populations reported improvements in quality of life.

There are recognized limitations of this study. Patients were not randomized into in-person and telehealth groups; rather, the external variable of the COVID-19 pandemic forced group membership in our sample. However, there were no between-group differences in QIDS-SR and Q-LES-Q admission scores at either the holistic level or by level of care, which suggests that no one group was more acute than the other at onset of treatment as indexed by these assessments. Further, matching in-person and telehealth groups for pre-treatment covariates—age, sex, race, and level of care—allowed us to examine the dependent variable of interest more closely, namely treatment outcomes, independent of these other factors. Additionally, findings are limited by the lack of a structured clinician-administered diagnostic assessment at admission which would strengthen the diagnoses established. However, licensed psychiatrists with years of experience diagnosing mental health disorders conducted the initial diagnostic assessments, and patient charts would be updated to reflect any changes to diagnoses made later in treatment.

Results have implications for both patients considering remote treatment options as well as organizations looking to implement alternatives to in-person care. Despite the difference in treatment delivery, our large sample size from a system-wide perspective supports telehealth as being as effective as in-person care for intensive-treatment-seeking individuals. For patients living far away from treatment centers or for patients who cannot leave home to attend in-person treatment, these results demonstrate telehealth treatment as a viable alternative to in-person treatment.

Author statement

Nyssa Z. Bulkes, Ph.D.: Methodology, Formal Analysis, Writing – Original Draft.

Kaley Davis, M.S.: Writing – Reviewing & Editing, Project Administration.

Brian Kay, Ph.D.: Resources, Writing – Reviewing & Editing, Conceptualization.

Bradley C. Riemann, Ph.D.: Resources, Writing – Reviewing & Editing, Conceptualization.

Declaration of competing interest

• American Psychiatric Association . American Psychiatric Pub; 2013. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®) [ Google Scholar ]
• Cheng C., Cheung M.W. Psychological responses to outbreak of severe acute respiratory syndrome: a prospective, multiple time‐point study. J. Pers. 2005; 73 (1):261–285. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Chong J., Moreno F. Feasibility and acceptability of clinic-based telepsychiatry for low-income Hispanic primary care patients. Telemed. e Health. 2012; 18 (4):297–304. [ PubMed ] [ Google Scholar ]
• Chong M.Y., Wang W.C., Hsieh W.C., Lee C.Y., Chiu N.M., Yeh W.C., et al. Psychological impact of severe acute respiratory syndrome on health workers in a tertiary hospital. Br. J. Psychiatr. 2004; 185 (2):127–133. [ PubMed ] [ Google Scholar ]
• Cuevas C.D.L., Arredondo M.T., Cabrera M.F., Sulzenbacher H., Meise U. Randomized clinical trial of telepsychiatry through videoconference versus face-to-face conventional psychiatric treatment. Telemed. e Health. 2006; 12 (3):341–350. [ PubMed ] [ Google Scholar ]
• Czeisler M.É., Lane R.I., Petrosky E., Wiley J.F., Christensen A., Njai R., et al. Mental health, substance use, and suicidal ideation during the COVID-19 pandemic—United States, June 24–30, 2020. MMWR (Morb. Mortal. Wkly. Rep.) 2020; 69 (32):1049. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Frueh B.C., Henderson S., Myrick H. Telehealth service delivery for persons with alcoholism. J. Telemed. Telecare. 2005; 11 (7):372–375. [ PubMed ] [ Google Scholar ]
• Hilty D.M., Ferrer D.C., Parish M.B., Johnston B., Callahan E.J., Yellowlees P.M. The effectiveness of telemental health: a 2013 review. Telemed. e Health. 2013; 19 (6):444–454. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Ho D.E., Kosuke I., King G., Stuart E.A. MatchIt: nonparametric preprocessing for parametric causal inference. J. Stat. Software. 2011; 42 (8) [ Google Scholar ]
• Hom M.A., Weiss R.B., Millman Z.B., Christensen K., Lewis E.J., Cho S., et al. Development of a virtual partial hospital program for an acute psychiatric population: lessons learned and future directions for telepsychotherapy. J. Psychother. Integrat. 2020; 30 (2):366. [ Google Scholar ]
• Jacob M.K., Larson J.C., Craighead W.E. Establishing a telepsychiatry consultation practice in rural Georgia for primary care physicians: a feasibility report. Clin. Pediatr. 2012; 51 (11):1041–1047. [ PubMed ] [ Google Scholar ]
• Lopez A.M., Cruz M., Lazarus S., Webster P., Jones E.G., Weinstein R.S. Case report: use of American sign language in telepsychiatry consultation. Telemed. e Health. 2004; 10 (3):389–391. [ PubMed ] [ Google Scholar ]
• Manfredi L., Shupe J., Batki S.L. Rural jail telepsychiatry: a pilot feasibility study. Telemed. e Health. 2005; 11 (5):574–577. [ PubMed ] [ Google Scholar ]
• Moreno F.A., Chong J., Dumbauld J., Humke M., Byreddy S. Use of standard Webcam and Internet equipment for telepsychiatry treatment of depression among underserved Hispanics. Psychiatr. Serv. 2012; 63 (12):1213–1217. [ PubMed ] [ Google Scholar ]
• Morland L.A., Greene C.J., Rosen C.S., Foy D., Reilly P., Shore J., et al. Telemedicine for anger management therapy in a rural population of combat veterans with posttraumatic stress disorder: a randomized noninferiority trial. J. Clin. Psychiatr. 2010; 71 (7):855–863. [ PubMed ] [ Google Scholar ]
• Mucic D. Transcultural telepsychiatry and its impact on patient satisfaction. J. Telemed. Telecare. 2010; 16 (5):237–242. [ PubMed ] [ Google Scholar ]
• Nelson E.L., Barnard M., Cain S. Treating childhood depression over videoconferencing. Telemed. e Health. 2003; 9 (1):49–55. [ PubMed ] [ Google Scholar ]
• Pakyurek M., Yellowlees P., Hilty D. The child and adolescent telepsychiatry consultation: can it be a more effective clinical process for certain patients than conventional practice? Telemed. e Health. 2010; 16 (3):289–292. [ PubMed ] [ Google Scholar ]
• Price-Haywood E.G., Burton J., Fort D., Seoane L. Hospitalization and mortality among black patients and white patients with Covid-19. N. Engl. J. Med. 2020; 382 (26):2534–2543. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• R Core Team . R Foundation for Statistical Computing; Vienna, Austria: 2021. R: A Language and Environment for Statistical Computing. https://www.R-project.org/ [ Google Scholar ]
• Rabinowitz T., Murphy K.M., Amour J.L., Ricci M.A., Caputo M.P., Newhouse P.A. Benefits of a telepsychiatry consultation service for rural nursing home residents. Telemed. e Health. 2010; 16 (1):34–40. [ PubMed ] [ Google Scholar ]
• Rajkumar R.P. COVID-19 and mental health: a review of the existing literature. Asian J. Psychiatr. 2020; 52 :102066. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Richardson L., McCauley E., Katon W. Collaborative care for adolescent depression: a pilot study. Gen. Hosp. Psychiatr. 2009; 31 (1):36–45. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Rojas S.M., Carter S.P., McGinn M.M., Reger M.A. A review of telemental health as a modality to deliver suicide-specific interventions for rural populations. Telemed. e Health. 2020; 26 (6):700–709. [ PubMed ] [ Google Scholar ]
• Rush A.J., Trivedi M.H., Ibrahim H.M., Carmody T.J., Arnow B., Klein D.N., Keller M.B. The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol. Psychiatr. 2003; 54 (5):573–583. [ PubMed ] [ Google Scholar ]
• Ruskin P.E., Silver-Aylaian M., Kling M.A., Reed S.A., Bradham D.D., Hebel J.R., et al. Treatment outcomes in depression: comparison of remote treatment through telepsychiatry to in-person treatment. Am. J. Psychiatr. 2004; 161 (8):1471–1476. [ PubMed ] [ Google Scholar ]
• Schechter D., Endicott J., Nee J. Quality of life of ‘normal’ controls: Association with lifetime history of mental illness. Psychiatr. Res. 2007; 152 (1):45–54. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shore J.H., Brooks E., Savin D., Orton H., Grigsby J., Manson S.M. Acceptability of telepsychiatry in American Indians. Telemed. e Health. 2008; 14 (5):461–466. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Soklaridis S., Lin E., Lalani Y., Rodak T., Sockalingam S. Mental health interventions and supports during COVID-19 and other medical pandemics: a rapid systematic review of the evidence. Gen. Hosp. Psychiatr. 2020; 66 :133–146. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Storch E.A., Ehrenreich May J., Wood J.J., Jones A.M., De Nadai A.S., Lewin A.B., et al. Multiple informant agreement on the anxiety disorders interview schedule in youth with autism spectrum disorders. J. Child Adolesc. Psychopharmacol. 2012; 22 (4):292–299. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Urness D., Wass M., Gordon A., Tian E., Bulger T. Client acceptability and quality of life–telepsychiatry compared to in-person consultation. J. Telemed. Telecare. 2006; 12 (5):251–254. [ PubMed ] [ Google Scholar ]
• Vahratian A., Blumberg S.J., Terlizzi E.P., Schiller J.S. Symptoms of anxiety or depressive disorder and use of mental health care among adults during the COVID-19 pandemic—United States, August 2020–February 2021. MMWR (Morb. Mortal. Wkly. Rep.) 2021; 70 (13):490. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Weiner M.F., Rossetti H.C., Harrah K. Videoconference diagnosis and management of Choctaw Indian dementia patients. Alzheimer's Dementia. 2011; 7 (6):562–566. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Whaibeh E., Mahmoud H., Naal H. Telemental health in the context of a pandemic: the COVID-19 experience. Curr. Treat. Options Psychiatr. 2020; 7 (2):198–202. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Ye J., Shim R., Lukaszewski T., Yun K., Kim S.H., Rust G. Telepsychiatry services for Korean immigrants. Telemed. e Health. 2012; 18 (10):797–802. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Yellowlees P., Shore J., Roberts L. Practice guidelines for videoconferencing-based telemental health–October 2009. Telemed. e Health. 2010; 16 (10):1074–1089. [ PubMed ] [ Google Scholar ]
• Zimmerman M., Benjamin I., Tirpak J.W., D'Avanzato C. Patient satisfaction with partial hospital telehealth treatment during the COVID-19 pandemic: comparison to in-person treatment. Psychiatr. Res. 2021:113966. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Zimmerman M., Terrill D., D'Avanzato C., Tirpak J.W. Telehealth treatment of patients in an intensive acute care psychiatric setting during the COVID-19 pandemic: comparative safety and effectiveness to in-person treatment. J. Clin. Psychiatr. 2021; 82 (2) 0-0. [ PubMed ] [ Google Scholar ]